VS Team System Test

Planning vs Testing

On a previous post by David Williamson, Activity Centers, created on Oct 26th (2009), it was mentioned that Microsoft Test and Lab Manager (MTLM) has a Plan center group, where you can organize your suites/tests, and a Test center group, where you actually run those tests.

Customers that have used our product are sometimes confused by the views, which look very similar. I’ll start a series of blog posts that describe what each of these views does, and will hopefully clear out any confusion. Both views exist for an important reason, and they look similar for an important reason as well.

I’ll start by saying that the suites that appear under ‘Contents’ in the Plan center group are not always the same suites that appear under ‘Run Tests’ in the Test center group. For instance, while you’re planning, you might have one of your suites marked as ‘In Planning‘. Well, this suite won’t show up when you Test until after you mark your suite as ‘In Progress’.

Here’s how Plan Contents looks like when a suite is marked as ‘In Planning’ (In the Plan center group):

And here is how the ‘Run Tests’ activity looks like (In the Test center group):

Notice that we need both views to be similar because we want to test whatever we are planning to test; but the views are still different (for a good reason).

More reasons of why these views exist, and their differences, in a later post.

Thanks,
David Gorena Elizondo

Lend a Helping Hand

Imagine a new tester starts working on your team. One of the first things you’ll need to do to help them get ramped up is to have them install Microsoft Test and Lab Manager (MTLM) and then connect to the right server (with or without SSL), port, project collection, and project.

Well, we hope to make this easier. The most work involves going through the connection experience. The person has to know what server to connect to, which gets tricky if their server requires SSL or has a non-default virtual application directory. They also have to find the right project collection and then project.

We can simplify this by allowing you to share all this information in a clickable URI. Imagine your new co-worker gets a new email from you, or perhaps visits the team wiki, that has a link to MTLM setup, and then a link to connect!

For now these links need to be hand rolled, but we’re investigating how to more easily create one directly from MTLM. Here is an example URI and a breakdown of it.

example:

mtlm://windfall/tfs/DefaultCollection/p:Camano

break down:

mtlm[s]://<server>[:port]/<virtual app dir>/<collection>/p:<project>

Note: The text in square brackets is optional. You don’t have to specify a port if your server uses the default (8080), and you only need mtlms if your server requires SSL.

We have more planned for release (as in, not in Beta2) to make this even richer.

 

Cheers,

David Williamson

Engineering Lead, MTLM

Single Instance Application

Microsoft Test and Lab Manager (MTLM) is, for the most part, a single instance application. That is, if you open it a second time from the Start Menu while an instance is already running, the previous instance is brought to the foreground. You will not get two windows of MTLM.

Note: There are other examples of this, like Office Communicator, Windows Live Messenger, and the Windows 7 Snipping Tool.

This is primarily because MTLM will respond to open requests from several different places, most notably from Visual Studio. For example, if you have a bug open in Visual Studio and it has a link to a test result, you might want to open that to learn more. There isn’t a test result viewer in VS for tests run from MTLM, so VS simply asks MTLM to open and show the test result.

If MTLM is not already running, one will start. It will connect to the server, project, and plan the test result is associated with… even if that is not a server, project, or plan you’ve ever manually connected to in MTLM.

If MTLM is already running, then you won’t end up with two instances, the current instance will open that test result for you. If that test result is from a different server or project, then you’ll be prompted to connect to it instead.

There is a way to have more than one copy of MTLM running at a time, however. Using the ‘Run as different user’ option off the context menu for the application shortcut, you can specify another user’s credentials which will run the process as that user. Note in order for this option to appear, you must hold down the shift key when right-clicking.

Cheers,

David Williamson

Engineering Lead, MTLM

Open Items and Manage Queries

In my last post, I discussed whether Manage Queries is a manager or artifact, and promised to cover other aspects of behavior.

One such distinction of behavior between managers and artifacts is that artifacts show up in Open Items. Managers are just relaunched from the chrome (navigation area), or you can get back to them with the Back and Forward buttons. Artifacts were a little harder to find in the first place since you had to find an item in a list, run a query, or use Control+G to specifically open the work item by ID, so we show them in the Open Items list.

Notice when you first start Manage Queries, the Open Items count grows by 1. Without doing anything in that view, browse away and relaunch from the chrome, and the same view will open. Your Open Items count did not grow.

Now select a query and run it, then browse away. Navigate back to Manage Queries from the chrome. A new item is opened and your Open Items count grows by 1. If you drop down Open Items you’ll see two Manage Queries views open.

This is designed so you can have multiple different queries open at the same time. However we don’t want to open a bunch of blank Manage Queries, so if a blank one exists it is simply brought to the foreground. However if an instance of Manage Queries has info, then you can decide whether you want a new one (via the Chrome) or the old one (via Open Items).

Cheers,

David Williamson

Engineering Lead, MTLM

Manage Queries – Artifact or Manager?

If the most recognizable part of MTLM for anybody who is familiar with VS’s Team Explorer from 2005 and 2008 is the work item, the next most recognizable piece of UI would be Manage Queries. This view is akin to the Work Items node in Team Explorer. It has a Team Queries node, and a My Queries node. Inside you’ll find all the team and personal queries that are also seen in Team Explorer.

So from that aspect this view acts more like a MTLM manager, much like Test Plan Manager, Test Case Manager, Shared Steps Manager, Test Configuration Manager, and Test Settings Manager. All of these views list a bunch of artifacts, which can be opened to view or change. These views can be refreshed, but they cannot be saved (since there is no way to get into a ‘dirty’ state).

However, unlike those manager views, Manage Queries opens the selected artifact inside the same view, rather than in a new window. As long as the item has not changed, the view still has nothing to be saved. As soon as a change happens, we do several things. First, the title gets an asterisk to indicate a save must happen to retain the changes. Second, we disable the tree view which lists all the items since selecting a new item would also lose your changes. If you press the refresh button, you will be prompted to discard your changes. If you try to close the view, same thing.

In this sense, it is definitely an artifact. I like to call it an artifact with a manager view built in. It is definitely a different kind of beast than the other manager and artifact views in MTLM, but it is also a unique kind of activity as well. It seems more likely one will be running several different queries in a row, without changing them. From this aspect, it is convenient to be able to do all that inline – no opening and closing of windows.

Stay tuned tomorrow when I’ll talk about another interesting aspect of this view’s behavior.

Cheers,

David Williamson

Engineering Lead, MTLM

Screenshots!

I’ve been silly lately. I’ve failed to include screenshots with my blog posts. I didn’t realize how easy it would be with Windows Live Writer. I’ve been using the app, but didn’t realize it would handle posting the images somewhere for me.

So, expect all future blogs to have screenshots… and I’ll work on going back through the old blogs and adding screenshots to each.

Humbly,

David Williamson

Engineering Lead, MTLM

 

P.S. Nice job Live Writer team. This app rocks!

Using Shared Steps

In MTLM, test cases can reference a reusable, sharable set of test steps. This cuts down on the amount of work it takes to author new test cases, because many test cases contain some of the same steps.

The ideal use for this is in common tasks. The quintessential shared step is for logging into the application. Assuming your application cares about identity as many do, this is something nearly every test case will need to do as part of the process to get where you want to test.

If you didn’t have shared steps, you would either have to rewrite those steps for every test case, or you’d simply have a test case that handled login and you’d position that test ahead of the others in the test suite.

Rewriting the steps in every test case is painstaking and annoying. I want to be productive as a tester, not waste my time doing repetitive tasks! Worse, if the steps change for logging in, I have to go and change every test case. That is a terrible option, and I would prefer to avoid it as much as possible.

I could just ignore obvious steps like this, but that might be difficult for people who take over my work someday to understand what is expected. It is best if I’m explicit

The other solution I mentioned was ordering test cases in a suite. With this option, I add a test case to the top of my suite called Login, and then have other test cases that require the tester to be logged in appear after. This has worked for many testers for years, and I think it is better than the previous option. However, it is a bit tedious. For one, I have to be very sure the test suite is always ordered correctly. Let’s say I have my Logout test case at the end of the suite. When a new test case is added, I have to be sure to update the order (again) to make sure it is inside of the Login/Logout scope. Also, if I want to include a test case that requires Login in another suite, then I have to make sure that suite also includes the Login (and probably Logout) test. So again, this is just more maintenance.

Also, now I’m testing Login and Logout explicitly in every test suite which may not be interesting to me.

If I go the Shared Steps route, then every test case that needs Login explicitly references it. There’s no need to keep track of the association every suite my test case is in.

The downside is that this implies I need to login and logout with every test case. This really could slow down execution time. Can’t I just run through all my test cases without logging in and out? Sure, I think you could. The steps don’t have to be mandatory. You might make the Login steps conditional (if not already logged in, then do the next few steps), and Logout could also be conditional (Logout if you will close the application).

In the test step control, it is easy to reference existing shared steps. Just click the ‘Insert shared steps’ button, and find the shared step you want in the list. If you need to create a shared step for the first time, you can even promote steps you’ve written to be a new shared step by selecting those rows and clicking the ‘Create shared steps and insert at the current position’ button.

Cheers,

David Williamson

Engineering Lead, MTLM

New features in Beta2 (compared to Beta1)

We recently announced Beta2 of VS2010. If you have installed the Beta1 release and played with it or even if you didn’t and are experiencing the product for the first time, you are perhaps wondering what is new in Beta2. The table below is an enumeration of the new features in Beta2. As you can see from the list below the team has been very busy reacting to your feedback and adding a lot of features to round off the product to address your needs. In a future post I will expand on each of the features and explain how to get to them so you can experience these for yourself. Do let me know which of these features you would like me to blog about first so I can prioritize them higher.

MTLM/Test Runner

·         Number of performance and scale improvements on both client and server side ·         Support for resetting results in a run ·         Improved handling of attachments when filing bugs ·         Better start up experience ·         History (Back & Forward) ·         Full support for any field on a work item being shown in grids ·         Single panel Test Runner with an easier and more intuitive interface ·         Advanced Test Settings support with a much better user experience ·         Full query editing in Test Case/Shared step manager and Add Requirement/Add Tests activities

Coded UI

·         Redesigned the Coded UI Test Builder UI to be more intuitive ·         Code generation enhanced to provide automatic parameterization and  ease of maintenance ·         Support added for WPF Datagrid and virtualized controls. ·         Support for specialized classes

Test Execution

·         Simplified test settings ·         Better surfacing of warnings and errors from data collectors

Remote execution

·         Controller manager in MTLM – improved functionality and usability ·         Interactive agent UI with config UI and systray app

Web Access

·         Repro steps and system information controls on the bug form now usable and editable from web access ·         Read-only Test Steps control on test case form now available in Web Access

Load Test

·         Support for comparing two load test performance runs in Excel

Others

·         Ability to publish test results from VS2008 to a VS2010 server (using the newly released VSTS2008 GDR)

Cheers,

Ram Cherala

Principal Program Manager, Visual Studio Test Tools

Test Case Authoring from the Test Plan Contents

A test case is just a work item. You can create a new one from Team Explorer in Visual Studio. You can open Excel and connect to the server from the Team add-in, and create one from there. In MTLM, you can also go to Test Case Manager and create test cases.

Those are all fine, but if you are authoring test cases in the context of a test plan or requirement, there is a better way. When you are in Test Plan Contents (the suite manager of MTLM) you can add existing test cases to a suite, and you can also create test cases from there as well. You get the same experience of the work item form, but when you are done, the item is also added to the suite.

Note, this works only for static and requirement-based suites.

In the static suite case, the test case is added to the selected suite and all default configs specified by that suite are added to the new test case.

In the requirement-based suite, the test case is linked to the requirement, which adds it to the suite, and all the default configs specified by that suite are added to the new test case.

If you didn’t author the test case from Test Plan Contents, then you’d have to manually go add the test to the suite of your choice after the fact. I said this was better, because it helps save you time by accomplishing both tasks at once (create, add to suite).

Cheers,

David Williamson

Engineering Lead, MTLM

‘Active’ Tests

In the Run Tests activity, you’ll see test cases grouped up by various states including Active, Failed, Blocked, and Completed.

This idea comes from the workflow many customers report to using where they run many tests only once per iteration. So the intent is to take a test that is Active (meaning it is ready and available to be run) and test it in the Test Runner.

Based on the outcome of that test result, the test state will change to Failed or Completed.

Completed tests are done, unless you decide you want to run that test again. You can simply select it and run, you can select it and hit Reset so it moves back to Active (ideally for another person to pick it up), or discover it and reset it as a recommended test based on code churn in the product code that the test covered.

Failed tests will generally have a bug associated with them, so they’ll be rerun from the Verify Bugs activity when the bug is resolved back to the tester.

Through these cycles, the goal is to get every test into the Completed state. Then it is time to do exploratory testing, or identify existing test cases to run again.

When you run tests from the right-hand pane of the Run Tests activity, the selected items are always taken into the run. However, if you select a suite and try to run it, only the tests in the Active state will be taken into the run. This is to avoid taking a Completed tests (one that reached the end of the cycle) into the run, or a Failed test which is still waiting on a bug fix.

Cheers,

David Williamson

Engineering Lead, MTLM

Lab Management?

MTLM stands for Microsoft Test and Lab Manager. This blog is dedicated to the testing aspects of this application (as well as the other testing features in VS).

Even though testers need to use the lab management activities they tend to be doing one type of activity or another, so we have a separate blog just for lab management-specific activities.

Test Configurations in a Test Plan

For many, many test teams, we hear that they don’t typically test on more than one configuration. This is a big departure from what we at Microsoft are used to. Obviously we have many operating system versions including languages, versions of IE, versions of Office, etc. We’re used to dealing with testing on many platforms because of the configuration matrix.

So for testing teams that don’t use different configurations, we are here to make your life as simple as possible. There’s no reason to make your life harder than it has to be. For you, we have an out of the box, default configuration. You can edit it if you want, or just ignore it entirely. Go on about your testing, and don’t worry about configs.

For all the poor souls who do need to test on multiple configurations, we haven’t forgotten you either. We use MTLM ourselves, so it must meet our own needs too.

Creating configurations is fairly straight forward. First, you visit the Test Configuration Manager (in Organize), and open the Edit Test Variables dialog. In here you can define the different factors you will have variations in, such as OS, Browser, language, etc. Then for each factor, define the variations in the right pane. Then you go back to the manager and begin defining test configurations. Give it a descriptive title, and then add the relevant factors to the grid at the bottom, giving each the correct value. This will be helpful later in reporting where you can look for test result trends across the factors rather than simply reporting on each independent configuration.

The other important value on a test config is whether it is the default for new test plans. If it is set to default, whenever a new test plan is created it will automatically include those configs. Yes, you can remove project defaults from your plan, and add non-defaults. The goal for the person defining the test config project defaults is to make sure new plans have the configs you expect.

Test config defaults on the plan are automatically applied to test cases added to the plan. In Test Plan Properties, you’ll find a test configuration selector where this is specified. Again this is meant to make your life easier, so plan ahead and make sure you have the configs set up on your plan before test cases are added. If you get it wrong, don’t worry, you can change configurations applied to test cases after the fact.

There’s yet another level of ways to scope defaults. We’ve already talked about project-level defaults, test plan defaults, and now we’ll introduce test suite level defaults. Any new suite in the plan will inherit from the parent. That can be changed so that all test cases add to it get a different set of configs applied. In the suite header in Test Plan Contents, look for a drop down next to the label that says ‘Test configurations (x):’.

Finally, test cases can be changed independently of the plan and suite. Just select the test cases in question, and click the Configurations button in Test Plan Contents.

One last thing, when you are looking at test cases in the suites in the Test Plan Contents activity, you’ll see one instance of the test case. However, since many configurations can be applied to that once instance of the test case in the suite, these entries will be multiplied when they are viewed in the Run Tests activity. This is because the test case needs to be run multiple times, and selected independently from the other instances so they can be run with other similar configs. To assist in that work flow, you can use the Filter button on the Run Tests suite tree view toolbar to only show a specific config. Now you can focus on testing just on that one config, then change the filter later when you are ready to move onto another config.

Cheers,

David Williamson

Engineering Lead, MTLM

What is a Test Plan?

A common question we hear is how and when to use a test plan in MTLM.

To most testers, a test plan is a document that describes the intended approach to testing for a feature, or more likely a group of features, for a specific release. It typically reviews the features to be tested, discusses the approach to testing, and it may even list some or all of the specific test cases to be run. It definitely covers timelines and deadlines.

In MTLM, we wanted to expand the concept of a test plan beyond just a document. The information in the test plan should be structured, able to be queried, and directly reference the test artifacts that eventually help define when the testing is complete.

So we start with the Test Plan Properties activity that exposes the high level values. We have a title, description, owner, state, start date, and end date. These values show up in the Test Plan picker and Test Plan Manager to help you find the right test plan you want to work in. It also holds information for running tests, like Test Settings and Test Environments for both manual and automated tests.

 

The test lead will probably set up this test plan artifact for the team. There is a lot of flexibility in defining scope and duration for a test plan. It really depends on how you want to use it. However, we can offer some guidance on when scope is too big or small, or when the duration is too short or too long.

Since we define a test plan as a test effort for an area for an iteration, it is up to you to define the area and iteration. For a product release, you could have 3 feature teams and 5 iterations… so should that be 15 test plans? 3? 1? 80?

Well, the answer to that, for me, is whatever is most convenient for you. However there are some hard requirements that we should review first. I think the biggest factor is who is driving and monitoring the test effort. If a project leader has to maintain and switch between several test plans, that ends up being tedious. So perhaps there are several feature teams working in a single test plan because there is a single owner, they’d each work out of different test suites. That way they could avoid stepping on each other, but… they could also have common areas which might be a bonus. It also means reporting can easily be scoped to the whole effort based on what is in the plan. The second most important factor has to do with test run settings. Do these efforts use the same build definition? Do they move forward to new builds at the same time? If not, it can be inconvenient for testers when they kick off test runs to have to manually select the real build they want to test with. So to sum up, the main factors for me are ownership, reporting, team affinity, and build selection.

The other activity that helps define a test plan, is the Test Plan Contents. This is where test cases are added to the plan, and optionally organized into test suites.

Both of these activities live under the Plan activity group.

Cheers,

David Williamson

Engineering Lead, MTLM

Test Steps in MTLM

In a test case, there is a control to enter test steps in the first tab. It has several columns, but typically a person will be adding a test title, and sometimes some expected text. There’s also the ability to add attachments to steps.

What you’ll see off the bat is an insertion row. Click inside it with your mouse, or Tab down to it with the keyboard, and type in your first step title. Assuming you don’t have any expected text for the first step, just hit Enter.

Continue to type in more steps hitting Enter after each one. If you want expected text, just Tab over, type it, then hit Enter. Note that steps with expected text automatically become ‘Validation Steps’. This means the step will be represented and treated differently in the Test Runner since the expectation is that the tester will ensure the application under test has met the requirements. Use these judiciously since you may not want to mark every step of a test. If you are like me, once you know your test, you’ll want to blow through several steps at a time without indicating success or failure.

Each time you hit Enter it commits the current row and gives you a new row. Feel free to save as you go with Control+S.

You can add attachments via the toolbar button, or via the context menu.

There are lots of helpful toolbar buttons including some to help you move steps up or down (supports multi-select too!), insert a step, and insert a reference to shared steps.

The final thing to add here is data parameters. Sometimes you may want your test case to use different data for a step. You can define a parameter here to refer to data you’ll define later. A good example of this is the user name and password to be used to login. While you are typing your step text, prefix the word that you want to represent your parameter with an at sign (@) like: Login to the application as @user with @password. In the grid below, you’ll see an entry for each data parameter appear. Later you can drop into that grid and begin adding values. Note the @ symbol needs to have a space before it, so an email address which has an @ symbol in the middle won’t signal what follows as a data parameter.

Cheers,

David Williamson

Engineering Lead, MTLM

Migrating Test Cases

Anutthara blogged about how to migrate test cases using Excel. This wouldn't be hard to do, except getting test steps into a test case via the standard Excel integration is not possible. Our team has written a migration tool to help with that.