Visual Studio Team Test

Tell Us What You Think

We love to hear your feedback and given that our final beta for Visual Studio 2010 is now available this is your best opportunity to influence the product. In Microsoft Test and Lab Manager(MTLM) we are using the Microsoft Customer Experience Improvement Program (CEIP) to gather application usage data. CEIP collects information about how our customers use Microsoft programs and contains no personally identifiable information. We use this information to improve the products and features customers use most often and to help solve problems.

Participation in the program is voluntary, and for beta release we automatically enable sending this data. Click Home –> Options

It has been a few weeks since folks have started using beta 2 and we are already seeing some good information come thru. Following are some of the metrics that we are collecting:

1) MTLM Usage

a. Most used views (~1000 launches): Run Tests, Plan Contents, Workitem Editor and Test Runner

b. Less frequently used views (~200 launches): Manage VM Template, SaveAs VM Template, Environment Editor, Recommended Tests

c. Views not launched at all: Configuration Editor, About and Settings dialog

2) Actions performed

a. Most folks navigate Back and not Forward (336 vs. 22 invocations of these actions respectively)

b. Docking commands are not used frequently, of 787 invocations of Test Runner, docking behavior was changed only 8 times

c. Help has 394 invocations.

d. Home button had 300 invocations.

3) Artifact Counts (test cases, plans, runs)

a. Artifact counts are < 99 for most items (plans, runs, configurations)

b. 3% users already have between 100 - 500 test cases

c. 10% users ran queries that returned more than 100 workitems

Some of the changes we are considering based on the data are mentioned below.

We saw that Test Plan Manager was launched 766 times but users opened Test Plan Editor from there only 90 times. Could this mean Test Plan Manager may not make sense as the first view in Organize group? The other views on the Organize group in order of preference were: Manage Configurations, Manage Test Cases, Manage S3s so should we elevate one of these to the first view?

Users find  the Go To Work Item functionality in TFC very useful so we provided the CTRL+G shortcut in MTLM too. But this has been invoked only once! We need to consider how we can make this more discoverable.

As time progresses we see more usage patterns and derive useful information to make MTLM more user friendly, but in the meantime your direct feedback is the best way for use to know what you think!

Regards,

Abhishek Lal

Create/Edit Test Settings in Test Plan Properties

In manual or automated testing, you will probably want to specify your own test settings when running tests. The test settings associated with the test plan will be used by default when you click Run button in Run Tests.

There are two ways to create or edit a test settings associated with the test plan. The first option, you can open Lab Center | Test Settings. In the Test Settings Manager, you can click New or Open a selected test settings. Then you can specify the test settings in Test Plan Properties.

The second option is much easier. You can directly create a new test settings from the Test Settings dropdown in Test Plan Properties. Click the <New…> dropdown item.

The test settings editor will be opened inside of the Test Plan Properties. Follow the wizard and fill in all the fields that you care and click Finish.

Now in the Test Plan Properties, you will see your newly created test settings automatically selected.

You can also open an existing test settings from the dropdown, edit and save it. This will be done within Test Plan Properties as well.

Thanks,

Christine Zhao

Clearing MTLM’s Settings

With MTLM, we store our settings in your user profile. Specifically the ‘Local App Data’ section of your user profile. These are simple XML files that have your settings from MTLM – window size/location, whether the Test Runner is docked or floating, the queries you’ve configured in Test Case Manager, and other fun & exciting settings.

To easily navigate to them on Windows Vista, 2008, 7, or 2008 R2, open this folder in Windows Explorer:

%localappdata%\Microsoft\TeamTest\v10.0\

Or, on XP, 2003, and 2003 R2:

%userprofile%\Local Settings\Application Data\Microsoft\TeamTest\v10.0\

In this directory you will see a number of *.config files. They will be prefixed with ‘mtlm’. The main configuration file is ‘mtlm.config’ – the other files you see are isolated files for each of the activities within MTLM. Feel free to open these files up; they’re just simple XML files with name-value pairs in them. Of course, change them at your own risk. :)

Sometimes, you might want to clear the settings that you’ve got stored. Either because you like to have things ‘clean’, or because you are trying to diagnose an issue. The simplest thing to do is to delete all the *.config files in the folder mentioned above. You should do this while MTLM is not open, since we save your settings on exit. Once you’ve done this, you’ll start MTLM in the clean pristine state you had it when you first installed it.

Yours,

Dominic Hopton

Navigating Back and Forward

In a previous post, I briefly mentioned the back and forward buttons. Today I want to talk a bit more in depth about the issues we had in adding this common and seemingly simple concept to Microsoft Test and Lab Manager. Although the general behavior of back and forward does what one wants, it may not make sense in some specific scenarios.

The back and forward paradigm is most recognizable from web browsers. I’m not 100% sure, but I think that is where the notion was first created. In the very least, that is where it became popular. From there, many applications have integrated the concept. Users demand it! If my mouse back button doesn’t do anything in an application that could support it, I get down right cranky! :)

Well it took some arguing, but we eventually got it added to MTLM, and you’ll see it in Beta 2. Most of the arguing has to do with how it would be implemented correctly. A browser is a very different beast than MTLM. Other applications I could find that supported back and forward also had significant differences in their navigation model that it was hard to find a good example to follow. We wouldn’t want to reinvent the wheel if we didn’t have to, mostly because then the behavior wouldn’t be intuitive.

The primary difference between a browser and MTLM in terms of back/forward buttons is that when you go back in a browser, the view you just left is no longer in memory. If you go forward, the browser may load the content from the cache or get a new copy from the server, but either way it will re-render the content. Any prior settings or state you had are probably lost. For instance, if you started writing some text into a field, then went back then forward, your text will probably be gone. The same is not true in MTLM. Each view is kept in memory until you close it.

So why does that matter? Well, if you go back in the navigation history a bit, and then open a new view you will orphan the former forward items. You’ve lost them from a back/forward navigation point of view. With MTLM we don’t want to lose views out of the back/forward navigation. They are still running taking up memory. You may want to get back to one of the items (e.g. a bug you opened up), having to go through all the steps you previously took to find it feels like a terrible waste of time. You could get back to it from Open Items, but only if it is an artifact. You could also get back to it by closing down every other views until the orphaned items began showing, but that is pretty destructive of your workspace.

So we resorted to inserting new views in the current position. Here is a sample walk through where a bold item is the current view:

What might seem really weird to many people is that in step 4 the forward button is still enabled and if you go forward you’ll bring the Run Tests view to the foreground. That is an experience you won’t find in the browser. However, you also won’t get the benefit of step 7 either, which is convenient.

Cheers,

David Williamson

Engineering Lead, MTLM

Working with and customizing work item categories

As this previous post explains, before Microsoft Test and Lab Manager, there was no real need to know which work items represent bugs or test cases. With VS2010, we introduced the concept of work item categories, which allows us to give a meaning to some of the general work item types we use in the product.

Why do you care about work item categories? Say you want to track information about two different types of bugs - for the purpose of this discussion we’ll use an example of upgrading from TFS 2008 to 2010 (more information about this scenario here). In this scenario, you want to keep your old bugs, but you also want to take advantage of all the new goodies you get with MTLM, like filling in your repro steps and system information field when creating or updating a bug. To do this you would create a new bug work item type, leaving the old bugs alone. You can do this easily by using witAdmin. The easiest way to do this is to start with downloading the new bug template, information about how to do this again contained in Chris Patterson’s blog.

When you are done downloading, you’ll want to open the Bug.xml file and make sure you change the name of the work item type, otherwise when importing you’ll overwrite the existing one. You can import your work item by doing:

witadmin importwitd /collection:[tfs collection] /p:[project] /f:[filename].xml

You now have two different work item types and can create & query information about both separately from VS, but to get the best experience in MTLM, you need to add the newly created bug work item type to the Bug Category. You again use witAdmin, and again, a good starting point is to export the categories:

witadmin exportcategories /collection:[tfs collection] /p:[project] /f:[filename].xml

You can modify this file by adding your work item inside the Bug Category. It will look like this:

Notice that one of the work item types has a tag DEFAULTWORKITEMTYPE while the other one has WORKITEMTYPE, this is used to specify the type of bug you want to create when doing New inside of MTLM’s Verify Bugs activity, or from the test runner, which in this case you want to change to the new bug type, so people can create bugs with all the rich information pre-filled.

Save and upload by doing:

witadmin importcategories /collection:[tfs collection] /p:[project] /f:[filename].xml

You will now see both types of bugs seamlessly in MTLM wherever queries for bugs are used!

All of this work item categories and custom work item templates has some implications on how you write queries. I’ll have more on that on a later post.

Thanks,
Guillermo Serrato

Uploading Large Attachments

During the course of a manual test run, you may end up with large attachments on the test result. This could be attached by hand, but more commonly it will be output from a data collector. Some of the data collectors create files of substantial size due to the type of data they collect. The two most prominent examples are the video and IntelliTrace collectors.

When you end the test case, the data collectors finalize their collection and put all that info into a file. When the test result is saved, those attachments are uploaded to TFS. Since they are large, that could take quite a long time, especially if you are on a slow link to TFS. MTLM will continue uploading those files in the background and allow you to continue working.

The downside here is that any bugs you may have created will have links to an attachment. If that attachment is still uploading then it will not yet be available to the developer. MTLM will continue to upload the file(s) as long as the process is running. If you attempt to shut down while a file upload is in progress, then you’ll get a warning.

Don’t worry. As the warning says, if you need to restart MTLM that is fine. The file(s) will continue uploading when you next start the application, as long as the local temporary files are note removed from the machine.

Cheers,

David Williamson

Engineering Lead, MTLM

Work Item Categories

New to TFS in VS2010 is the concept of work item categories. Previously, a work item was known only by its name, and it was up to the user to distinguish what a work item was meant to represent based on the name. Out of the box, a work item may have the name of ‘Bug’, but your organization may choose to rename it (as ours has to Dev10 Bug). Furthermore, a work item name is also localizable so anyone interpreting the name of a work item also has to be aware of what that item is called in other languages.

If you run the project creation wizard and select one of the out of the box templates, it will specify the work item categories for bug, test case, shared step, and requirement. To see this information, you will use the witadmin.exe utility. Here is sample output from our team development server (command in yellow, output in green):

Work item categories are only used in MTLM so far. We use them to restrict the type of work item to show in a given list, and to offer a New button for that work item type in that list. Examples include the Test Plan Contents and the Verify Bugs views.

You are probably wondering and hoping at this point that this information can be customized. The answer is Yes! We’ll have a follow up blog post explaining the main scenarios for customizing work item category information and instructions on how to do it. Also, we’ll have a post on how you as a user should be aware of this concept when writing work item queries.

Cheers,

David Williamson

Engineering Lead, MTLM

Planning vs Testing

On a previous post by David Williamson, Activity Centers, created on Oct 26th (2009), it was mentioned that Microsoft Test and Lab Manager (MTLM) has a Plan center group, where you can organize your suites/tests, and a Test center group, where you actually run those tests.

Customers that have used our product are sometimes confused by the views, which look very similar. I’ll start a series of blog posts that describe what each of these views does, and will hopefully clear out any confusion. Both views exist for an important reason, and they look similar for an important reason as well.

I’ll start by saying that the suites that appear under ‘Contents’ in the Plan center group are not always the same suites that appear under ‘Run Tests’ in the Test center group. For instance, while you’re planning, you might have one of your suites marked as ‘In Planning‘. Well, this suite won’t show up when you Test until after you mark your suite as ‘In Progress’.

Here’s how Plan Contents looks like when a suite is marked as ‘In Planning’ (In the Plan center group):

And here is how the ‘Run Tests’ activity looks like (In the Test center group):

Notice that we need both views to be similar because we want to test whatever we are planning to test; but the views are still different (for a good reason).

More reasons of why these views exist, and their differences, in a later post.

Thanks,
David Gorena Elizondo

Lend a Helping Hand

Imagine a new tester starts working on your team. One of the first things you’ll need to do to help them get ramped up is to have them install Microsoft Test and Lab Manager (MTLM) and then connect to the right server (with or without SSL), port, project collection, and project.

Well, we hope to make this easier. The most work involves going through the connection experience. The person has to know what server to connect to, which gets tricky if their server requires SSL or has a non-default virtual application directory. They also have to find the right project collection and then project.

We can simplify this by allowing you to share all this information in a clickable URI. Imagine your new co-worker gets a new email from you, or perhaps visits the team wiki, that has a link to MTLM setup, and then a link to connect!

For now these links need to be hand rolled, but we’re investigating how to more easily create one directly from MTLM. Here is an example URI and a breakdown of it.

example:

mtlm://windfall/tfs/DefaultCollection/p:Camano

break down:

mtlm[s]://<server>[:port]/<virtual app dir>/<collection>/p:<project>

We have more planned for release (as in, not in Beta2) to make this even richer.

Cheers,

David Williamson

Engineering Lead, MTLM

Single Instance Application

Microsoft Test and Lab Manager (MTLM) is, for the most part, a single instance application. That is, if you open it a second time from the Start Menu while an instance is already running, the previous instance is brought to the foreground. You will not get two windows of MTLM.

This is primarily because MTLM will respond to open requests from several different places, most notably from Visual Studio. For example, if you have a bug open in Visual Studio and it has a link to a test result, you might want to open that to learn more. There isn’t a test result viewer in VS for tests run from MTLM, so VS simply asks MTLM to open and show the test result.

If MTLM is not already running, one will start. It will connect to the server, project, and plan the test result is associated with… even if that is not a server, project, or plan you’ve ever manually connected to in MTLM.

If MTLM is already running, then you won’t end up with two instances, the current instance will open that test result for you. If that test result is from a different server or project, then you’ll be prompted to connect to it instead.

There is a way to have more than one copy of MTLM running at a time, however. Using the ‘Run as different user’ option off the context menu for the application shortcut, you can specify another user’s credentials which will run the process as that user. Note in order for this option to appear, you must hold down the shift key when right-clicking.

Cheers,

David Williamson

Engineering Lead, MTLM

Open Items and Manage Queries

In my last post, I discussed whether Manage Queries is a manager or artifact, and promised to cover other aspects of behavior.

One such distinction of behavior between managers and artifacts is that artifacts show up in Open Items. Managers are just relaunched from the chrome (navigation area), or you can get back to them with the Back and Forward buttons. Artifacts were a little harder to find in the first place since you had to find an item in a list, run a query, or use Control+G to specifically open the work item by ID, so we show them in the Open Items list.

Notice when you first start Manage Queries, the Open Items count grows by 1. Without doing anything in that view, browse away and relaunch from the chrome, and the same view will open. Your Open Items count did not grow.

Now select a query and run it, then browse away. Navigate back to Manage Queries from the chrome. A new item is opened and your Open Items count grows by 1. If you drop down Open Items you’ll see two Manage Queries views open.

This is designed so you can have multiple different queries open at the same time. However we don’t want to open a bunch of blank Manage Queries, so if a blank one exists it is simply brought to the foreground. However if an instance of Manage Queries has info, then you can decide whether you want a new one (via the Chrome) or the old one (via Open Items).

Cheers,

David Williamson

Engineering Lead, MTLM

Manage Queries – Artifact or Manager?

If the most recognizable part of MTLM for anybody who is familiar with VS’s Team Explorer from 2005 and 2008 is the work item, the next most recognizable piece of UI would be Manage Queries. This view is akin to the Work Items node in Team Explorer. It has a Team Queries node, and a My Queries node. Inside you’ll find all the team and personal queries that are also seen in Team Explorer.

So from that aspect this view acts more like a MTLM manager, much like Test Plan Manager, Test Case Manager, Shared Steps Manager, Test Configuration Manager, and Test Settings Manager. All of these views list a bunch of artifacts, which can be opened to view or change. These views can be refreshed, but they cannot be saved (since there is no way to get into a ‘dirty’ state).

However, unlike those manager views, Manage Queries opens the selected artifact inside the same view, rather than in a new window. As long as the item has not changed, the view still has nothing to be saved. As soon as a change happens, we do several things. First, the title gets an asterisk to indicate a save must happen to retain the changes. Second, we disable the tree view which lists all the items since selecting a new item would also lose your changes. If you press the refresh button, you will be prompted to discard your changes. If you try to close the view, same thing.

In this sense, it is definitely an artifact. I like to call it an artifact with a manager view built in. It is definitely a different kind of beast than the other manager and artifact views in MTLM, but it is also a unique kind of activity as well. It seems more likely one will be running several different queries in a row, without changing them. From this aspect, it is convenient to be able to do all that inline – no opening and closing of windows.

Stay tuned tomorrow when I’ll talk about another interesting aspect of this view’s behavior.

Cheers,

David Williamson

Engineering Lead, MTLM

Screenshots!

I’ve been silly lately. I’ve failed to include screenshots with my blog posts. I didn’t realize how easy it would be with Windows Live Writer. I’ve been using the app, but didn’t realize it would handle posting the images somewhere for me.

So, expect all future blogs to have screenshots… and I’ll work on going back through the old blogs and adding screenshots to each.

Humbly,

David Williamson

Engineering Lead, MTLM

P.S. Nice job Live Writer team. This app rocks!

Using Shared Steps

In MTLM, test cases can reference a reusable, sharable set of test steps. This cuts down on the amount of work it takes to author new test cases, because many test cases contain some of the same steps.

The ideal use for this is in common tasks. The quintessential shared step is for logging into the application. Assuming your application cares about identity as many do, this is something nearly every test case will need to do as part of the process to get where you want to test.

If you didn’t have shared steps, you would either have to rewrite those steps for every test case, or you’d simply have a test case that handled login and you’d position that test ahead of the others in the test suite.

Rewriting the steps in every test case is painstaking and annoying. I want to be productive as a tester, not waste my time doing repetitive tasks! Worse, if the steps change for logging in, I have to go and change every test case. That is a terrible option, and I would prefer to avoid it as much as possible.

I could just ignore obvious steps like this, but that might be difficult for people who take over my work someday to understand what is expected. It is best if I’m explicit

The other solution I mentioned was ordering test cases in a suite. With this option, I add a test case to the top of my suite called Login, and then have other test cases that require the tester to be logged in appear after. This has worked for many testers for years, and I think it is better than the previous option. However, it is a bit tedious. For one, I have to be very sure the test suite is always ordered correctly. Let’s say I have my Logout test case at the end of the suite. When a new test case is added, I have to be sure to update the order (again) to make sure it is inside of the Login/Logout scope. Also, if I want to include a test case that requires Login in another suite, then I have to make sure that suite also includes the Login (and probably Logout) test. So again, this is just more maintenance.

Also, now I’m testing Login and Logout explicitly in every test suite which may not be interesting to me.

If I go the Shared Steps route, then every test case that needs Login explicitly references it. There’s no need to keep track of the association every suite my test case is in.

The downside is that this implies I need to login and logout with every test case. This really could slow down execution time. Can’t I just run through all my test cases without logging in and out? Sure, I think you could. The steps don’t have to be mandatory. You might make the Login steps conditional (if not already logged in, then do the next few steps), and Logout could also be conditional (Logout if you will close the application).

In the test step control, it is easy to reference existing shared steps. Just click the ‘Insert shared steps’ button, and find the shared step you want in the list. If you need to create a shared step for the first time, you can even promote steps you’ve written to be a new shared step by selecting those rows and clicking the ‘Create shared steps and insert at the current position’ button.

Cheers,

David Williamson

Engineering Lead, MTLM

New features in Beta2 (compared to Beta1)

We recently announced Beta2 of VS2010. If you have installed the Beta1 release and played with it or even if you didn’t and are experiencing the product for the first time, you are perhaps wondering what is new in Beta2. The table below is an enumeration of the new features in Beta2. As you can see from the list below the team has been very busy reacting to your feedback and adding a lot of features to round off the product to address your needs. In a future post I will expand on each of the features and explain how to get to them so you can experience these for yourself. Do let me know which of these features you would like me to blog about first so I can prioritize them higher.

Cheers,

Ram Cherala

Principal Program Manager, Visual Studio Test Tools