Visual Studio Team Test

Order Test Cases

Sometimes, you want to run your tests in a specific order. For example, you are testing an online shopping site. You want to test the work flow of Login, Browse items, Search items, Add items to cart and Check out. You have the test cases for each step in the wok flow. You want to execute the test cases in the above work flow order. By default, MTLM orders the test cases in the order you add them to the suite. However, you may not have added the test cases in the order you want, like what I have in the first picture. In this case, you can click the Order button on the toolbar.

In the following dialog, change the order number. Click Update order button to preview the change. You can have duplicate numbers, MTLM will order the duplicates in the order they show in the list. When you feel good about the new order, you can click Apply Changes to save it.

After that, open Run Tests, select all the tests and click Run. Your tests will be run in the right order!

Thanks,

Christine Zhao

Planning vs Testing (Part 3)

In part 1 and part 2 of the series, I have mentioned reasons why we use similar views for planning and executing test cases (and I have discussed what makes these views different). Today, I will focus on yet another specific reason: Configurations.

We can use the ‘Planning view’ under ‘Contents’ (Plan center group) to assign configurations to one or more test cases that live in our test plan. The view will show how many configurations are associated to any given test case.

However, the ‘Testing view’ under ‘Run tests’ (Test center group), will actually show ‘n’ number of the same test case (‘n’ being the number of configurations that the test case is associated with), each with its configuration, so that the tester can easily run the test that he is interested in (depending on his environment):

Thanks,
David Gorena Elizondo

The View Toolbar

You might have noticed above many grids there are two distinct toolbars. The primary one has options for interacting with items in the list. The other one that is right-aligned is for managing the view. Here are some examples:

1. Verify Bugs – change which query is used: bugs assigned to me, bugs created by me, or a custom query definable by you. Also an Edit button to modify the custom query.

2.  Analyze Test Runs – a toggle button for whether or not to show manual runs, and a date filter to only show recent items

3. Recommended Tests – not the same style as the other two, but changes the entire toolbar and grid out so you can see either recommended tests (tests impacted by code changes in each build in the selection) or the related work items (from all the builds in the selection)

Cheers,

David

Engineering Lead, MTLM

Beta2 online help

Our doc team (we call them the User Education or the UE team) has been hard at work supplementing the product with great deal lot of additional documentation to help you understand the various features in the product and to best leverage them. The table below is a quick way to get to some of the more important links. Feel free to browse around and if you find the information useful or not let us know what you think.

Thanks,

Ram Cherala, Principal Program Manager, Visual Studio Test Tools

Artifact Updates

One thing you may not notice, but would if it didn’t work correctly, is what happens when there are two representations of the same item in different places and when one of the two instances changes. One example is opening a manager activity which lists many items:

Open the item, and modify a property:

If you switched back to the manager view at this point, you’ll see the item’s title has not change. This is because these are different instances. The editor does not share the same info as the item in the list. The item has unsaved changes, and those changes will not be visible by anyone else, including the manager view, until the artifact is saved.

So next we click Save and Close on the test case, which will return us to the Test Case Manager. Without refreshing the list manually, the item will update and the updated title will be visible in the list:

This artifact update works only on the local client. Other clients are not notified of changes. Just like in Team Explorer, other clients must refresh their own view to see changes by others. Sometimes one can get into a situation where two different people are trying to update an item. If they both open up the item at the same revision, make changes, and try to save, the second person will get an error. This is to prevent the second person from overwriting the first person’s committed changes.

Cheers,

David Williamson

Engineering Lead, MTLM

Terry Clancy Posts About Partner Integration Opportunities

Terry Clancy, a Business Development Manager with Microsoft, has posted about opportunities for partners. It is also a good overview of the features across Team Test and Team Lab, how things fit together, and includes plenty of screenshot eye candy.

http://blogs.msdn.com/terryclancy/archive/2009/11/18/visual-studio-2010-test-tools-partner-integration-opportunities.aspx

Writing Queries for Custom Work Items and Work Item Categories

I talked about customizing work items and how this can be used with categories to get the best out of MTLM before, now I will expand on that topic and show you how to compose queries to include this extra level of information.

If you go to the Queries activity in MTLM (under the Tracking menu) and select the out of the box query Team Queries->My Bugs, you will see that these queries are constructed with the clause Work Item Type = Bug.

This will only show work items of the specific type Bug, and we want to show all of the different work items which are part of the Bugs Category. To do this, you need to modify your query to change the clause “Work Item Type = Bug” to use “Work Item Type In Group Bug Category”.

Notice how the different bug work item types show up in the results.

Thanks,
Guillermo Serrato

Planning vs Testing (Part 2)

In a previous blog post, I mentioned some of the reasons why the Planning and Testing views in MTLM look very similar but behave differently. Today, in this post, I will continue talking about this.

As you can see when playing around with the product, activity pages contain toolbars, with buttons which invoke specific actions. There are certain actions that are needed for planning purposes, and others –very different for execution. Since we have limited space in these toolbars (and because planning actions are very different from execution actions), we decided to have two separate views.

Look at the toolbars in the Planning view:

We have the following actions:

• New – to create static or query-based suites
• Add requirements – to add existing requirements
• Copy test suites – to copy test suites from other existing test plans
• Delete – To delete existing suites
• Open – to open an existing test case
• Add – to add one or multiple test cases to any given suite
• New – to create a new test case and add it to a suite
• Copy test case – to make a copy of a selected test case
• Remove – to remove a test case from the suite
• Assign – to assign testers to test cases in the suite
• Configurations – to assign one or more configurations to the test cases in the suite
• Order – to give test cases an order of execution

Now compare these planning actions to those actions in toolbars that live in the Testing view:

• Run – to run all active tests in a given suite
• Filter – to filter whatever is shown in this activity (based on ‘tester’ and/or ‘configurations’)
• Run/Run with options – to run one or more test cases in a suite
• View results – to view the latest results of a selected test case
• Open test case – to open the selected test case
• Block – to block the selected test case
• Reset – to reset the selected test case to an active state

As you can see, actions that can be taken on suites/test cases in each view are very different. Many of these actions will also affect how test cases appear in each view, and I will be talking about this in a later post.

Thanks,
David Gorena Elizondo

Zen and the Art of Test Case Composition

I’ve posted an article on my MSDN blog about strategies and considerations when composing your test cases.  If you’re familiar with Test Case Authoring in MTLM (Ryan Vogrinec and David R. Williamson both have helpful posts on the subject), you can read more about the possibilities and tradeoffs in Test Case factoring at the following link:

http://blogs.msdn.com/joemu/archive/2009/11/16/zen-and-the-art-of-test-case-composition.aspx

Enjoy,

Joe Allan Muharsky

Copy/Paste Test Steps

Test steps are very important in manual testing because they provide the detailed semantics on how each test case is to be validated and ultimately completed.  That said, if you consider most manual test cases will have several steps each and most organizations will have a number of test cases, this results in a large number of steps to manage.  Copying and pasting from popular tools like Microsoft Excel and Notepad reduces the time needed to properly manage, compose, and translate steps. 

The test steps control inside the test case work item supports bulk copying and pasting of data from a variety of data sources, including other test steps, Microsoft Excel, Notepad, or any other tool that is capable of copying tab-delimited data onto the clipboard. 

Copying steps from other data sources

Copying steps from Microsoft Excel requires that you use only two columns in your table of steps: the first column will map to "Action” and the second column will map “Expected Result” in the test steps work item control.  Copy the range of two-column rows that you want to insert to the clipboard and paste them using either the context menu or keyboard (Ctrl-V):

You can also copy steps from the test steps control and paste them into either other test cases, or other data sources.  Select a range of rows in the test steps control and either copy with the keyboard (CTRL-C), or use the context menu to copy the steps.  Notice they are rendered in notepad as readable TAB-delimited rows:

Pasting the text above into a new test case provides you the same user experience as when working with Microsoft Excel:

Cheers,

Ryan Vogrinec

Tell Us What You Think

We love to hear your feedback and given that our final beta for Visual Studio 2010 is now available this is your best opportunity to influence the product. In Microsoft Test and Lab Manager(MTLM) we are using the Microsoft Customer Experience Improvement Program (CEIP) to gather application usage data. CEIP collects information about how our customers use Microsoft programs and contains no personally identifiable information. We use this information to improve the products and features customers use most often and to help solve problems.

Participation in the program is voluntary, and for beta release we automatically enable sending this data. Click Home –> Options

It has been a few weeks since folks have started using beta 2 and we are already seeing some good information come thru. Following are some of the metrics that we are collecting:

1) MTLM Usage

a. Most used views (~1000 launches): Run Tests, Plan Contents, Workitem Editor and Test Runner

b. Less frequently used views (~200 launches): Manage VM Template, SaveAs VM Template, Environment Editor, Recommended Tests

c. Views not launched at all: Configuration Editor, About and Settings dialog

2) Actions performed

a. Most folks navigate Back and not Forward (336 vs. 22 invocations of these actions respectively)

b. Docking commands are not used frequently, of 787 invocations of Test Runner, docking behavior was changed only 8 times

c. Help has 394 invocations.

d. Home button had 300 invocations.

3) Artifact Counts (test cases, plans, runs)

a. Artifact counts are < 99 for most items (plans, runs, configurations)

b. 3% users already have between 100 - 500 test cases

c. 10% users ran queries that returned more than 100 workitems

Some of the changes we are considering based on the data are mentioned below.

We saw that Test Plan Manager was launched 766 times but users opened Test Plan Editor from there only 90 times. Could this mean Test Plan Manager may not make sense as the first view in Organize group? The other views on the Organize group in order of preference were: Manage Configurations, Manage Test Cases, Manage S3s so should we elevate one of these to the first view?

Users find  the Go To Work Item functionality in TFC very useful so we provided the CTRL+G shortcut in MTLM too. But this has been invoked only once! We need to consider how we can make this more discoverable.

As time progresses we see more usage patterns and derive useful information to make MTLM more user friendly, but in the meantime your direct feedback is the best way for use to know what you think!

Regards,

Abhishek Lal

Create/Edit Test Settings in Test Plan Properties

In manual or automated testing, you will probably want to specify your own test settings when running tests. The test settings associated with the test plan will be used by default when you click Run button in Run Tests.

There are two ways to create or edit a test settings associated with the test plan. The first option, you can open Lab Center | Test Settings. In the Test Settings Manager, you can click New or Open a selected test settings. Then you can specify the test settings in Test Plan Properties.

The second option is much easier. You can directly create a new test settings from the Test Settings dropdown in Test Plan Properties. Click the <New…> dropdown item.

The test settings editor will be opened inside of the Test Plan Properties. Follow the wizard and fill in all the fields that you care and click Finish.

Now in the Test Plan Properties, you will see your newly created test settings automatically selected.

You can also open an existing test settings from the dropdown, edit and save it. This will be done within Test Plan Properties as well.

Thanks,

Christine Zhao

Clearing MTLM’s Settings

With MTLM, we store our settings in your user profile. Specifically the ‘Local App Data’ section of your user profile. These are simple XML files that have your settings from MTLM – window size/location, whether the Test Runner is docked or floating, the queries you’ve configured in Test Case Manager, and other fun & exciting settings.

To easily navigate to them on Windows Vista, 2008, 7, or 2008 R2, open this folder in Windows Explorer:

%localappdata%\Microsoft\TeamTest\v10.0\

Or, on XP, 2003, and 2003 R2:

%userprofile%\Local Settings\Application Data\Microsoft\TeamTest\v10.0\

In this directory you will see a number of *.config files. They will be prefixed with ‘mtlm’. The main configuration file is ‘mtlm.config’ – the other files you see are isolated files for each of the activities within MTLM. Feel free to open these files up; they’re just simple XML files with name-value pairs in them. Of course, change them at your own risk. :)

Sometimes, you might want to clear the settings that you’ve got stored. Either because you like to have things ‘clean’, or because you are trying to diagnose an issue. The simplest thing to do is to delete all the *.config files in the folder mentioned above. You should do this while MTLM is not open, since we save your settings on exit. Once you’ve done this, you’ll start MTLM in the clean pristine state you had it when you first installed it.

Yours,

Dominic Hopton

Navigating Back and Forward

In a previous post, I briefly mentioned the back and forward buttons. Today I want to talk a bit more in depth about the issues we had in adding this common and seemingly simple concept to Microsoft Test and Lab Manager. Although the general behavior of back and forward does what one wants, it may not make sense in some specific scenarios.

The back and forward paradigm is most recognizable from web browsers. I’m not 100% sure, but I think that is where the notion was first created. In the very least, that is where it became popular. From there, many applications have integrated the concept. Users demand it! If my mouse back button doesn’t do anything in an application that could support it, I get down right cranky! :)

Well it took some arguing, but we eventually got it added to MTLM, and you’ll see it in Beta 2. Most of the arguing has to do with how it would be implemented correctly. A browser is a very different beast than MTLM. Other applications I could find that supported back and forward also had significant differences in their navigation model that it was hard to find a good example to follow. We wouldn’t want to reinvent the wheel if we didn’t have to, mostly because then the behavior wouldn’t be intuitive.

The primary difference between a browser and MTLM in terms of back/forward buttons is that when you go back in a browser, the view you just left is no longer in memory. If you go forward, the browser may load the content from the cache or get a new copy from the server, but either way it will re-render the content. Any prior settings or state you had are probably lost. For instance, if you started writing some text into a field, then went back then forward, your text will probably be gone. The same is not true in MTLM. Each view is kept in memory until you close it.

So why does that matter? Well, if you go back in the navigation history a bit, and then open a new view you will orphan the former forward items. You’ve lost them from a back/forward navigation point of view. With MTLM we don’t want to lose views out of the back/forward navigation. They are still running taking up memory. You may want to get back to one of the items (e.g. a bug you opened up), having to go through all the steps you previously took to find it feels like a terrible waste of time. You could get back to it from Open Items, but only if it is an artifact. You could also get back to it by closing down every other views until the orphaned items began showing, but that is pretty destructive of your workspace.

So we resorted to inserting new views in the current position. Here is a sample walk through where a bold item is the current view:

What might seem really weird to many people is that in step 4 the forward button is still enabled and if you go forward you’ll bring the Run Tests view to the foreground. That is an experience you won’t find in the browser. However, you also won’t get the benefit of step 7 either, which is convenient.

Cheers,

David Williamson

Engineering Lead, MTLM

Working with and customizing work item categories

As this previous post explains, before Microsoft Test and Lab Manager, there was no real need to know which work items represent bugs or test cases. With VS2010, we introduced the concept of work item categories, which allows us to give a meaning to some of the general work item types we use in the product.

Why do you care about work item categories? Say you want to track information about two different types of bugs - for the purpose of this discussion we’ll use an example of upgrading from TFS 2008 to 2010 (more information about this scenario here). In this scenario, you want to keep your old bugs, but you also want to take advantage of all the new goodies you get with MTLM, like filling in your repro steps and system information field when creating or updating a bug. To do this you would create a new bug work item type, leaving the old bugs alone. You can do this easily by using witAdmin. The easiest way to do this is to start with downloading the new bug template, information about how to do this again contained in Chris Patterson’s blog.

When you are done downloading, you’ll want to open the Bug.xml file and make sure you change the name of the work item type, otherwise when importing you’ll overwrite the existing one. You can import your work item by doing:

witadmin importwitd /collection:[tfs collection] /p:[project] /f:[filename].xml

You now have two different work item types and can create & query information about both separately from VS, but to get the best experience in MTLM, you need to add the newly created bug work item type to the Bug Category. You again use witAdmin, and again, a good starting point is to export the categories:

witadmin exportcategories /collection:[tfs collection] /p:[project] /f:[filename].xml

You can modify this file by adding your work item inside the Bug Category. It will look like this:

Notice that one of the work item types has a tag DEFAULTWORKITEMTYPE while the other one has WORKITEMTYPE, this is used to specify the type of bug you want to create when doing New inside of MTLM’s Verify Bugs activity, or from the test runner, which in this case you want to change to the new bug type, so people can create bugs with all the rich information pre-filled.

Save and upload by doing:

witadmin importcategories /collection:[tfs collection] /p:[project] /f:[filename].xml

You will now see both types of bugs seamlessly in MTLM wherever queries for bugs are used!

All of this work item categories and custom work item templates has some implications on how you write queries. I’ll have more on that on a later post.

Thanks,
Guillermo Serrato