Sharing content in Microsoft Test Manager

Sometimes when you are working in Microsoft Test Manager (MTM) you want to share the artifact you are looking at, or you need someone else work on a specific artifact. This can be done by asking the user to:

1. Start MTM
2. Connect To plan 7
3. Go to Test Case 57

Or, you can share URLs!

mtm://server:8080/tfs/ProjectCollection/p:Project/testing/testcase/open?id=67

Any machine that has MTM (Test Professional, or Visual Studio Ultimate Edition) can open these links, and automatically open those items without manual intervention.

On-boarding people

A great use of this feature, is for on-boarding new people in your team. When new people arrive, or switch to your project you don’t want to spend time guiding them through getting connected. Generating a URL for a plan is simple. Click the Home button from the Testing Center, and you’ll see this:

Clicking ‘Copy URL for plan’ will place a MTM URL on the clipboard, ready to email out.

One caveat, is that not all mail clients will turn this straight into a clickable URL, so in that situation just use your mail client to turn it into a real link, and paste the URL into the ‘Link to section’, and it’ll all be good.

Try this feature out, and let us know what you think.

 

Regards,

Dominic, Senior Engineering Lead, VS Team Test.

Details

MTM supports a number of different URL formats for opening, and performing operations:

• Open Test results
• Open Test Cases
• Open Test Runs
• Open Test Plans
• Connect to test plans

This all happens through two URL protocols:

• mtm:// (for http connections to tfs)
• mtms:// (For https connections to tfs)

All URLs take the form:
mtm://<server name>:<port>/<tfs vdir>/<Collection name>/p:<project name>/<center group>/<group specific>

So, for example, to open test plan 123, on testServer, with the DefaultCollection collection, in the Woodgrove project:

mtm://testServer:8080/tfs/DefaultCollection/p:Woodgrove/testing/testplan/open?id=123

This will invoke the installed MTM, connect to the server & project, and then open the specified test plan.

URL formats
It’s assumed in all these examples that the connection is to the testServer server, DefaultCollection project collection, and the Woodgrove project

Test Cases

mtm://testServer/tfs/DefaultCollection/p:Woodgrove/testing/testcase/open?id=<ID>

For these this item, you can actually pass any type of work item ID here, not just test cases

Test Plans

Mtm://testServer/tfs/DefaultCollection/p:Woodgrove/testing/testplan/open?id=<ID>

This opens the test plan.

Mtm://testServer/tfs/DefaultCollection/p:Woodgrove/testing/testplan/connect?id=<ID>

Connects the specified plan, making it the active test plan.

Test Runs

Mtm://testServer/tfs/DefaultCollection/p:Woodgrove/testing/testrun/open?id=<ID>

Opens the test run supplied

Test Results

Mtm://testServer/tfs/DefaultCollection/p:Woodgrove/testing/testresult/open?id=<runrelative ID>?runid=<run id>

Note that to get the run-relative ID can only be obtained by looking at the test result using the API. It’s not displayed in the UI at any point.

What’s New for Testing Tools in the RC?

Those of you who have played around with the Beta2 release of VS2010 may be wondering what new pieces of functionality we’ve added to the testing tools for the RC release (planned for the week of Feb 8) in response to all of the wonderful feedback.  Here is a table summarizing the key changes, with links to more details and screenshots below.

Microsoft Test Manager / Test Runner

Name changed from “Microsoft Test and Lab Manager 2010” to “Microsoft Test Manager 2010” Performance enhancements help you work faster Create tasks, requirements, and any other type of work item in your team project Create link-based queries Create and share links to open your test plans in MTM Add external links to your test plan Customize your bug queries Use keyboard shortcuts to quickly navigate through MTM Improved experience for saving a test-in-progress Get a warning if you forget to mark a validation step File bugs while performing exploratory testing

Coded UI Testing

Specialized classes (e.g., HtmEdit) are now part of the product binaries instead of the UserControls.cs file generated earlier Greater maintainability for CUIT via support for Multiple UI Maps per test project Resilient cross-platform support for using Windows Start –> Search to launch your app under test WPF DateTimePicker/MonthCalendar controls are now supported Enhanced SharePoint support Enhanced experience for generating code from manual test case TFS Work Item Picker has been enhanced to filter on valid test cases Existing Plugins can be extended to create new plugins Waiters APIs have been added to support Waiters in playback.  Playback.EncryptText API has been added to enable password encryption Extensibility APIs have enhanced usability and richer documentation Please note we will provide an automatic upgrade script to move your automation from Beta2 to RC – watch this team blog for more information.

Unit Testing

Faster remote connection in Visual Studio or MTM via IPv6 Overall performance improvements to loading/executing tests in Visual Studio Breaking changes to Data Diagnostic Adapter API

PowerTools

TCM Migrator (Excel) Tool – Will be updated for use with RC TCM Migrator (MHT) Tool – New for RC! Test Scribe – will be updated for use with RC

Read on for more information and screenshots on these new features…

 

Microsoft Test Manager / Test Runner

We’ve added a number of exciting new features to the TCM client.

New name – We’ve changed the name of the TCM client to “Microsoft Test Manager 2010” (MTM).  This was in response to feedback about confusion between VS lab management and MTLM.  Note that this is a change in name only – the MTM client continues to include the Lab Center view.  Look for the new name in your Start Menu:

 

 

Better performance – With the RC release you should experience noticeable performance improvements in the MTM across most scenarios, and particularly in the following areas:

• Connecting to remote machines via IPv6
• Testing in remote server scenarios
• Transitioning between MTM and the Test Runner (loading tests, “Save and Close” and “Close” operations)

 

Create tasks, requirements, or any other type of work item that exists in your team project using the “New <work item>” dropdown in MTM.  (Formerly, MTM limited you to creating bugs, test cases, and shared steps only.)  You’ll find the “New” dropdown on the MTM chrome, adjacent to the “Open Items” menu:

 

 

Create link-based queries from MTM, just as you can from VS Team Explorer.  The “New query” control is now a dropdown that lets you harness the power of tree queries and direct-links queries:

 

 

Copy and share your test plan URL – Now you can create and share links for opening the MTM application and connecting to specific test plan.  Simply click on the “Copy URL for plan” button on the MTM Home page to copy a plan’s URL to the clipboard, then send that link to co-workers and/or use it to create shortcuts in your browser or on your desktop.  When a user clicks the link – voila! – MTM will launch (if not already running) and open to show the specified plan.

 

Add external links to your test plan to documents, resources, or even a team website.  You’ll find the new “Links” section on the test plan Properties activity:

 

Customize your bug queries to better meet your needs.  Formerly, in the Beta2 bits, the “Assigned to me” and “Created by me” queries were hardcoded to exclude bugs that had been closed more than one day earlier.  Now you can tailor those queries to better suit your working style.  Look for the “Filter” drop down in the Verify Bugs activity:

 

Keyboard shortcuts are now available to help you navigate more quickly around MTM:

Ctrl + ~	Activity Center menu
Ctrl + {1, 2, 3, … 0}	Activities
Ctrl + Shift + {1, 2, 3, … 0}	Actions

 

Saving from Test Runner is more flexible – Now you can click ‘Save’ in midflight – while running a test – then continue running with ability to perform all possible operations such as filing bugs, adding comments, etc.  (Formerly, in Beta2, the whole test would be disabled after a save operation.)

 

Validation guidance – Marking a validation step as pass/fail is important when you are creating an action recording, because it creates a recording boundary where you can later perform a manual validation during playback.  In the RC release, you will see a warning if you forget to mark any validation steps:

 

File bugs in exploratory test mode – While doing exploratory testing, you may want to file a bug for the last few minutes of testing rather than the complete session, as the information recorded earlier may not be relevant to this bug.  In the RC, you can using the new “Create exploratory bug” option and specify the time range for retaining information collected in the event log and action log, attachments, etc.

 

 

Coded UI Testing

Here’s what’s new for CUIT.

Specialized classes (HtmEdit, WinWindow, WpfButton etc) have been added to the product.  (Previously these classes were auto-generated in the UserControls file. This file will no longer be added to the Coded UI Test Project.)

Please note that this is a breaking change.  We will make available an automated tool for upgrading projects created in Beta2 to RC.   Watch this team blog for more information.

Multiple UI Maps are now supported. Previously a Coded UI Test Project could have only one UI Map file. Now we allow addition of multiple UI Maps to enable test projects to scale up to real world applications. Also, this will enable multiple UI automation engineers to work simultaneously on the test project without running into merge conflicts.

Start -> Search action filters – If you launch the application under test by typing its name in the Start -> Search box, the actions will be aggregated to an ApplicationUnderTest.Launch action.  This greatly enhances the resilience of the test and you can now record this on Win7 or Vista and playback your recording on XP or Win2K3.

WPF DateTimePicker/MonthCalendar controls are now supported. 

SharePoint Support has been enhanced – SharePoint 2007 is now fully supported except where it interacts with Office applications like Excel/InfoPath.

TFS Work Item Picker has been enhanced to allow selection of only valid test cases.  When you choose a manual test case to generate code from, the experience now lets you look at test case work item types and filter on those that have automation strip associated with them. 

Waiters – New APIs have been added to support Waiters in playback.  These help your test code be faster and more resilient by getting rid of the sleep APIs in the test code where you don’t know how long you need to wait for a control.

Playback.EncryptText API has been added to enable users to encrypt text that can be set on a password field

Extensibility APIs have enhanced usability and richer documentation.

New Plugins can now be created by extending existing Plugins. 

 

Unit Testing

Here are the key improvements on the unit testing front.

Faster remote connection thanks to support for connecting via IPv6.

Overall performance improvement in loading and executing tests in Visual Studio.

Breaking changes to the Data Diagnostic Adapter API – An API documentation update will be made available on the RC release timeframe.

 

PowerTools

On the RC release timeframe we will be refreshing the Beta2 PowerTools, as well as adding new functionality to the Test Case Migrator tool.

Test Case Migrator (Excel) Tool – This tool enables importing test cases expressed in XL format into VSTS2010.  The tool was originally released for use with Beta2, and will be updated to work with the RC bits.  Watch for updates on CodePlex: http://tcmimport.codeplex.com/documentation.

 

Test Case Migrator (MHT) Tool – New for RC! The Test Case Migrator tool has been enhanced to support importing legacy tests stored in *.mht format into VSTS2010. With the new drop, the Test Case Migrator tool will support import of Excel and MHT formats. Watch for updates on codeplex: http://tcmimport.codeplex.com/

 

Test Scribe Tool – This tool enables you to export your MTM test plan data to the Word document format.  Also originally released for Beta2, it will be updated to work with the RC bits.  (Link to release point on MSDN not yet available as of the time of this writing.)

 

 

As you can see, we’ve been quite busy responding to customer feedback and adding finishing touches to the product.  On behalf of my team I offer sincere thanks to those of you who have experimented with the earlier betas and provided us with invaluable feedback.  We encourage your ongoing participation.  Please download the RC when it becomes available next week, try out these new features, and let us know what you think.

Thanks and Regards,

Michael Rigler

Sr. Program Manager, Visual Studio Test Tools

Configure search properties used by recorder\code generation?

Note: Cross posted from Gautam Goenka (MSFT).
Permalink

One question that keeps coming up is “How to configure search properties used by the recorder\code generation for identifying the control?”. For example, inform recorder that the Name of the certain control is dynamic and not to use it to identify the control....

Note: Cross posted from Gautam Goenka (MSFT).
Permalink

Generation of Private Accessors (Publicize) and Code Generation for Visual Studio 2010

We are letting you know of a few things that are happening in Visual Studio 2010 in regards to generation of Private Accesors and Code Generation. 

What is Publicize in Visual Studio testing?

Publicize is the ability to take internal application programming interfaces (API) and create public counterpart API that you can call in your tests, which would in turn, call into the internal APIs of your product.  This comes in the form of a stand alone executable, or can be seen when using code generation to create test method stubs of internal or private APIs.

What is Code Generation in Visual Studio testing?

Code Generation allows users to create test stubs and generates a little code snippet inside that stub.  This feature can be used in conjunction with Publicize to create test method stubs from internal APIs.

What is happening in Visual Studio 2010?

We have stopped working on these features for Visual Studio 2010 and may remove them from the product in following releases.  This is due to the following reasons:

1. Lack of resources and time: The focus for this release has been to improve the experience for manual testers, so the priority for the code generation and publicize features has been lowered.  There have also been other issues with the publicize functionality that we utilize that have not been addressed.
2. New features by Language teams:  As the language teams have made modifications to their project types and languages, we have been unable to respond to the changes they have made and have not been able to work with the new features they have introduced.

What is available for me then?

For those who wish to continue testing internal APIs, you have three options:

1. Use the Microsoft.VisualStudio.TestTools.UnitTesting.PrivateObject class to assist in accessing internal and private APIs in your code.  This is found in the Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll assembly.
2. Create a reflection framework that would be able to reflect off your code to access internal or private APIs.
3. If the code you are trying to access is internal, you may be able to access your APIs using the InternalsVisibleToAttribute so your test code can have access to the internal APIs.

However, there is not any good replacement for Code Generation for the new features added by the lanugage teams.  You may create the TestMethod stubs and then remove the internal code.  You only need to keep the stub itself.

Will it be added back in or fixed in future releases?

At the moment, there are no plans to do this.  It may change based on priority and focus. Feedback welcome about this content or other items you would like us to consider for future releases.

Thanks

Bruce Taimana
Program Manager

Understanding the Window Search and Windowed Properties

Note: Cross posted from Siddhartha Pandey - MSFT.
Permalink

There has been questions regarding how do you search a particular control giving ControlName or ControlId etc. So I thought it is a good opportunity to blog about it. I will be covering the following :-

a) Which plugin gets picked up for which controls

b) Introducing MSAA plugin

c)  Introducing what does it mean that a control is windowed.

d) Properties specific to Windowed control.

e) How to do Windowed based search and how to search a particular non windowed control using a windowed properties.

Plugins currently picked up (Beta2)

Below is the info in which plugins gets picked up for which controls

1) IEPlugin (web) :- This is picked up for web controls inside IE 7/8.

2) UIA:- This is picked up for WPF controls.

3) MSAA (Default plugin):- This is picked for Winforms Controls, Win32 controls, MFC applications. In short which ever controls that are not specified in above two plugins gets picked up by this plugin which is a default plugin. Thing to note here is that if for some platform we are sure that default plugin wont work (for eg. FireFox, SIlverlight etc) this plugin also doesn’t get picked up and we throw exception.

 

Introducing MSAA plugin

MSAA plugin uses “Micorsoft Active Accessibility” to identify controls. So if a control has a good Accessibility support then it could be identified by the plugin (if there is no other plugin that support that control this plugin is hit. To see how to write your own plugin for the platform that are not supported please go through this.)

Internally the plugin generates search properties/exposes properties given by interface IAccessible. For richer properties for Winforms/Win32 controls it uses Windows messages to get those properties. Also some of the SearchProperties are windowed specific properties. In the next couple of topic we will be seeing what does a windowed control means and what are the specific properties that this plugin generate.

 

What are windowed controls?

By Windowed controls we mean all the controls who have their own window handle. You can identify if a particular control has a window handle of it’s own by using SPY++. Below are the steps to check if a control has a window handle of it’s own or not.

1) Open SPY++.

2) Press Ctrl+F.

3) Use the Cross hair and drag it to task bar button of any application that is opened. You will notice that the whole task bar is getting highlighted instead of the buttons. This is because it uses a window handle from point and highlights it. In this case the buttons don’t have their own window handle instead the whole task bar’s window. This means all the these buttons inherits the window handle of the parent task bar window control.

Note: Not all controls with ControlType = Window are windowed and also even a control which is of ControlType other than Window can also be a Windowed control (eg the hosting control inside the Internet Explorer who’s controltype is a client)

Properties of Windowed Control

Some the properties that a windowed control can have are :-

1) WinProperties.Common.ControlName (The ControlName which is got by Sending WM_GETCONTROLNAME message to the window handle of the control)

2) WinProperties.Common.ControlId (Got using GWL_ID message)

3) WinProperties.Common.ClassName (Got using GetClassName Pinvoke)

4) WindowText (Got using GetWindowText PInvoke)

5) AccessibleName (Got using IAccessible pointer for the control)

All of these properties are got using window handle of the control and hence these are specific to windowed controls. The last property though (as the name suggest is not a property got using window handle but using IAccessible pointer.

How is Name and AccessibleName different?

With Respect to the MSAA plugin Name is IAccessible name for a non windowed control. In case of Windowed controls the name Property returns WindowText and AccessibleName returns IAccessible Name. For Non Windowed controls AccessibleName property should not return anything and you will get IAccessible name using Name property as mentioned before.

 

Searching of a control

You can do two type of Search using MSAA plugin :-

1) Window based search :- This is done by navigating through all the window either inside Desktop or inside a specified Window handle.

2) MSAA search :- This uses IAccessible based navigation to search a particular control.

 

Search a Window Control using Window Search

If you want to search a window control you can do that by specifying atleast one Window Property (as mentioned above)

So you can do this to search a control who’s ControlName is “foo” and who’s controltype is window

WinWindow window = new WinWindow (containerelement);
window.SearchProperties.Add(WinProperties.Common.ControlName, "foo");

Internally the plugin checks if there is even a single window property mentioned in SearchProperty. Seeing that it does a Window based search.

Note: Following are the properties you can mention as part of SearchProperties so that a window based search kicks in.

WinProperties.Common.ControlName

WinProperties.Common.ControlId

WinProperties.Common.ClassName

“AccessibleName”

 

Searching a Non Windowed control using Windowed property

Let’s take an example of a Simple Windows Form Application which has a button inside it. I want to search the button if I know it’s parent window’s ControlName (let’s say “foo”) and don’t want to search using button’s name property. Below code will show how to search a button using it’s window’s controlname property

WinWindow window = new WinWindow (topLevelElement);

// Assign a ControlName property to the window control so that a window search kicks in.
window.SearchProperties.Add(WinProperties.Common.ControlName, "button1");

// Now you search a button control without specifying any property here

WinButton button = new WinButton (window);
Mouse.Click(button);

Let’s understand the code. The window instance is having a Windowed SearchProperty assigned to it. Hence it will hit a window search which matches the control name property before returning the element.

Now we have assigned the Container element as window of the button control (see the ctor of the button control). So internally a search for the window will happen first using window search  and then button will be searched under the window control using MSAA based search.

Note: Cross posted from Siddhartha Pandey - MSFT.
Permalink

Recording of MSDN Webcast on "Extending the Coded UI Test"

Folks can use the following link to view the recording of the MSDN web cast on "Extending the Coded UI Test".

Cross posted from Siddhartha Pandey blog

Wanna know about VS2010 ALM in 5 mins?

The VSTS Rangers have come up with this great set of content on VS2010 ALM that is basically intended to answer the q “Tell me about VS2010 ALM in 5 mins”.

The Visual Studio 2010 Quick Reference Guidance consists of compact cheat sheets for Team Foundation Server (TFS) 2010 and Visual Studio (VS) 2010, addressing folks that are unaware of Visual Studio and Team Foundation Server capabilities or have little time to invest in detailed education.

The artifacts include an overview document and poster that allows you to quickly focus on individual areas like testing, by providing crisp and compact guidance sheets and quick reference posters. You can take these to your 5-min coffee break discussions or use them as a stepping stone to the more detailed and in-depth guidance you will find on MSDN.

Check out the testing posters here – I loved them! :)

Cross posted from Musings of a tester testing test tools…

Microsoft Test Runner series – part 7 – Exploratory Testing

Hi Again,

So far we have talked about how features & benefits of MTLM when used for the tests that are predefined. E.g. creating actionable bugs, using fast forwarding & so on.

Many times testers test the application in ways that are not predefined (i.e. no fixed flow of steps is prescribed by the author). Some customers feel that this way many new & interesting bugs are found. Bug bashes done by various teams are one of the best ways to find new bugs.

However, there is one pain point that testers feel while performing exploratory tests in this way: it is very hard to remember what they did to cause the bug. Many times the bugs are not reproducible & developers reject the bug as non repro.

When using MTLM, testers can still produce actionable bugs while doing such testing. Here is how:

Create a dummy test case lets’s say “Exploratory test” with one step “Explore the app”. When you load this test in MTR, it shows you the test with one step. Start performing actions on the application as you would while performing any other exploratory test. When you encounter a bug, click on the create a bug & you would have the same actionable bug. i.e. the bug will have all the detailed actions you performed, the video of the recording, system information, intellitrace & any other logs you have configured to collect in test settings. If you had associated any attachments during the test, they are will show up in the bug.

So this time when developer gets a bug, he will be able to get a complete picture of what went in the system that caused the bug.

While creating the bug, you also have the option to select the timeline for which for want the information (e.g. if you want attachments that were added only in last “n” mins of testing, you could do that by choosing to create exploratory bug).

Many times, when a bug is found in this way, testers want to create a test case, so that the issue can be verified later or the test should be added to be part of regular test cycle.

To achieve this MTLM provides a way to make it easy to create a test case using the information in the bug. Choose a bug in MTLM & click on create test case from bug.

A new test case is created & the actions in the action log collected from the bug are converted to the steps. You can edit this newly created test case e.g. delete/add some steps. Name this test case & save it. You have created a new test case to regress the bug easily. The newly created test case is also automatically associated with the bug.

Tip: Some customers even use this functionality to explore the app & then use the steps collected in the action log (from Bug) to create new test cases to start the project.

Whichever way you want to use the functionality, go ahead & explore your app. Happy testing.

Other links:

- Video ALM 7 - Exploratory testing and creating a test case from Bug

 

Thanks

Suresh Budhiraja

Sr. Program Manager

Microsoft Test Runner series – part 6 - Shared Steps

Hi Again,

Let’s talk about an important concept in test case design – reusability. As testers you know that there are few steps that may be common in many test cases e.g. steps to log into the application may be common to multiple test cases. In other cases the cleanup steps (or any other steps in between) can be common. These steps have to be authored as part of these test cases. If something changes in these common steps, (e.g. steps to login have now changed) many tests have to be altered.

MTLM uses the concept of shared steps to make it easy to handle such scenarios. Steps that are common (or shared) between multiple test cases can be written as shared steps. Shared steps look like a mini test case & can be imported into any test case at any point (not necessarily at the start or end of test case). When imported into a test case, while executing that test, MTR shows the steps as any other steps in the test. The steps can be action step Or can be validate steps (with expected results).

Sometimes, author does not anticipate shared steps & realizes later that these steps are being used in many places. In these scenarios, MTLM makes is possible to select a number of steps of a test case & convert into a shared step (using context menu). A new shared step is created & saved in the repository. From now on, any other test case can reference it.

Next natural question is what happens to shared steps in a fast forwarding scenario: When tester is running the test (& have selected to record actions), the actions of the shared steps are also saved. The extra advantage is that the recorded actions are associated with the shared steps. Now the shared steps has associated action recording. User can fast forward the actions of the shared steps at any time. Not just in the test case it was recorded, but also in any other test case where the shared step is being referenced.

When the steps of the shared step changes (e.g. for login), just re-record actions for the shared step in one test case & all other test cases automatically get the benefit.

For some users, it seems more convenient to use the concept of shared steps as a component library. MTLM not only allows shared steps to be authored, just like a mini test case, users can also create action recording for a shared step without adding it to a test case first. This can be achieved by going to the shared step tab in MTLM & choosing to create actions recording for a shared step. It launches MTR in a special mode for this purpose only.

Note that shared steps cannot be executed by itself & does not have a result. They have to be executed as part of a test case.

Shared steps can also have parameters just like any other step. However, as they cannot be executed by themselves, so they cannot have associated data for testing. When imported into a test case, the test case automatically shows extra columns for parameters used in the shared steps. This way, different test cases can be authored that supply different data sets to be used in the common steps.

Please note that although shared steps can’t be executed & need any associated data, adding one row of data is possible with shared step. You may want to provide this single row of data if you plan to create action recording for the test case without adding it to any test case (i.e. recording shared steps in standalone mode of MTR as described earlier). This allows the tester to create action recording with correct binding between the parameters & the controls on the application. Other than that there is no use of this single row of data & is ignored when shared step is added to the test case.

So go ahead, divide your test cases in reusable sections & reuse these shared steps to gain flexibility & productivity.

Other links:

- Using Shared steps

- Video ALM 5 - Author test and shared steps

 

Thanks

Suresh Budhiraja

Sr. Program Manager

Microsoft Test Runner series – part 5 – Data Driving

Hi again,

Today we will talk about data driving a test. As is very clear to all testers, if a test requires some data input in AUT, it should be tested with different data sets (e.g. verify that the application works with boundary conditions, wrong data, etc…). Different teams use different mechanism to represent the data set relationships.

In MTLM, a test case can have data sets associated with it. While authoring a test, user can add parameters in the step (a parameter name would start with @). Associated data sets can be defined in the test. Each data set will become an iteration of the test when loaded in MTR.

Data driving manual tests: When tester is testing the first iteration, MTR, not only shows the step, but also shows the data that should be used while testing this iteration. When next iteration is loaded, the steps remain the same; however data shown is for the next data set. This way tester can know that they have tested the test case with all the data sets.

Parameters can not only be used in test step (instruction), but can also be used in expected result.

When tester selects a step that has some parameters in it, the step shows the data for each parameter. It selects the first parameter by default (shown by an arrow next to it). It also copies the data associated with this parameter in the clipboard. When tester is in the control where data needs to be entered, she can simply paste the data & move to the next control. If the tester had selected to create action recording, the tool figures out that the data has been used & selects the next parameter in the step, also copies this data on the clipboard.

Hence it is possible to fill a long form quickly, by simply doing CTRL V, Tab, CTRL V, CTRL V Tab…. Wow that makes the data entry so easy J.

If in some case tester wants to enter a different parameter value in the control (may be the sequence of control was changed in the UI), user can also select the parameter by clicking on it & that parameter becomes the current parameter (& hence the value is copied to the clipboard).

Fast forwarding a data driven test: As we discussed in previous blog, fast forwarding helps testers quickly reach the step that they are interested in paying more attention. We also know that the fast forwarding plays back the same actions that were recorded earlier. So how would data driving scenario work with fast forwarding?

When tester tests a test case manually & reaches a step where data entry is required, as we saw earlier, tool shows the data that tester is suppose to enter. Tester can enter this data by typing it in the control Or by pasting is (as the data is already available on the clipboard). In Either case, the recorder is intelligent enough to recognize that the data has been used. If the data typed or pasted matches the data that parameter had, it makes a relationship between the parameter & the control. Hence the recording is not a simple action recording any more, it actually records a parameterized action.

During fast forwarding when playback reaches this parameterized action, it automatically picks up the data from the current iteration & pastes it in the control. So it is possible for tester to record actions while testing with a data set & fast forward using a different data set without doing any extra effort.

Cool huh…

Try it now, select first iteration, perform manual testing. Finish this test case. Select next iteration, play the steps, watch it using the data from second iteration. It will make the testing easy, just record the first iteration, use “Play All” for second iteration, third iteration, fourth,……

Helpful tips:

- For binding to happen correctly, make sure that you select the step, before typing the data in the AUT

- Correct the bindings if needed, before you mark the step result (the bound icon will show up next to the parameter to tell that that the control has been successfully bound).

- Choose an iteration that has data for all the parameters while recording; this will make sure you get binding for all parameters.

- While recording make sure to set the value in the control that needs to be bound (if the data is already present in the control, please delete it & paste again).

Other links:

- Various videos referred in previous blogs shows how to author a data driven test & use it.

 

Thanks

Suresh Budhiraja

Sr. Program Manager

Microsoft Test Runner series – part 4 – Fast Forwarding

Hi again,

In one of the previous blogs we had talked that to reduce the time spend on less important tasks, MTLM provides a functionality called Fast Forwarding. The goal of this feature was to allow testers to quickly go to the step in the test case where they want to focus their testing during this test execution. It may be while making sure that the bug is real Or it may be during regression testing etc.

History: So how did it came into existence: When we went on customer visits, I noticed that a tester who just found a bug. However, before deciding to file the bug, he actually performed ALL the steps again (couple of times) to make sure if it a real bug. When asked for more details, he mentioned that although first few steps were not important for this bug, he had to perform these to reach this point. Note that the first few steps had a lot of data enter & hence it was taking time. Another tester did the same thing while verifying a bug that was fixed in last night’s build.

Talking to various testers we found that there are many scenarios, where performing certain steps are necessary to reach the point in test case where they want to focus their energy. They wished if they had magic wand to reach the step they wanted to test more thoroughly. When asked, why can’t you automate these sections that are not important but are necessary, the answers we got were:

- It is not easy to automate the tests/portions require skill, tools & time

- The UI keeps changing very often in initial phase of development & ROI is not worth.

- In many cases, it is not possible to plan which areas of tests should be automated e.g. for regressing a bug, tester can’t plan ahead what steps should be automated.

So we started thinking what can be done to reduce their pain. We came up with the idea of using record & playback in such a way that it can help testers in such scenarios. We knew that R&P has its own limitations; however, using it in this scenario fits very well, where a human is sitting in front of the screen. Also internally we had a technology that was being used successfully & had functionality to overcome basic issues with R&P.

Fast forwarding: The idea - When tester is testing a test case manually, record UI actions in the background (no extra overhead for the tester). One of the primary goals was to keep the manual testing as is i.e. manual tester should not be asked to do something different for creating the action recording. However we don’t want to produce one big recording for the whole test case, so it was decided that we find a way to sync what tester is doing on AUT & what manual test step is being performed. The idea of marker segments came into existence. i.e. as soon as tester marks a step result (pass/fail), the tool would know that testing this step is complete & hence create a boundary for the recorded actions. So more steps are marked by the manual tester the granular the action recording boundaries become.

When test is completed, the action recording just created would be attached to the test case. When tester tests this test case again, the action recording is available for fast forwarding any steps. Now tester does not have to perform the same steps again that she does not want to (but have to). She can simply select the steps (or or the barker region) & choose to play.

The playback engine will perform the same actions that were performed while testing is manually the first time. When playback is done, manual tester can perform the steps manually. The best part is that after performing few critical manual steps, she can again choose the rest of the steps & play them for fast forwarding.

Isn’t it great to be able to fast forward whatever steps you want, whenever you want!!!.

A tester can also choose to select all steps & play (just watch the actions being performed on the AUT). I know what you are thinking, but remember, you are a manual tester; don’t overdo it (your job is to find bugs) :)

In the next blog we will talk about what happens if I want to test the same test case with different data set & how fast forwarding can help in that scenario.

Other links:

- How to remove unwanted actions from recording in manual test runner

- Creating resilient action recording using test runner

- Fast Forward Testing – Part 1

- Fast Forward testing – Part 2

- Video ALM 13 - Regress bug using Fast Forward for Manual Testing

 

Thanks

Suresh Budhiraja

Sr. Program Manager

Microsoft Test Runner series – part 3 – Filing Actionable Bugs

Hi Again,

The next topic to cover is the filing actionable bugs using the tool. However, I already have an old blog that explains this in detail:

http://blogs.msdn.com/vstsqualitytools/archive/2009/05/26/create-actionable-bugs.aspx

Other links:

- Video ALM 6 - Perform manual test and file actionable bug

 

Thanks

Suresh Budhiraja

Sr. Program Manager

Microsoft Test Runner series – part 2 – Manual Testing

Hi again,

As we discussed in last blog in this series, let’s explore the features & functionalities of Microsoft Test Runner (MTR). The component is used for execution of manual tests.

Manual tests can be authored in the MTLM by going to Test Plan activity & choosing to create a new test. User can define the steps they want to follow while testing. The granularity of the step depends on the preferences of the testers. Some testers like to keep it high level, others keep it very granular. One thing that should be kept in mind is that the granularity of fast forwarding & the status reporting will depend on the granularity of the steps. Test author can also specify the expected results. These expected results will be visible to the tester along with the step instruction while performing the tests.

Note that when author specifies an expected result, the corresponding step is marked as a validate step. You will notice a small icon representing this fact. Validate steps are displayed just like any other step in MTR. It is expected that the manual tester will verify that the AUT behaves as expected. If the tester does not mark the validate step, the step is assumed to be “Failed”.

While authoring, an author can also attach files to the test step. These are generally useful to provide extra information to the tester. E.g. the test author can attach screenshots for comparison purpose OR attach a script that tester needs to run before a step & so on … Multiple attachments can be added to a step.

If the same test needs to be tested with different data sets, parameters & corresponding data sets can also be added to a test case. We will get into more details of this in a separate blog on data driving tests.

When the tests are loaded in MTR for testing, tester can choose any test that he/she may want to test first. An overlay shows a checkbox asking if the actions should be recorded. These actions can be used for fast forwarding later.

Tester can start the test; the steps are shown. Test case level details (e.g. attachments etc.) are visible in the top section of MTR. Tester can expand the section to see more details. Also the comments can be added for the test case there.

Highlight the step that you are testing & perform the actions on AUT. It is a good idea to mark the step result as pass/fail. As discussed earlier, tester does not need to move the mouse to mark the step results. Just use the global shortcut keys to mark the step as passed & another key to move to next step. Note that the global shortcut keys can be changed in the configuration file to suite your needs. Tester can also add comments to any step if needed (these will be visible from the bug and the test results).

When a step fails, tester can mark the step as failed by using either of the following ways:

- Using the dropdown of the result icon

- Clicking on the result icon two times (as the icon cycles through the pass/fail states).

- Using the global shortcut key

On step failure, the comment section for the step is automatically expanded as it is likely that tester will add some comments on why the step is marked as failed.

Tester can add more details & file bug from MTR (we will see more details when we explore filing actionable bugs).

When the test is completed, tester can end the test result. Note that the tool auto computes the results of the test case based on the step results. Following logic is followed to compute the result:

- If any step is marked as failed the test result is marked as failed.

- If any “Validate” step is not marked, the step is assumed to have failed & hence the test case result is computed as failed.

- If no step was marked as failed & no validate step was left unmarked, the test result is computed as passed.

The result of the test case is visible on the right of the test case name in the test case section at the top. In rare circumstances, if user wants to override the result of the test case, she can change the result by clicking on the test result icon.

User can move between tests using the buttons on the test case section. The left & right buttons load the next/previous test case/iteration available. The dropdown next to the test case name shows all the test cases/iterations loaded & their results so far.

On pressing the end test case (or when explicitly pressing the save button from top toolbar), the test case result & all attachments are stored on the test server.

Tester can also resume back to the full view of MTLM at any time during execution, if the test case was in progress, it is paused. When user comes back to MTR view, the test can be resumed.

Hopefully this gives an idea about the very basic functionality for testing a manual test case. In next blogs, we will talk about some interesting capabilities of the tool.

Other links:

- How test case result is computed in Test runner

- How can I configure MTLM to use my custom bug/test case type

- Using Global Shortcut keys

- Video ALM 4 - Create test suites and assign tests to team

- Video ALM 5 - Author test and shared steps

- Video ALM 6 - Perform manual test and file actionable bug

 

Thanks

Suresh Budhiraja

Sr. Program Manager

Microsoft Test Runner series – part 1 – History & Goals

Hi everyone,

I hope you are using the MTLM beta2 release & finding it useful. I wanted to blog a few posts & talk about the Microsoft Test Runner component of the product that is used for execution of manual tests. I will talk about how it got started & what functionalities it provides to make the life of a generalist tester better.

History: When I joined the team, our goal was to define the vision of the product. We set out to find the pain points of testers. As we went to various companies, we saw that a lot of testing is still manual. All the testers were very professional & the felt proud of finding bugs during product cycle, so that the real end users don’t have to deal with them. It was a very good experience to meet such dedicated testers.

When observed closely, we saw that a lot of their time is spent on mundane tasks e.g. adding details to a bug, verifying & re-verifying that what they found was a real bug; testing the same test case over & over again with different data sets etc. While performing the steps, many of them had the high level steps written/printed by their side. Sometimes it was in the form of a printout from a word doc or excel sheet & other times they had the soft copy open on their computer & were flipping back & forth between the steps/instructions & the Application Under Test (AUT). A lot of time was also spent on reporting what they had tested.

We decided that we will focus a lot of energy in making the life of generalist tester a bit better & productive. Our goals were to let the tester focus on finding real bugs & reduce the mundane tasks that they need to perform.

The first goal was to reduce the ping pong between developer & tester about what the bug is & is it really an issue. We wanted to create an actionable bug that provides enough information to developer for him to fix the bug & not ask for more details from tester. We automatically added information like repro steps, detailed action logs, video of the testing, system information & data collector logs etc. Together these would help the developer get to the root cause of the issue, without tester needing to spend a lot of time filling this information in the bug.

The other goal was to provide a way for tester to see the steps & perform the actions on AUT. However, it was decided that the solution should be such that the tester can focus his/her main attention on AUT & just glance at this app whenever needed (e.g. to see next step or expected results). The interaction with MTR should be minimized as every interaction with MTR is actually taking tester’s attention from AUT & from finding bugs. Various strategies were considered e.g. providing a full screen view in a browser OR provide a minimal view shown as only one row. Ultimately, we decided to go with the small vertical app that is shown on the side of the tester’s machine. This view of Microsoft Test Runner not only shows the steps information, but also let’s user do most common tasks, e.g. take screenshots, file bugs & so on. To reduce the need to move the mouse between AUT & MTR, it supports global shortcut keys for common operations such as move to next step, pass/fail a step etc.

To reduce the effort required by tester while reporting the status/results, MTR allows testers to mark the result of the step/test case, add comments etc. while performing the test. This information is uploaded to server, when users save the results. This reduces the need for the tester to spend time in filling detailed status reports etc. Lead/manager can view the results anytime & can generate various reports.

The next goal was to reduce the time it takes to test a test case. One easy way out generally is to automate the test case. However, in reality it is not that easy or straightforward. Firstly, the tester needs the knowledge & skills to create the automation. Secondly not all tests can be automated completely (not worth the ROI). Thirdly, we wanted to provide the flexibility of choosing what portions to automate vs test manually. Generally, this information can’t be planned in advance, i.e. as a tester I want the flexibility that today, I want to test one section manually & the other section I want to skip, as it is not that interested Or is pretty stable. This way I can focus my energy on what I think may have most bugs.

Accomplishing this goal was the tough one. We argued about various choices (some that are being followed in industry). After lot of thought, we decided to try something new. Use Record & playback technology for fast forwarding the sections that are not very interesting. R&P has been used earlier & has it’s own issues – i.e. may not be used in all scenarios of automation. However, in this case we were talking about fast forwarding some sections of the test case. Also, there was some advance techniques that were being used within the company & had proven to be much more stable than myths that surround R&P. As this topic is long, we will talk about more in a separate blog how the tool is using R&P to provide fast forwarding capability for testers.

The other goal was to provide more insight into what went in the builds & what is impacted by the changes made to the code. Based on previous experience, we knew that one of dilemma testers face is when a small change is made to the code before a major release. The question comes, what should be tested & what can be omitted. If major portion of testing is done manually, then it becomes more difficult question Developers always say it’s a small change & will not impact any functionality. As if the testers can believe that J. So we came up with a solution to provide test impact of the code changes that helps testers prioritize the testing.

Lastly, one of the question that every test team need to answer is “Are we ready to ship?”. The solution needed to help the team answer that question.

In next few blogs we will talk about the solutions that the tool provides in detail & how it tries to fulfill the goals we set out with.

Other links:

- Video Key Benefits of using Visual Studio Test Elements 2010 & Visual Studio Lab Management 2010

- Video  Key Benefit 2 - No More No Repro

 

Thanks

Suresh Budhiraja

Sr. Program Manager

Test Scribe: Test Plan Documentation for MTLM Plans

I'd like to announce the beta availability of Team Test's first PowerTool release for Visual Studio 2010: Test Scribe.  This tool allows users of Visual Studio 2010 Ultimate Beta 2 to generate a Word 2007-compatible Test Plan Document from their plan, suites, test cases, and other artifacts.  Using the tool is a fairly straightforward process, including:

   1. Launch the Test Scribe tool.
   2. Enter your server/collection URL (e.g. http://myserver:8080/tfs/DefaultCollection)
   3. Select a Project.
   4. Select a Test Plan (previously created in MTM).
   5. Click the Generate button.

The resulting document will contain (among other things) a list suites with test cases and steps detail and pie charts detailing the overall progress of your Test Plan.  You can see a screenshot below showing several sections of a generated document.  Feedback is welcome and appreciated, and you can find the tool download at http://visualstudiogallery.msdn.microsoft.com/en-us/d18873c7-909d-4788-a56e-0c496a1d8bb9.

Many thanks and appreciation to everyone who helped get this tool out the door.