Visual Studio Team Test

Planning vs Testing (Part 5 – Final)

VSTS Quality Tools — Wed, 25 Nov 2009 19:14:46 GMT

Today, I will finally end this series of blog posts where I’ve been talking about the main differences that you will find in Planning and Testing views (although they appear to be very similar at a first glance). You can find the previous posts here:

Double clicking (or hitting enter on a selected item) throughout MTLM will trigger the action that we thought was the most basic one for each activity.

Well, for any given test case, if you’re in the Planning view (Plan center group, under ‘Contents’), double clicking or hitting enter on a selected item will open the test case:

However, in the Testing view (Test center group, under ‘Run tests’), this same action will actually open the latest result for that given test case (assuming that there is a result; if there isn’t, nothing will happen):

Thanks,
David Gorena Elizondo

Planning vs Testing (Part 4)

VSTS Quality Tools — Tue, 24 Nov 2009 19:36:19 GMT

In previous posts, I’ve been discussing why Planning and Testing views look similar but behave differently. You can refer to them by clicking on Part 1, Part 2, or Part 3.

Sometimes, depending on the organization, planning is done by some people, and execution is done by others. And even if it is the same people who do both, it will be common to want to look only at a specific set of test cases when actually executing them. This is another big difference between both views:

Under ‘Contents’ (Plan center group), you will always look at the complete hierarchy of suites with their test cases. Notice how for some test cases, if different configurations are assigned to different testers, we will show the word ‘Multiple’ under the ‘Testers’ column (as I discussed earlier in the series, in the Planning view, we only show one test case, even if it is assigned to different configurations)

Although you will be able to filter on some columns (like Priority), you won’t be able to filter based on Tester or Configurations, because we don’t show individual test cases for every configuration:

However, under ‘Run tests’ (Test center group), you will be able to ‘Filter’ by Tester and/or Configurations:

You can also filter from within the columns:

Thanks,
David Gorena Elizondo

Recording Hover’s in CodedUITest

VSTS Quality Tools — Mon, 23 Nov 2009 11:02:00 GMT

Mouse Hover is an important part of a Application WorkFlow. Hovering on a control brings about changes in the application which leads to Controls appearing/disappearing , Properties of the controls being changed ,new windows coming up etc. Needless to say it is pertinent that all the Hover Actions are identified correctly during recording to ensure a proper Playback.

IMPLICIT HOVER

For recordings on WEB Applications CodedUITest intelligently identifies that an Hover action has been performed on the application and records a Hover-Action on the control on which the Mouse was hovered . This type of auto-generated Hover-Action is referred to as IMPLICIT HOVER. The action such generated was not explicitly performed by the User (however not recording the action would most certainly lead to a faulty recording) and as such is tagged with the ContinueOnError flag. This indicates that even if the action is not played back the Test wont fail and execution will pass on to the next Step. The code generated for Implicit Hovers will look like

// Set flag to allow play back to continue if non-essential actions fail. (For example, if a mouse hover action fails.)
Playback.PlaybackSettings.ContinueOnError = true;

// Mouse hover 'My Bay' link at (56, 3)
Mouse.Hover(uIMyBayHyperlink, new Point(56, 3));

// Reset flag to ensure that play back stops if there is an error.
Playback.PlaybackSettings.ContinueOnError = false;

Implicit Hover is a very cool feature in CodedUITest and greatly improves User experience.

The Implicit Hovers can be turned off by modifying the CodedUITestBuilder.exe.config and changing the value of the RecordImplicitHover key. By Default it is turned ON.

<add key="RecordImplicitHover" value="true"/>

This can be similarly turned OFF in the Fast-Forward a Manual Test scenario by updating the mtlm.exe.config.

EXPLICIT HOVER

For Client Side Applications Hover-Actions are not recorded implicitly. User has to record the Action Explicitly by Hovering the Mouse on the desired location and pressing a combination of keys to record the Hover-Action. The default Key combination are CTRL + SHIFT + R. The default value can be changed by updating the config files as mentioned above. Explicit Hovers are actual User Actions and are not marked with ContinueOnError flag as a result.

<add key="HoverKeyModifier" value="Control, Shift , Alt"/>
<add key="HoverKey" value="R"/>

Explicit Hover action can be generated for Web Applications as well..

ciao

Rituparna

<< CrossPosted from my MSDN Blog http://blogs.msdn.com/rituparna/archive/2009/11/23/recording-hover-s-in-codeduitest.aspx >>

Order Test Cases

VSTS Quality Tools — Sat, 21 Nov 2009 01:04:57 GMT

Sometimes, you want to run your tests in a specific order. For example, you are testing an online shopping site. You want to test the work flow of Login, Browse items, Search items, Add items to cart and Check out. You have the test cases for each step in the wok flow. You want to execute the test cases in the above work flow order. By default, MTLM orders the test cases in the order you add them to the suite. However, you may not have added the test cases in the order you want, like what I have in the first picture. In this case, you can click the Order button on the toolbar.

In the following dialog, change the order number. Click Update order button to preview the change. You can have duplicate numbers, MTLM will order the duplicates in the order they show in the list. When you feel good about the new order, you can click Apply Changes to save it.

After that, open Run Tests, select all the tests and click Run. Your tests will be run in the right order!

Thanks,

Christine Zhao

Planning vs Testing (Part 3)

VSTS Quality Tools — Fri, 20 Nov 2009 20:14:20 GMT

In part 1 and part 2 of the series, I have mentioned reasons why we use similar views for planning and executing test cases (and I have discussed what makes these views different). Today, I will focus on yet another specific reason: Configurations.

We can use the ‘Planning view’ under ‘Contents’ (Plan center group) to assign configurations to one or more test cases that live in our test plan. The view will show how many configurations are associated to any given test case.

However, the ‘Testing view’ under ‘Run tests’ (Test center group), will actually show ‘n’ number of the same test case (‘n’ being the number of configurations that the test case is associated with), each with its configuration, so that the tester can easily run the test that he is interested in (depending on his environment):

Thanks,
David Gorena Elizondo

The View Toolbar

VSTS Quality Tools — Fri, 20 Nov 2009 08:00:00 GMT

You might have noticed above many grids there are two distinct toolbars. The primary one has options for interacting with items in the list. The other one that is right-aligned is for managing the view. Here are some examples:

1. Verify Bugs – change which query is used: bugs assigned to me, bugs created by me, or a custom query definable by you. Also an Edit button to modify the custom query.

2. Analyze Test Runs – a toggle button for whether or not to show manual runs, and a date filter to only show recent items

3. Recommended Tests – not the same style as the other two, but changes the entire toolbar and grid out so you can see either recommended tests (tests impacted by code changes in each build in the selection) or the related work items (from all the builds in the selection)

Cheers,

David

Engineering Lead, MTLM

Beta2 online help

VSTS Quality Tools — Thu, 19 Nov 2009 21:25:52 GMT

Our doc team (we call them the User Education or the UE team) has been hard at work supplementing the product with great deal lot of additional documentation to help you understand the various features in the product and to best leverage them. The table below is a quick way to get to some of the more important links. Feel free to browse around and if you find the information useful or not let us know what you think.

Topic	Link
Running Tests Setting Up Test Machines to Run Tests or Collect Data Reviewing Test Results	http://msdn.microsoft.com/en-us/library/dd286680(VS.100).aspx http://msdn.microsoft.com/en-us/library/dd293551(VS.100).aspx http://msdn.microsoft.com/en-us/library/ms182493(VS.100).aspx
Running Manual Tests Using Test Runner	http://msdn.microsoft.com/en-us/library/dd286725(VS.100).aspx
Submitting Bugs Tracking Software Quality	http://msdn.microsoft.com/en-us/library/dd286746(VS.100).aspx http://msdn.microsoft.com/en-us/library/dd286599(VS.100).aspx
Quick Start Guide Managing New Testing Efforts Defining Your Testing Efforts	http://msdn.microsoft.com/en-us/library/dd380763(VS.100).aspx http://msdn.microsoft.com/en-us/library/dd286594(VS.100).aspx http://msdn.microsoft.com/en-us/library/dd286581(VS.100).aspx
Creating and Managing Tests	http://msdn.microsoft.com/en-us/library/dd286729(VS.100).aspx
Verifying Code by Using Unit Tests	http://msdn.microsoft.com/en-us/library/dd264975(VS.100).aspx

Thanks,

Ram Cherala, Principal Program Manager, Visual Studio Test Tools

Artifact Updates

VSTS Quality Tools — Thu, 19 Nov 2009 20:09:30 GMT

One thing you may not notice, but would if it didn’t work correctly, is what happens when there are two representations of the same item in different places and when one of the two instances changes. One example is opening a manager activity which lists many items:

Open the item, and modify a property:

If you switched back to the manager view at this point, you’ll see the item’s title has not change. This is because these are different instances. The editor does not share the same info as the item in the list. The item has unsaved changes, and those changes will not be visible by anyone else, including the manager view, until the artifact is saved.

So next we click Save and Close on the test case, which will return us to the Test Case Manager. Without refreshing the list manually, the item will update and the updated title will be visible in the list:

This artifact update works only on the local client. Other clients are not notified of changes. Just like in Team Explorer, other clients must refresh their own view to see changes by others. Sometimes one can get into a situation where two different people are trying to update an item. If they both open up the item at the same revision, make changes, and try to save, the second person will get an error. This is to prevent the second person from overwriting the first person’s committed changes.

Cheers,

David Williamson

Engineering Lead, MTLM

Terry Clancy Posts About Partner Integration Opportunities

VSTS Quality Tools — Wed, 18 Nov 2009 21:26:49 GMT

Terry Clancy, a Business Development Manager with Microsoft, has posted about opportunities for partners. It is also a good overview of the features across Team Test and Team Lab, how things fit together, and includes plenty of screenshot eye candy.

http://blogs.msdn.com/terryclancy/archive/2009/11/18/visual-studio-2010-test-tools-partner-integration-opportunities.aspx

Writing Queries for Custom Work Items and Work Item Categories

VSTS Quality Tools — Tue, 17 Nov 2009 21:41:00 GMT

I talked about customizing work items and how this can be used with categories to get the best out of MTLM before, now I will expand on that topic and show you how to compose queries to include this extra level of information.

If you go to the Queries activity in MTLM (under the Tracking menu) and select the out of the box query Team Queries->My Bugs, you will see that these queries are constructed with the clause Work Item Type = Bug.

This will only show work items of the specific type Bug, and we want to show all of the different work items which are part of the Bugs Category. To do this, you need to modify your query to change the clause “Work Item Type = Bug” to use “Work Item Type In Group Bug Category”.

Notice how the different bug work item types show up in the results.

Thanks,
Guillermo Serrato

Planning vs Testing (Part 2)

VSTS Quality Tools — Mon, 16 Nov 2009 23:26:55 GMT

In a previous blog post, I mentioned some of the reasons why the Planning and Testing views in MTLM look very similar but behave differently. Today, in this post, I will continue talking about this.

As you can see when playing around with the product, activity pages contain toolbars, with buttons which invoke specific actions. There are certain actions that are needed for planning purposes, and others –very different for execution. Since we have limited space in these toolbars (and because planning actions are very different from execution actions), we decided to have two separate views.

Look at the toolbars in the Planning view:

We have the following actions:

New – to create static or query-based suites
Add requirements – to add existing requirements
Copy test suites – to copy test suites from other existing test plans
Delete – To delete existing suites
Open – to open an existing test case
Add – to add one or multiple test cases to any given suite
New – to create a new test case and add it to a suite
Copy test case – to make a copy of a selected test case
Remove – to remove a test case from the suite
Assign – to assign testers to test cases in the suite
Configurations – to assign one or more configurations to the test cases in the suite
Order – to give test cases an order of execution

Now compare these planning actions to those actions in toolbars that live in the Testing view:

Run – to run all active tests in a given suite
Filter – to filter whatever is shown in this activity (based on ‘tester’ and/or ‘configurations’)
Run/Run with options – to run one or more test cases in a suite
View results – to view the latest results of a selected test case
Open test case – to open the selected test case
Block – to block the selected test case
Reset – to reset the selected test case to an active state

As you can see, actions that can be taken on suites/test cases in each view are very different. Many of these actions will also affect how test cases appear in each view, and I will be talking about this in a later post.

Thanks,
David Gorena Elizondo

Zen and the Art of Test Case Composition

VSTS Quality Tools — Mon, 16 Nov 2009 21:46:56 GMT

I’ve posted an article on my MSDN blog about strategies and considerations when composing your test cases. If you’re familiar with Test Case Authoring in MTLM (Ryan Vogrinec and David R. Williamson both have helpful posts on the subject), you can read more about the possibilities and tradeoffs in Test Case factoring at the following link:

http://blogs.msdn.com/joemu/archive/2009/11/16/zen-and-the-art-of-test-case-composition.aspx

Enjoy,

Joe Allan Muharsky

Copy/Paste Test Steps

VSTS Quality Tools — Mon, 16 Nov 2009 21:29:29 GMT

Test steps are very important in manual testing because they provide the detailed semantics on how each test case is to be validated and ultimately completed. That said, if you consider most manual test cases will have several steps each and most organizations will have a number of test cases, this results in a large number of steps to manage. Copying and pasting from popular tools like Microsoft Excel and Notepad reduces the time needed to properly manage, compose, and translate steps.

The test steps control inside the test case work item supports bulk copying and pasting of data from a variety of data sources, including other test steps, Microsoft Excel, Notepad, or any other tool that is capable of copying tab-delimited data onto the clipboard.

Copying steps from other data sources

Copying steps from Microsoft Excel requires that you use only two columns in your table of steps: the first column will map to "Action” and the second column will map “Expected Result” in the test steps work item control. Copy the range of two-column rows that you want to insert to the clipboard and paste them using either the context menu or keyboard (Ctrl-V):

You can also copy steps from the test steps control and paste them into either other test cases, or other data sources. Select a range of rows in the test steps control and either copy with the keyboard (CTRL-C), or use the context menu to copy the steps. Notice they are rendered in notepad as readable TAB-delimited rows:

Pasting the text above into a new test case provides you the same user experience as when working with Microsoft Excel:

Cheers,

Ryan Vogrinec

Tell Us What You Think

VSTS Quality Tools — Fri, 13 Nov 2009 23:46:26 GMT

We love to hear your feedback and given that our final beta for Visual Studio 2010 is now available this is your best opportunity to influence the product. In Microsoft Test and Lab Manager(MTLM) we are using the Microsoft Customer Experience Improvement Program (CEIP) to gather application usage data. CEIP collects information about how our customers use Microsoft programs and contains no personally identifiable information. We use this information to improve the products and features customers use most often and to help solve problems.

Participation in the program is voluntary, and for beta release we automatically enable sending this data. Click Home –> Options

It has been a few weeks since folks have started using beta 2 and we are already seeing some good information come thru. Following are some of the metrics that we are collecting:

1) MTLM Usage

a. Most used views (~1000 launches): Run Tests, Plan Contents, Workitem Editor and Test Runner

b. Less frequently used views (~200 launches): Manage VM Template, SaveAs VM Template, Environment Editor, Recommended Tests

c. Views not launched at all: Configuration Editor, About and Settings dialog

2) Actions performed

a. Most folks navigate Back and not Forward (336 vs. 22 invocations of these actions respectively)

b. Docking commands are not used frequently, of 787 invocations of Test Runner, docking behavior was changed only 8 times

c. Help has 394 invocations.

d. Home button had 300 invocations.

3) Artifact Counts (test cases, plans, runs)

a. Artifact counts are < 99 for most items (plans, runs, configurations)

b. 3% users already have between 100 - 500 test cases

c. 10% users ran queries that returned more than 100 workitems

Some of the changes we are considering based on the data are mentioned below.

We saw that Test Plan Manager was launched 766 times but users opened Test Plan Editor from there only 90 times. Could this mean Test Plan Manager may not make sense as the first view in Organize group? The other views on the Organize group in order of preference were: Manage Configurations, Manage Test Cases, Manage S3s so should we elevate one of these to the first view?

Users find the Go To Work Item functionality in TFC very useful so we provided the CTRL+G shortcut in MTLM too. But this has been invoked only once! We need to consider how we can make this more discoverable.

As time progresses we see more usage patterns and derive useful information to make MTLM more user friendly, but in the meantime your direct feedback is the best way for use to know what you think!

Regards,

Abhishek Lal

Create/Edit Test Settings in Test Plan Properties

VSTS Quality Tools — Fri, 13 Nov 2009 22:48:48 GMT

In manual or automated testing, you will probably want to specify your own test settings when running tests. The test settings associated with the test plan will be used by default when you click Run button in Run Tests.

There are two ways to create or edit a test settings associated with the test plan. The first option, you can open Lab Center | Test Settings. In the Test Settings Manager, you can click New or Open a selected test settings. Then you can specify the test settings in Test Plan Properties.

The second option is much easier. You can directly create a new test settings from the Test Settings dropdown in Test Plan Properties. Click the <New…> dropdown item.

The test settings editor will be opened inside of the Test Plan Properties. Follow the wizard and fill in all the fields that you care and click Finish.

Now in the Test Plan Properties, you will see your newly created test settings automatically selected.

You can also open an existing test settings from the dropdown, edit and save it. This will be done within Test Plan Properties as well.

Thanks,

Christine Zhao