Visual Studio Team Test : Testing

Understanding the code generated by “Coded UI Test” – Part 2

VSTS Quality Tools — Mon, 28 Dec 2009 12:39:24 GMT

This is Part 2 of two part series. In Part 1 we focused on -

Why is generated code not a straight-line code?

In this part, we will focus on -

How is generated code making reusability & customization easier?...

Note: Cross posted from Gautam Goenka (MSFT).
Permalink

Understanding the code generated by “Coded UI Test” – Part 1

VSTS Quality Tools — Mon, 28 Dec 2009 12:38:50 GMT

This is Part 1 of two part series. In this part we will focus on -

Why is generated code not a straight-line code?

In the Part 2, we will focus on -

How is generated code making reusability & customization easier? ...

Note: Cross posted from Gautam Goenka (MSFT).
Permalink

How can I configure MTLM to use my custom bug / test case type?

VSTS Quality Tools — Tue, 15 Dec 2009 12:37:00 GMT

Note: Cross posted from Rubel's Blog.
Permalink

How can I configure MTLM to use my custom bug / test case type?

I got this Q on my previous blog and thought of posting a separate post on it as it's a bit tricky to do it and hence other folks may need this info.

In Microsoft Test and Lab Manager (MTLM), we don’t provide any UI way to set a particular work item type as default work item type for a category. Basically, if I’ve added a new bug type say “CustomBug” to TFS and want MTLM to refer to that type always while opening a new bug etc then there is no UI way to achieve it.

But yes, you can do it through witadmin using importcategories / exportcategories command.

First you need to exports the XML definition of categories defined on the Team Foundation Server by using exportcategories command e.g.

The following command exports the work item type categories defined for the AdventureWorks project to the Categories XML file:

C:\Program Files\Microsoft Visual Studio 10.0\VC>witadmin exportcategories /collection:"http://AdventureWorksServer:8080/AWTeam/Collection1”

/p:AdventureWorks /f:"Categories.xml"

Categories.xml will look this:

<?xml version="1.0" encoding="utf-8"?>

<cat:CATEGORIES xmlns:cat="http://schemas.microsoft.com/VisualStudio/2008/workitemtracking/categories">

</CATEGORY>

</CATEGORY>

</CATEGORY>

</CATEGORY>

</cat:CATEGORIES>

As you can see that there is default work item type for each category like Bug, Test Case, Shared Step.

You just need to modify the xml file so as to point DEFAULTWORKITEMTYPE of required category to yours newly created work item type. E.g. If I want to use work item type CustomBug as default work item type for Bug category then my modified XML will look like this:

<?xml version="1.0" encoding="utf-8"?>

<cat:CATEGORIES xmlns:cat="http://schemas.microsoft.com/VisualStudio/2008/workitemtracking/categories">

</CATEGORY>

</CATEGORY>

</CATEGORY>

</CATEGORY>

</cat:CATEGORIES>

After that you just need to import this XML back into team project of TFS using importcategories. e.g. The following example imports categories from the myCategories XML file to the AdventureWorks project:

C:\Program Files\Microsoft Visual Studio 10.0\VC>witadmin importcategories /collection:"http://AdventureWorksServer:8080/AWTeam/Collection1”

/p:AdventureWorks /f:"Categories.xml"

Now if you open any bug in Test Runner / MTLM then it will open your CustomBug work item type form instead of default bug form. Similarly you can do required changes to use customized Test Case / any other work item.

Required Permissions

For the team project where the work item types are defined, you must have the following permissions set:

· To export categories of work item types, you must be a member of the Readers group or have your View work items in this node permission set to Allow

· To import categories of work item types, you must be a member of the Team Foundation Administrators security group or the Project Administrators security group.

For more details on TFS permissions check here

Note: Cross posted from Rubel's Blog.
Permalink

Planning vs Testing (Part 5 – Final)

VSTS Quality Tools — Wed, 25 Nov 2009 19:14:46 GMT

Today, I will finally end this series of blog posts where I’ve been talking about the main differences that you will find in Planning and Testing views (although they appear to be very similar at a first glance). You can find the previous posts here:

Double clicking (or hitting enter on a selected item) throughout MTLM will trigger the action that we thought was the most basic one for each activity.

Well, for any given test case, if you’re in the Planning view (Plan center group, under ‘Contents’), double clicking or hitting enter on a selected item will open the test case:

However, in the Testing view (Test center group, under ‘Run tests’), this same action will actually open the latest result for that given test case (assuming that there is a result; if there isn’t, nothing will happen):

Thanks,
David Gorena Elizondo

Planning vs Testing (Part 4)

VSTS Quality Tools — Tue, 24 Nov 2009 19:36:19 GMT

In previous posts, I’ve been discussing why Planning and Testing views look similar but behave differently. You can refer to them by clicking on Part 1, Part 2, or Part 3.

Sometimes, depending on the organization, planning is done by some people, and execution is done by others. And even if it is the same people who do both, it will be common to want to look only at a specific set of test cases when actually executing them. This is another big difference between both views:

Under ‘Contents’ (Plan center group), you will always look at the complete hierarchy of suites with their test cases. Notice how for some test cases, if different configurations are assigned to different testers, we will show the word ‘Multiple’ under the ‘Testers’ column (as I discussed earlier in the series, in the Planning view, we only show one test case, even if it is assigned to different configurations)

Although you will be able to filter on some columns (like Priority), you won’t be able to filter based on Tester or Configurations, because we don’t show individual test cases for every configuration:

However, under ‘Run tests’ (Test center group), you will be able to ‘Filter’ by Tester and/or Configurations:

You can also filter from within the columns:

Thanks,
David Gorena Elizondo

Recording Hover’s in CodedUITest

VSTS Quality Tools — Mon, 23 Nov 2009 11:02:00 GMT

Mouse Hover is an important part of a Application WorkFlow. Hovering on a control brings about changes in the application which leads to Controls appearing/disappearing , Properties of the controls being changed ,new windows coming up etc. Needless to say it is pertinent that all the Hover Actions are identified correctly during recording to ensure a proper Playback.

IMPLICIT HOVER

For recordings on WEB Applications CodedUITest intelligently identifies that an Hover action has been performed on the application and records a Hover-Action on the control on which the Mouse was hovered . This type of auto-generated Hover-Action is referred to as IMPLICIT HOVER. The action such generated was not explicitly performed by the User (however not recording the action would most certainly lead to a faulty recording) and as such is tagged with the ContinueOnError flag. This indicates that even if the action is not played back the Test wont fail and execution will pass on to the next Step. The code generated for Implicit Hovers will look like

// Set flag to allow play back to continue if non-essential actions fail. (For example, if a mouse hover action fails.)
Playback.PlaybackSettings.ContinueOnError = true;

// Mouse hover 'My Bay' link at (56, 3)
Mouse.Hover(uIMyBayHyperlink, new Point(56, 3));

// Reset flag to ensure that play back stops if there is an error.
Playback.PlaybackSettings.ContinueOnError = false;

Implicit Hover is a very cool feature in CodedUITest and greatly improves User experience.

The Implicit Hovers can be turned off by modifying the CodedUITestBuilder.exe.config and changing the value of the RecordImplicitHover key. By Default it is turned ON.

<add key="RecordImplicitHover" value="true"/>

This can be similarly turned OFF in the Fast-Forward a Manual Test scenario by updating the mtlm.exe.config.

EXPLICIT HOVER

For Client Side Applications Hover-Actions are not recorded implicitly. User has to record the Action Explicitly by Hovering the Mouse on the desired location and pressing a combination of keys to record the Hover-Action. The default Key combination are CTRL + SHIFT + R. The default value can be changed by updating the config files as mentioned above. Explicit Hovers are actual User Actions and are not marked with ContinueOnError flag as a result.

<add key="HoverKeyModifier" value="Control, Shift , Alt"/>
<add key="HoverKey" value="R"/>

Explicit Hover action can be generated for Web Applications as well..

ciao

Rituparna

<< CrossPosted from my MSDN Blog http://blogs.msdn.com/rituparna/archive/2009/11/23/recording-hover-s-in-codeduitest.aspx >>

Order Test Cases

VSTS Quality Tools — Sat, 21 Nov 2009 01:04:57 GMT

Sometimes, you want to run your tests in a specific order. For example, you are testing an online shopping site. You want to test the work flow of Login, Browse items, Search items, Add items to cart and Check out. You have the test cases for each step in the wok flow. You want to execute the test cases in the above work flow order. By default, MTLM orders the test cases in the order you add them to the suite. However, you may not have added the test cases in the order you want, like what I have in the first picture. In this case, you can click the Order button on the toolbar.

In the following dialog, change the order number. Click Update order button to preview the change. You can have duplicate numbers, MTLM will order the duplicates in the order they show in the list. When you feel good about the new order, you can click Apply Changes to save it.

After that, open Run Tests, select all the tests and click Run. Your tests will be run in the right order!

Thanks,

Christine Zhao

Planning vs Testing (Part 3)

VSTS Quality Tools — Fri, 20 Nov 2009 20:14:20 GMT

In part 1 and part 2 of the series, I have mentioned reasons why we use similar views for planning and executing test cases (and I have discussed what makes these views different). Today, I will focus on yet another specific reason: Configurations.

We can use the ‘Planning view’ under ‘Contents’ (Plan center group) to assign configurations to one or more test cases that live in our test plan. The view will show how many configurations are associated to any given test case.

However, the ‘Testing view’ under ‘Run tests’ (Test center group), will actually show ‘n’ number of the same test case (‘n’ being the number of configurations that the test case is associated with), each with its configuration, so that the tester can easily run the test that he is interested in (depending on his environment):

Thanks,
David Gorena Elizondo

The View Toolbar

VSTS Quality Tools — Fri, 20 Nov 2009 08:00:00 GMT

You might have noticed above many grids there are two distinct toolbars. The primary one has options for interacting with items in the list. The other one that is right-aligned is for managing the view. Here are some examples:

1. Verify Bugs – change which query is used: bugs assigned to me, bugs created by me, or a custom query definable by you. Also an Edit button to modify the custom query.

2. Analyze Test Runs – a toggle button for whether or not to show manual runs, and a date filter to only show recent items

3. Recommended Tests – not the same style as the other two, but changes the entire toolbar and grid out so you can see either recommended tests (tests impacted by code changes in each build in the selection) or the related work items (from all the builds in the selection)

Cheers,

David

Engineering Lead, MTLM

Beta2 online help

VSTS Quality Tools — Thu, 19 Nov 2009 21:25:52 GMT

Our doc team (we call them the User Education or the UE team) has been hard at work supplementing the product with great deal lot of additional documentation to help you understand the various features in the product and to best leverage them. The table below is a quick way to get to some of the more important links. Feel free to browse around and if you find the information useful or not let us know what you think.

Topic	Link
Running Tests Setting Up Test Machines to Run Tests or Collect Data Reviewing Test Results	http://msdn.microsoft.com/en-us/library/dd286680(VS.100).aspx http://msdn.microsoft.com/en-us/library/dd293551(VS.100).aspx http://msdn.microsoft.com/en-us/library/ms182493(VS.100).aspx
Running Manual Tests Using Test Runner	http://msdn.microsoft.com/en-us/library/dd286725(VS.100).aspx
Submitting Bugs Tracking Software Quality	http://msdn.microsoft.com/en-us/library/dd286746(VS.100).aspx http://msdn.microsoft.com/en-us/library/dd286599(VS.100).aspx
Quick Start Guide Managing New Testing Efforts Defining Your Testing Efforts	http://msdn.microsoft.com/en-us/library/dd380763(VS.100).aspx http://msdn.microsoft.com/en-us/library/dd286594(VS.100).aspx http://msdn.microsoft.com/en-us/library/dd286581(VS.100).aspx
Creating and Managing Tests	http://msdn.microsoft.com/en-us/library/dd286729(VS.100).aspx
Verifying Code by Using Unit Tests	http://msdn.microsoft.com/en-us/library/dd264975(VS.100).aspx

Thanks,

Ram Cherala, Principal Program Manager, Visual Studio Test Tools

Terry Clancy Posts About Partner Integration Opportunities

VSTS Quality Tools — Wed, 18 Nov 2009 21:26:49 GMT

Terry Clancy, a Business Development Manager with Microsoft, has posted about opportunities for partners. It is also a good overview of the features across Team Test and Team Lab, how things fit together, and includes plenty of screenshot eye candy.

http://blogs.msdn.com/terryclancy/archive/2009/11/18/visual-studio-2010-test-tools-partner-integration-opportunities.aspx

Writing Queries for Custom Work Items and Work Item Categories

VSTS Quality Tools — Tue, 17 Nov 2009 21:41:00 GMT

I talked about customizing work items and how this can be used with categories to get the best out of MTLM before, now I will expand on that topic and show you how to compose queries to include this extra level of information.

If you go to the Queries activity in MTLM (under the Tracking menu) and select the out of the box query Team Queries->My Bugs, you will see that these queries are constructed with the clause Work Item Type = Bug.

This will only show work items of the specific type Bug, and we want to show all of the different work items which are part of the Bugs Category. To do this, you need to modify your query to change the clause “Work Item Type = Bug” to use “Work Item Type In Group Bug Category”.

Notice how the different bug work item types show up in the results.

Thanks,
Guillermo Serrato

Planning vs Testing (Part 2)

VSTS Quality Tools — Mon, 16 Nov 2009 23:26:55 GMT

In a previous blog post, I mentioned some of the reasons why the Planning and Testing views in MTLM look very similar but behave differently. Today, in this post, I will continue talking about this.

As you can see when playing around with the product, activity pages contain toolbars, with buttons which invoke specific actions. There are certain actions that are needed for planning purposes, and others –very different for execution. Since we have limited space in these toolbars (and because planning actions are very different from execution actions), we decided to have two separate views.

Look at the toolbars in the Planning view:

We have the following actions:

New – to create static or query-based suites
Add requirements – to add existing requirements
Copy test suites – to copy test suites from other existing test plans
Delete – To delete existing suites
Open – to open an existing test case
Add – to add one or multiple test cases to any given suite
New – to create a new test case and add it to a suite
Copy test case – to make a copy of a selected test case
Remove – to remove a test case from the suite
Assign – to assign testers to test cases in the suite
Configurations – to assign one or more configurations to the test cases in the suite
Order – to give test cases an order of execution

Now compare these planning actions to those actions in toolbars that live in the Testing view:

Run – to run all active tests in a given suite
Filter – to filter whatever is shown in this activity (based on ‘tester’ and/or ‘configurations’)
Run/Run with options – to run one or more test cases in a suite
View results – to view the latest results of a selected test case
Open test case – to open the selected test case
Block – to block the selected test case
Reset – to reset the selected test case to an active state

As you can see, actions that can be taken on suites/test cases in each view are very different. Many of these actions will also affect how test cases appear in each view, and I will be talking about this in a later post.

Thanks,
David Gorena Elizondo

Zen and the Art of Test Case Composition

VSTS Quality Tools — Mon, 16 Nov 2009 21:46:56 GMT

I’ve posted an article on my MSDN blog about strategies and considerations when composing your test cases. If you’re familiar with Test Case Authoring in MTLM (Ryan Vogrinec and David R. Williamson both have helpful posts on the subject), you can read more about the possibilities and tradeoffs in Test Case factoring at the following link:

http://blogs.msdn.com/joemu/archive/2009/11/16/zen-and-the-art-of-test-case-composition.aspx

Enjoy,

Joe Allan Muharsky

Copy/Paste Test Steps

VSTS Quality Tools — Mon, 16 Nov 2009 21:29:29 GMT

Test steps are very important in manual testing because they provide the detailed semantics on how each test case is to be validated and ultimately completed. That said, if you consider most manual test cases will have several steps each and most organizations will have a number of test cases, this results in a large number of steps to manage. Copying and pasting from popular tools like Microsoft Excel and Notepad reduces the time needed to properly manage, compose, and translate steps.

The test steps control inside the test case work item supports bulk copying and pasting of data from a variety of data sources, including other test steps, Microsoft Excel, Notepad, or any other tool that is capable of copying tab-delimited data onto the clipboard.

Copying steps from other data sources

Copying steps from Microsoft Excel requires that you use only two columns in your table of steps: the first column will map to "Action” and the second column will map “Expected Result” in the test steps work item control. Copy the range of two-column rows that you want to insert to the clipboard and paste them using either the context menu or keyboard (Ctrl-V):

You can also copy steps from the test steps control and paste them into either other test cases, or other data sources. Select a range of rows in the test steps control and either copy with the keyboard (CTRL-C), or use the context menu to copy the steps. Notice they are rendered in notepad as readable TAB-delimited rows:

Pasting the text above into a new test case provides you the same user experience as when working with Microsoft Excel:

Cheers,

Ryan Vogrinec