Introduction to Record and Playback Engine in VSTT 2010

In an earlier blog entry, we have talked about the Test Runner activity in Microsoft Test and Lab Manager (MTLM) and how it allows you to record and capture the UI actions (such as mouse clicks and keyboard input) that the user takes on an Application under Test (AuT). The Test Runner tool also allows a tester to create an action recording while performing a manual test. Next time the tester is running the same test case, she has the option to playback the action recording and “fast-forward” through the UI actions quickly, instead of having to manually repeat each step.

The CodedUI Test feature also allows you to record UI actions and generate test code from the recording.

Both Test Runner and CodedUI feature use a common action recording and playback (RnP) engine to enable their scenarios. In this post we will take a look at the inner workings of this engine.

Record and Playback Engine Architecture

The following figure shows the high-level architecture of the RnP engine and how it is consumed by Test Runner and CodedUI Test.

The Recorder component is responsible for listening and capturing the UI actions. The Playback engine is responsible for interpreting the recording and playing back the equivalent actions. The Technology abstraction layer is responsible for abstracting away the UI technology specific details and provides a consistent interface to call into. All the logic for handling UI technology specific details are implemented in technology specific plug-ins that implement a standard public interface. Since the interface is public, third party vendors can implement their plug-in to handle technologies that we do not support out of the box.

Recorder Logic

So how do we record a UI action? For example, say a tester wants to record a click on the OK button of a FileUpload Control

At a high level, the recorder will do the following:

• Listen for Mouse Events
• On mouse button down event it will get the X, Y coordinate
• Call technology specific apis to get the Control (e.g. button) at X,Y (in this example it will be a call into the Web plug-in)
• Fetch the properties to be used for searching the button
• Fetch the control hierarchy of the button
• Construct an internal query string that includes information about technology type, control hierarchy and control properties that will get used to search for the control during playback
• Capture the Actions (Mouse click in this case)
• Aggregate actions if needed
• Generate an XML representation of the recording (this can be used for interpreted playback in Test Runner or Code generation in CodedUI Test)

The quality of the recording will depend on the technology specific plug-in being able to provide good information about the control, its properties and the UI hierarchy.

I have highlighted the “aggregate actions” step above and would like to call it out in more detail. One of the goals and differentiating features of the Recorder is to provide an “Intent-Aware” recording. Instead of just capturing raw user actions, the engine has been designed to capture the intent of the user. For example, a user intending to record a file-upload operation in Internet Explorer will perform five UI actions as shown below. The RnP recorder is intelligent to aggregate a series of related actions into an equivalent action that captures the intent of the user. In the example below, only two actions are recorded instead of five. Reducing the number of raw actions that were recorded also makes the recording more robust and resilient to playback failures.

The following is the XML snippet for the recorded actions for the above example. Only two actions are captured, one is a setvalue in the fileinput control and one is a click on the ok button.

<SetValueAction UIObjectName=

      "CurrentUIMap.UploadDocumentWindowWindow.HttpstsindiasitesPriClient.

       UploadDocumentDocument.NameFileInput">

  <ParameterName />

  <Value Encoded="false">C:\Users\abhiag\Documents\FileToUpload.txt</Value>

  <Type>String</Type>

</SetValueAction>

<MouseAction UIObjectName=

       "CurrentUIMap.UploadDocumentWindowWindow.HttpstsindiasitesPriClient.

         UploadDocumentDocument.OKButton">

  <ParameterName />

  <ModifierKeys>None</ModifierKeys>

  <IsGlobalHotkey>false</IsGlobalHotkey>

  <Location X="20" Y="16" />

  <WheelDirection>0</WheelDirection>

  <ActionType>Click</ActionType>

  <MouseButton>Left</MouseButton>

</MouseAction>

The following XML snippet shows the properties about the OK button that is captured to enable search during playback. If you see closely, we collect two set of properties, primary and secondary. Primary properties are required to complete a successful search, secondary property are used optionally to refine the results of a search.

<UIObject ControlType="Button" Id="OKButton" FriendlyName="OK" SpecialControlType="None">

  <TechnologyName>Web</TechnologyName>

  <AndCondition Id="SearchCondition">

  <AndCondition Id="Primary">

  <PropertyCondition Name="ControlType">Button</PropertyCondition>

  <PropertyCondition Name="Id">ctl00_PlaceHolderMain_ctl00_RptControls_btnOK</PropertyCondition>

  <PropertyCondition Name="Name">ctl00$PlaceHolderMain$ctl00$RptControls$btnOK</PropertyCondition>

  <PropertyCondition Name="TagName">INPUT</PropertyCondition>

  <PropertyCondition Name="Value">OK</PropertyCondition>

  </AndCondition>

   <FilterCondition Id="Secondary">

  <PropertyCondition Name="Type">button</PropertyCondition>

  <PropertyCondition Name="Title" />

  <PropertyCondition Name="Class">ms-ButtonHeightWidth</PropertyCondition>

  <PropertyCondition Name="ControlDefinition">accessKey=O id=ctl00_PlaceHolderMain_ctl</PropertyCondition>

  <PropertyCondition Name="TagInstance">8</PropertyCondition>

  <PropertyCondition Name="RelativePosition">{X=1225,Y=274}</PropertyCondition>

  </FilterCondition>

  </AndCondition>

  <Descendants />

</UIObject>

If you are using the Test Runner, you can view the attached recorded XML files to get an idea of what information is being recorded.

Playback Logic

The playback engine can be invoked via the interpreter in the Test Runner, which reads the XML and plays back the action or it can be invoked via automation apis in a CodedUI test. At a high-level the playback engine performs the following steps:

• Search for the Control: The playback engine uses the control information captured during recording to search for the control. For search purpose, the UI is visualized as a tree of UI elements rooted at Desktop. It employs innovative heuristics to provide a resilient and fast search.
• Ensure Control is Visible: In order to interact with the control, the control should be visible. The playback engine performs the necessary actions such as scrolling etc to ensure the control is in view.
• Wait for Control to be ready (WFR): The playback needs to ensure that the control is ready to be acted upon before performing the action. The engine uses smart algorithms to ensure this. Plug-ins can override this behavior by providing technology specific implementation.
• UI Synchronization: Playback tries to ensure that the control that was supposed to have received an action has actually received it.
• Perform Action: The playback finally performs the UI action on the control. Currently, the playback supports the following primitive actions:
  • Check, Uncheck, DoubleClick, MouseHover, MouseMove, MouseButtonClick, MouseWheel, SendKeys, SetValueasComboBox, Drag, DragDrop, Scroll, SetFocus

Technology Support

The engine supports a wide range of platforms out-of-the box covering thick client applications built with .Net technologies to latest web applications

• Support for Windows Internet Explorer 7.0 and Windows Internet Explorer 8.0
  • Applications written in ASP.Net, php, jsp, asp, Ajax, SharePoint and web 2.0 applications are supported. The web plug-in uses The IEDOM for identifying properties of the controls and playing back the actions.
• Support for Win-forms Controls for Microsoft .Net framework 2.0, 3.5 SP1 and 4.0
  • For Win-forms applications, the plug-in uses windows accessibility technology Microsoft Active Accessibility (MSAA) to identify Control properties and playback actions.
• Windows Presentation Framework (WPF) for .Net framework 3.5 SP1 and 4.0
  • For WPF applications, the plug-in uses the UI Automation (UIA) accessibility framework in Windows to identify Control properties and playback actions.
• Custom Controls
  • Since we rely on windows accessibility technologies, controls that follow accessibility guidelines are also supported. We are also working with partners to enable support for some popular custom controls shipped by third party vendors. Additionally, vendors can used our extensibility APIs to add improved support for their custom controls.
• Firefox 3.0 Support
  • We are currently working on adding Firefox support. It will not be available in the box at RTM but we are actively working on making it available as add-on shortly afterwards.

Engine Extensibility

The engine has been designed with Extensibility in mind. We realize that there are scenarios where our default engine will not meet the needs of our customers. These include being able to test applications built on a different or new UI technology (e.g. flash) or certain application domains that require customized action recording support. Via our extensibility APIs, customers, third party vendors/ISVs can build plug-ins for these specific scenarios. Users can install and register the plug-ins with the engine and enable their scenarios.

Hope the above article has given you some insight into the inner workings of RnP engine and how it enables the Test Runner and the CodedUI Test features. In a future post, we will go into more details about the extensibility of the RnP engine and show how customers and third-party can extend it.

Abhishek Agrawal, Program Manager, Visual Studio Team Test

New VSTS Load Test Integration on CodePlex

Project Description

This tool takes a WCF trace file and a WCF client proxy, or a WCF interface contract, and generates a unit test that replays the same sequence of calls found in the trace file. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing.
The tool generates code for both Visual Studio 2005 and Visual Studio 2008. It also installs a wizard into both editions of Visual Studio for creating the trace and processing it from inside Visual Studio. If both editions are present the tool is installed into both editions. The source code is a Visual Studio 2005 project.

Latest Release June 2009

A new beta was released in June 2009 adding support for ASMX web services. Download it from here.

Features

• Replay of captured scenario in a unit test that can be included in a load test.
• Support for the DataContractSerializer.
• Support for message contracts.
• ASMX support (beta)
• Support for proxies generated using svcutil.
• Support for clients that create proxies at run time from contract interfaces.
• Supports calls to multiple services in a single scenario.
• Supports multiple calls to the same service operation.
• Filtering by SOAP action of which messages in the trace to replay.
• Readable and modifiable code is generated.
• Automatic association of trace message with proxy method (requires all operations to have a unique SOAP action).
• Support for client and server side traces.
• A command line tool for processing traces and generating code.
• Visual Studio 2005/2008 integration (Team Developer, Team Test, Team Suite and for 2008 also Professional)) that can be used instead of the command line tool.

10-4 Episode 23: An Introduction to Manual Testing

Brian Keller has posted a video "An Introduction to Manual Testing" on Channel 9. Check it out here. This is a very nicely done video of the concepts of manual testing leveraging key features of VSTS 2010 Team Test. You can learn about other features of this product at the team blog site here.

Cheers,

Ram Cherala, Princial PM Visual Studio Team Test

Automated User Interface Testing with Coded UI Test

Microsoft® Visual Studio® Team System 2010 introduces a new Test Template named "Coded UI Test".  You can verify the functionality of user interfaces with Coded UI Test. You can create a  Coded UI Test  by recording actions and generating code. You can also add assertions on properties of controls.

In this article, I will describe the steps to create a Coded UI Test, run it and view the results. I will also briefly describe the structure of a Coded UI Test Project.

Create a Coded UI Test Project

In Visual Studio Team System 2010, click on Test menu and choose New Test

This brings up the Add New Test Dialog. Choose Coded UI Test and click OK.

NOTE:- You can choose to create a Visual C# test project or a Visual Basic test project.

In the New Test Project dialog, change the Test Project name and click Create.

This creates a Coded UI Test Project and adds the required references. In addition it also creates blank Coded UI Test files which can now be populated with the test code.

In the Coded UI Test dialog, choose Use the Recorder. Other options are available on the screen to create a Coded UI Test from an action recording created in Microsoft Test and Lab Manager or Use the UI Control locator to add User interface controls to a UI Control map and then write code against them.

Record Actions and Generate Code

In the previous step, you have created a Coded UI Test Project and chose to use the recorder to generate code. This will start the  Coded UI Test Builder.

Click on Record Actions button to bring up the recorder.

Now perform the desired test steps. The Recorder window will become semi-transparent when you move focus away from it.

For this article, we are going to test the Add function in Calculator

The Test steps are

a. Launch Calculator

b. Click 7

c. Click the + sign

d. Click 2

e. Click = button

Note that a text description of actions will be shown in the recorder window.

Change the Method Name to TestAdd and click Generate Method.

Recorder dialog is closed and Coded UI Test Builder dialog opens up. The name of the generated method is shown.

Status bar will show messages from the last action performed.

Add Assertions to the Coded UI Test

In the previous steps, you have recorded actions and generated code. Now you can add assertions to validate controls on the user interface.

Click on the Add Assertions tab.

Click on the UI Control Locator button to launch the locator and add UI controls to the UI Control Map. You can then add validations on these controls.

Drag the crosshair over the calculator edit box and drop it when the edit box is highlighted in blue.

The Control Type and the name of the selected UI Control will be shown in the UI Control Locator dialog.

Click on Add Control to add it to the UI Control Map. Click on Show Properties button to show the properties of this control and add assertions.

In the UI Control Properties list, click on the Checkbox next to Text and click OK.

Now we have added an assertion that the result of the operation is equal to 9.

The UI Control Properties overlay is closed and the Coded UI Test Builder dialog comes up.

Click on the Save button to save the UI map and assertions.

Close the Coded UI Test Builder dialog by clicking on the X button.

Structure of a Coded UI Test Project

In the previous steps, you have created a coded ui test by recording actions and added assertions. The Coded UI Test file is now opened within the Visual Studio IDE.

NOTE the following characteristics about the generated code.

a. The CodedUITest1 class is tagged with a Coded UI Test attribute.

b. The CodedUITetMethod1 method is tagged with a Test Method attribute.

c. In the CodedUITestMethod1, there is an invokation of the TestAdd method we created from Coded UI Test Builder.

d. Also in the CodedUITestMethod1, we have the code for the assertion we added with the UI Control Locator.

Note the following in the Solution Explorer.

a. All required references for Coded UI Test are added automatically.

b. 4 files are added to the Test Project

i. CodedUITest1.cs - main Coded UI Test class, Test Methods and Assertions.

ii. RecordedMethods.cs - code for actions performed while recording

iii. UIMap.uitest - Representation of the UI Map (a container of UI Controls)

1. UIMap.Designer.cs - Code generated for the UIMap

iv. UserControls.cs - Specialized classes representing various UI Controls referenced in the Test.

Run Coded UI Test

Right-click inside the Coded UI Test Method and choose Run Tests

The code is compiled and then the test is run.

All actions recorded earlier will now be played back.

After the test run is completed, choose the Test Run and view Test Results from the context menu.

Summary

In this article, you saw how to

• Create a Coded UI Test
• Record actions and generate Code
• Locate Controls and add assertions
• Run the Coded UI Test

You were also introduced to the structure of a Coded UI Test.

Additional information is available on MSDN.

Try out these features in Beta1 and send us your feedback.

Mathew Aniyan – I am a Program Manager in Visual Studio Team System – Test Edition and focus on Coded UI Test and the Functional Testing Automation API. I work out of Hyderabad in India.

Check out Lab Management team blog

I don't know if you have noticed but the Lab Management team has been posting some exciting updates on the features they are building in to the next release of VSTS.

Check out their team blog at:

http://blogs.msdn.com/lab_management

Watch for further updates at this site and let them know what you think of the features.

Thanks for visiting!!

Ram Cherala, Principal Program Manager, Visual Studio Test Business

No More Missed Requirements

Testing organizations in today's world find it challenging to plan a testing effort based on the requirements or user stories being written by business analysts, and to stay ahead of the developers' implementation schedule.  Quite commonly test cases are authored in response to a requirement already having been implemented, which introduces a wasteful lag in the testing schedule and, more importantly, a collaborative gap between the testers and other players in the org.

How can we get the testers involved earlier?

Microsoft Test and Lab Manager (MTLM), available for download with Visual Studio Team System 2010 Beta 1, makes it easy for testers to track requirements and plan test coverage in parallel with implementation so that testing can begin at first opportunity.

In this article we'll walk through an end-to-end scenario in which you, as a test manager, create a test suite that is based upon a newly authored requirement, keep the suite parked in a ready state pending implementation of the requirement by developers, then kick off testing by your team when the requirement becomes available.

Create a Requirement-based Suite

Once you've connected to your team project and created a test plan in Microsoft Test and Lab Manager (MTLM) you can begin adding test suites that are based on requirements or user stories. 

To start, go to the Testing Center in MTLM, click Plan, then click Contents and then click Add requirements:

This brings up a dialog box listing all requirement work items in your project.  You can use the searching, sorting and filtering capabilities of this dialog to help you zero in on the particular requirements that you want to test in the current plan.  Select these items and click Add requirements to plan.

A requirement-based suite is created for each item you add to the plan.  By default, the suite is given a name based on the ID and Title of its corresponding requirement, a name such as "2338: Calculator enables user to perform basic arithmetic".  You can change the name by double clicking the suite or right clicking and pointing to Rename. 

Newly created suites have State: In progress by default; but you should change this to State: In planning if you are not yet ready to run the tests in a suite.  By parking the suite in a planning state you keep it from displaying in the Run Tests activity until you decide conditions are right for testing to begin.  Later, when your suite is ready to be run -- and the requirement is ready to be tested -- you can flip back to State: In progress, thereby surfacing the suite in the Run Tests activity and enabling the testers on your team to begin running the tests.

In the meantime, you'll want to finish your test suite. A requirement-based suite is automatically populated with any test cases in the project that are currently linked to the requirement work item in the TFS project.  You can link new or existing test cases by clicking New or Add. 

Click New to add a new test case to your requirement-based suite.  Give the new test case a Title and enter details including the actions and expected results for each test step, and then click Save and Close.  Your new test case is automatically added to the test suite, and is also automatically linked to the corresponding requirement work item.

To add existing test cases to your requirement-based suite, click Add, which brings up a dialog box listing all test cases in your project.  You can use the searching, sorting and filtering capabilities of this dialog to help you zero in on the test cases that cover this requirement.  Select these items and click Add test cases to suite.  Note that by adding test cases to the requirement-based suite, you are also linking those test cases to the requirement work item.

If you are concerned with testing your requirement across a matrix of configurations of hardware, OS, software, browsers, languages, or other characteristics, MTLM gives you fine control over which tests are to be run on which test configurations.

When you add test cases to the suite they automatically inherit the default test configurations associated with your test plan; but you can click the down arrow adjacent to Default configurations to modify the defaults for your requirement-based suite.  Furthermore, you can click the Configurations button to override the suite-level defaults and specify the exact configuration coverage of each test case in the suite.  You might do this, for example, if you wanted to economize by running your highest priority test cases on multiple configurations while restricting your lower priority test cases on a single configuration.  Or you might have a subset of test cases in your suite aimed at functionality that is only relevant to a particular configuration. 

Now that you've built a requirements-based suite covering a matrix of test configurations, you can click the Assign button to delegate testing to individual testers on your team.  This enables you to load balance resources across your team, or to assign testing based on areas of expertise or people's access to machines with particular configurations.

Let's say you've made your testing assignments and your requirement-based suite is good to go.  At this point, if the requirement is already available in current builds of your product you can immediately pull the trigger by setting the suite State: In progress, thereby enabling your test team to begin running tests from the MLTM Run Tests activity.

In an ideal world, however, you would have done your planning early enough to beat the developers to the punch, and would now be poised to kick off testing when the developers finish their implementation tasks and check the requirement into the build.

Determining Which Builds Have New Requirements

As a tester, you can use MTLM to "tune in" to the build cycle (TFS Team Build) and track the developers' progress toward implementing a requirement.  You achieve this by assigning a build number to a test plan and subscribing to information about the availability and content of future builds. 

From the Testing Center in MTLM, click Plan, then click Properties, and then click the down arrow next to the Build Candidates value, which is initially set to "Any definition or build quality" for a new test plan.

When you click the down arrow a small dialog box pops up to help you specify which build definition and build quality tag you want to track with the current test plan. 

Having established a build definition to track, click the Modify link adjacent to the current Build in use to open the Assign Build activity.   This activity shows you which build numbers are currently available and lets you assign one to your plan as a starting point for testing.

Note that while a particular build number is assigned to the plan, that build number is displayed on the Run Tests activity and stored with the results of any tests that are run.

You can access the Assign Build activity directly from the MTLM Testing Center by clicking Track and then clicking Assign Tests.  Use this activity to monitor the availability of new builds, view associated TFS work items such as resolved bugs, tasks and requirements, and evaluate whether and when to take a new build into test.

In particular, you'll want to know when the requirement for which you've created a test suite is available for testing with a particular build.  The developer performing a check in using Visual Studio Team System 2010 has the opportunity to add a comment to a change set:

The developer can also associate bugs, task or other work items, and as a best practice should associate any tasks resolved by the check in:

This change set information surfaces in the MLTM Assign Build activity when the new build is available:

You now have the opportunity to drill down on the associated task items and the requirement they link to, and can open the detailed build and change set information, if needed, to ascertain whether the requirement is indeed ready to test.  Having made that determination you can click Assign to plan to take the new build into test. 

As a final step, return to Testing Center > Plan > Contents and change the state of your requirement-based suite to State: In progress.

Now your requirements suite is open for testing in the Run Tests activity.  The assigned testers can begin running the tests in the suite.

No More Missed Requirements

To recap what we've accomplished in this scenario with Microsoft Test and Lab Manager (MTLM):

• We created a test suite based on a new requirement
• We prepared the suite for testing by --
  • Authoring and adding test cases
  • Ensuring adequate coverage of test configurations
  • Assigning tests to individual testers on the team
• With the suite ready to go we monitored information about new builds
• And finally kicked off a round of testing when the requirement became available

This type of workflow brings the testers into earlier and stronger collaboration with the people in their org who are authoring and implementing requirements, thereby increasing the impact, effectiveness and efficiency of the test org.

Please give this a try with your Beta1 build and let us know what you think.  We'd love to hear your feedback.

Thanks and Regards,

Michael Rigler

The Evolution of the UI Design of Test and Lab Manager

As many of you know, Team test has been developing Test and Lab Manager, aka codename Camano, in an Agile Manner.  Below is a description of how the UI design for Camano has evolved as design assumptions were tested and feedback was received.

Starting out

After doing extensive research on the testing market, we were fairly certain that in the next release, we wanted to create a product targeting Generalist Testers (70% of the testing market) who are uncomfortable with the amount of options available to them in Visual Studio. We were however fairly uncertain on the exactly which features to include in the product all the scenarios which we would have time to handle.

Being an Agile team who was open to change along the way, we decided to start designing and developing the product with the understanding that change would come along the way.

Iteration 1: Test Case Management inside Visual Studio

To get going on the project, while our other release (Visual Studio 2008) was winding down, we decided to create the basics of testing tools inside Visual Studio. Our thinking was that we wanted to get some tools around Test Case Management in the box so we could leverage the infrastructure which we had created in future sprints.

Below are a couple of shots of how we envisioned doing Test Case Management inside Visual Studio.  The first shows managing configurations, the second shows authoring a test case (which is a work item). The mockups below were created by Moneta Ho.

Iteration 2: Creating codename Camano

As more research studies became available and we learned more about our target market's likes and dislikes, the team decided to create a standalone environment for generalist testers and to scrap the abilities to do TCM in Visual Studio. Our studies had told us that our target audience was not extremely technical and was a little overwhelmed by Visual Studio.  We hoped that in doing so, testers would feel much more comfortable using our tools for their testing.  The sprint of work that we did inside Visual Studio was not throw away work since the majority of it was merged into Camano later on.

UI Breakdown: Although, we were unsure which actual features would end up in the product, we envisioned that the majority of the tasks would fall in to 5 broad categories: Dashboard Information, Planning, Testing, Triaging Bugs and Reporting. To accommodate these groups, we chunked the UI into 5 groups: Home, Planning, Testing, Defects and Reports, with the perl on the left reserved for items which didn't seem to fit into any group.

Workflows: We wanted to make the product as simple to use as possible so we thought to guide the user through the workflow of various tasks. To this end, we modeled much of the UI after traditional tasked based UIs such as Microsoft Money, with the left portion the UI dedicated to guiding the user through various workflows via hyperlinks. Tasks which were specific to the activity displayed in the activity window would be displayed as "common activities," while tasks specific to whatever is selected in the page would appear under "selected activities."

Multi-tasking: As we felt that it would be quite common to work on multiple things in parallel, we introduced a work in progress area to the left hand pane. The area worked by like a stack in that when a user navigated away from an activity, the activity from which the user went was placed at the top of the stack.

Below are a few mock ups, created by Moneta Ho, of the initial design of Camano.  The first shows the broad breakdown of the UI, the second shows the UI with a few artifacts.

Iteration 3: Simple Tweaks

Problems

We conducted a round of usability studies with local testers. We took away the following details from the study:

• Many people did not understand where they were in the UI at any given point
• Many didn't understand why the contents of the left hand pane was changing all the time or find it useful
• Nobody used the work in progress area to switch between items, rather they preferred to just navigate to it directly

After observing users crying out in frustration in the usability study, we decided to make some tweaks to the Camano shell.

Solution

Since most users did not correlate between selecting an item in the grid and then selecting an action to be performed on it far away, we moved all actions that could be performed on items in a grid to a toolbar directly above the item.  Further, we created a toolbar to contain items for the page displayed in the Activities pane.  Together, this allowed us to get rid of the selected item activities and paved the way to getting rid of the left hand pane entirely later on.

As we iterated, we found that we really did not need the Perl in the top left hand corner in the UI. We therefore removed it from the product as well.

Below is a screenshot of the tweaks that were made, one with a blue theme, one with a dark theme.  Both themes were in implemented in the the product at various times.

Iteration 4: Major Overhaul

Problems

As more of Camano began to take shape and more users internally began to use it, we increasingly received feedback from almost all of our channels including TAP, MVPs, the SIG and internal users that the product was not easy to use.  In our effort to make something that would allow people to plan all the steps of their testing, we made something which exposed too many artifacts in the UI at one time, did not guide a user through any particular workflow and to some extent got in the way of a user doing their testing. Although we initially designed band aid solutions such as activities dedicated to streamlining the out of the box experience via a wizard,  filtering a tester’s view by various factors, and workflow diagrams built into the page, we felt we needed to do something more dramatic. 

Further, it was felt that our current UI: 

• Had too much space (around 40%) dedicated to navigation.  Much of the navigation was redundant with 2 or 3 areas dedicated to doing the same thing.
• Our history metaphor was not working.  Our work in progress area quickly filled with items which the user was not interested in navigating back to.
• Did not make it easy to perform simple tasks.  2 or 3 steps were needed to just perform simple things without the tool guiding the user through the process.

In a typical tester’s day, we began to understand that:

• Most testers would work within the context of a Single Test Plan.  Examining items that span multiple plans would be a 20% scenario.
• The majority of their time would be spent in 3 types of activities: Authoring Tests, Running Tests, and Examining Bugs

Solution

Object Model: After much discussion, between both server and client teams, it was realized that the object model on which Camano was based could be dramatically simplified if we always forced the user to plan their test cases for execution at the plan level rather than as stand-alone suites which could be used from plan to plan. 

Some of the initial sketches of this idea by Ed Glas and the server team were as follows:

User Interface: To assist with the issues mentioned above, we decided to make several changes to the UI.  Specifically, we decided to optimize the UI on common workflows, make it easier to access the most common activities, eliminate the left hand pane, change the history metaphor, add back and forward buttons to the UI and to force the user to work within a single plan.

By doing so, I think that we were able to naturally filter the perspective of the tester to the items they are interested in (their currently connected plan), and streamline workflows. With these changes made, workflows become much more natural since objects are always created in the context of a plan.

Below is a mock up of design, by Nigel Wolters, that shows the changes, followed by a screenshot of what was actually implemented.  Note that the amount of space dedicated to navigation was minimized and that when the user is in the “Tester’s Center,” the user’s perspective is filtered to the Test Plan to which they are currently connected.  Further, access to the most common areas (to author or run tests or look at bugs), are just one or two clicks away at any given point.

Iteration 5: Simple Tweaks

After making the major overhaul to the object model and accompanying changes in the UI, we started to get raving reviews.  Folks were particularly happy to have the full horizontal screen for their activity content and that they were able to get testing quickly out of the box. All of our feedback channels started telling us that the product felt much more polished and easy to use.

As folks in Visual Studio started to look at it, it was pointed out that the colors were not in sync with the new blue Visual Studio theme in VS2010.  We decided to do a simple update to make Camano blue in addition to making it a little sexier.  I’m quite happy with how it ended up.

Below is a mockup created by David Culberton in conjunction with Nigel Wolters. It is very close to how the Beta 1 product looks and feels.

Conclusion

As you can see, in the run up to the Beta 1 release of codename Camano, the product has been completely redesigned several times.   My hat goes off to the Agile process and management which has allowed us to respond to various levels of feedback to make design changes when appropriate. 

I should note how much of a team effort designing Camano has been.  Each design decision has been thoroughly scrutinized to make sure that the right trade-offs are considered and the right optimizations are made.  Nigel Wolters, David Culbertson, Moneta Ho, Mark Mydland, David R. Williamson, Ed Glas, Michael Rigler, Dominic Hopton, Euan Garden and myself (Naysawn Naderi) have have all been thoroughly involved in making Camano as easy on the eye as we think it is easy to use.

Along these lines, we are also incredibly indebted to our MVPs, SIG representatives and many other unsung feedback heros who have been invaluable in praising us when appropriate and holding us out on the ringer when we deserve to as well.  My thanks goes out to you for helping to shape the design of codename Camano.

- Naysawn Naderi

Links to online help for Visual Studio Team Test 2010 Beta1

Reposted after fixing some broken links: 

As you download Beta1 and explore your way around the product you may find these links useful. Happy exploring!! Let us know what you think of the product and the documentation.

Main point of entry for testing information:

http://msdn.microsoft.com/library/ms182409(VS.100).aspx

Quick Start Guide for Manual Testers to Get Up and Running:

http://msdn.microsoft.com/library/dd380763(VS.100).aspx

Managing New Testing Efforts with Team Test: (Overview of Test and Lab Manager)

http://msdn.microsoft.com/library/dd286594(VS.100).aspx

Defining your testing effort using test plans:
http://msdn.microsoft.com/library/dd286581(VS.100).aspx

Creating manual test cases:
http://msdn.microsoft.com/library/dd286715(VS.100).aspx

How to: Associate an Automated Test With a Manual Test Case and Run It:
http://msdn.microsoft.com/library/dd380741(VS.100).aspx

How to: Create Test Cases from an Assembly of Automated Tests:
http://msdn.microsoft.com/library/dd465191(VS.100).aspx

Setting Up Machines and Diagnostic Information to be Collected Using Test Settings:
http://msdn.microsoft.com/library/dd286743(VS.100).aspx

Creating A Diagnostic Data Adapter to Collect Custom Data or Impact a Test System:
http://msdn.microsoft.com/library/dd286727(VS.100).aspx

Running Manual Tests with Test Runner:
http://msdn.microsoft.com/library/dd286725(VS.100).aspx

Recording and Playing Back Manual Tests:
http://msdn.microsoft.com/library/dd286714(VS.100).aspx
 

Determining Which Builds Have Bug Fixes, New Features or Requirements:

http://msdn.microsoft.com/library/dd286586(VS.100).aspx

Recommending Tests to Run That are Affected by Code Changes:

http://msdn.microsoft.com/library/dd286589(VS.100).aspx

Ram Cherala, Principal PM, Visual Studio Team Test

Create Actionable Bugs

Now that Visual Studio Team System 2010 Beta 1 is released, let’s talk about some of the cool features it has in test area.  We will start with taking about the ability to easily create actionable bugs:

Create actionable bugs: The problem of no-repro is a big issue between developers & testers. Testers spend enormous amount of time finding and reporting a bug, but many of the bugs are rejected by developers because the information in the bug report is not comprehensive enough, or they can’t reproduce the bugs on their systems. Microsoft Test and Lab Manager (MTLM) goes a long way in solving this problem. Let us see how:

Once you create a manual test case  in Microsoft Test and Lab manager, you can run it.  Test case(s) will be loaded in Test Runner activity of MTLM.  Start by clicking on “Start Test”. 

You can follow the steps defined in the test case to test the your Application Under Test (AUT). As you test your app, mark the step in Test Runner as “Pass” as appropriate.  Once you find an issue in the application under test, mark the step as “Fail”. 

At this point testers normally open the bug form and start typing the repro steps etc.  However, as you are using the Microsoft Test and Lab Manager, let’s see how it makes your job easy.  In the bottom pane of the Test Runner, add any comments about the issue you found while testing this step.  Now let’s attach a screenshot to prove that you have actually seen the issue.  From the Test Steps tool bar, click on the camera icon & take a snapshot of the AUT (you can take a screenshot of the rectangular area/window/complete screen).  The screenshot gets automatically attached to the step.

Now let us submit the bug.  Click on the New bug button on the toolbar.

It opens up a TFS bug form.  Notice that some of the required fields are automatically filled for you.  Also in the bottom tab of the bug form repro steps have been captured (including information about what steps you had passed/failed). Any comments/screenshots are also attached to the step. 

The second column has detailed actions, open any section & see that it has captured detailed user actions (yes each and every UI action).  If you had configured video recording in test settings, you will see video links for every action in third column.  Now the developer can see exactly what was done in the application before the bug was found.

The System info tab has automatically collected information about the system on which you were running the app. 

The test case tab has a link that has been automatically created between this bug & the test case.

The “Other links” tab has many other useful links (depending on the test settings).  E.g. it has the links to the action log file, video of testing, historical debugging log, event log etc.

You can add title to the bug and save it.  That’s it; you have created a rich actionable bug and the chances are that it won’t come back to you as non-repro. 

Visual Studio Team System 2010 Beta 1 is released!!

Today, we announced the broad availability of Visual Studio Team System 2010 Beta 1 and it is now available for download. We encourage you to download, evaluate and provide us your feedback. This is the culmination of the work done by the team and we are eagerly looking forward to your feedback. This is your opportunity to influence the direction of the product through the next Beta and release of the product, so do take advantage of this.

Please visit Visual Studio Download Page to find out how to download the beta, how to share your feedback in the forums and how to file a bug.

Thanks,

Team Test team

P.S: The team members will be blogging additional details of the various features in the upcoming weeks and we encourage you to visit this site frequently to check out the updated information.

Using Instrumented Builds to Analyze Testing and Find Dead Code

As part of my weekly duties since joining the Quality Tools team, I have been providing code coverage analysis to the team, ensuring that our automated tests cover a required percentage of our shipping code.  This has been a fun exercise for me, allowing me to both gain a better understanding of what code coverage means from a quality standpoint, as well as flexing my knowledge of PivotTables and charts in Excel to provide user-friendly results.  Performing basic code coverage of automated tests (such as unit tests) can be done by following the instructions at the Quality Tools blog at http://blogs.msdn.com/vstsqualitytools/archive/2005/06/08/426979.aspx.  A further dive into the details of Code Coverage can be found at https://blogs.msdn.com/ms_joc/articles/406608.aspx.

For this particular exercise, rather than analyzing the coverage of automation tests, we opted for a different approach: instrument a build and collect coverage information as the tester performs ad-hoc/manual testing.  The process (in a nutshell) looks like:

·         Build the Application Under Test (AUT)’s binaries.

·         Instrument the binaries that you wish to collect coverage on. (using vsinstr –coverage)

·         Create a Setup Package containing the instrumented assemblies.

·         Each Tester installs the Setup Package, and turns on data collection (using vsperfcmd)

·         Each Tester performs a pass through the AUT.

·         When vsperfcmd is shut down, a .coverage file is generated.

·         Analyze the results of all of the .coverage files.

·         Merge the separate .coverage files into a single result.

Now, once you’ve accomplished the work above, you’ve got yourself two important artifacts:

·         One coverage file for each tester’s work.

·         One coverage file that merges all tester’s work.

The two of these can used to be answer some interesting questions about both the testing process, and the application under test.

·         What classes/methods were never covered?

o   How can you tell?  Look for classes/methods with 0% coverage in the merged result.

o   What does this mean?  This generally indicates either scenarios that aren’t being tested, or “dead code” that, while covered by the automated tests that were created when it was checked in, is no longer used by the application.

o   What else can I tell from this?  What kind of classes are getting 0% coverage?  If there are lots of exception classes that are uncovered, this may indicate that testers aren’t evaluating error conditions (ex: opening a file from the application with invalid data).  You may also find that there was an application state (such as docked, minimized, or placed in the system tray) that is required to test certain code paths.

·         Is each tester testing the same things?

o   How can you tell?  Look for classes where the coverage for each tester is reasonably similar to each other and the overall merged result. 

o   What does this mean?  If the merged result is higher than the top individual tester coverage, this indicates that each tester exercised different code.  On the other hand, a class where all testers cover ~70%, and the merged result is also ~70%, this indicates that all testers were covering the same scenarios and code paths.

Workaround to upgrade VSTS 2005 Test Projects in Visual Studio Team System 2008 – Development Edition

Some users report an issue in Visual Studio Team System (VSTS) 2008 – Development Edition. In the Development Edition, they failed to upgrade VSTS 2005 Test Projects. We have verified this is a product defect only in  VSTS Development Edition and Visual Studio Professional Edition. We understand the impact of this issue, we are working to produce an update to resolve it. In the meantime, if you are looking for a workaround, please look at this article that we posted to our MSDN forum: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2534717&SiteID=1

We encourage you visit our forum often – ask questions and give us feedback, we will make our product better.

Thanks,

Christine Zhao

Visual Studio Team System Test

What's new in November 2007 CTP: Automation for Navigation

Manual Test Runner (MTR) is one of the key portions of Rosario with which we are hoping to capture the imagination of the manual tester. Read more...

AfN-Automation for Navigation is a feature that we are adding to MTR to enrich the testing experience, to facilitate the tedious manual testing! It helps the tester to automate the repetitive and time consuming parts of a manual testcase so that more focus can be laid on testing the functionality of the application, thus increasing efficiency.

What happens exactly is that when the tester does manual testing for the first time, we start a background recorder that captures all the manual actions and creates a recording strip which is essentially an XML file containing all the actions performed and is exposed to the tester. The strip has markers inserted at all those points where a step was marked in the middle of the recording. Here is a snippet of the XML file that’s generated:

<?xml version="1.0" encoding="utf-8"?><UITest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Name="" Id="83a5f887-245b-4a71-9d5e-20def6ebfd6d" Version="1.0">

<InitializeActions />

  <ExecuteActions>

    <TestStepMarker MarkerInformation="">

      <StepId>0</StepId>

    </TestStepMarker>

    <MouseAction UIObjectName="CurrentUIMap.ItemWindow.RunningApplicationsToolBar.VshostexeButton">

      <ModifierKeys>None</ModifierKeys>

      <Location X="53" Y="13" />

      <WheelDirection>0</WheelDirection>

      <ActionType>Click</ActionType>

      <MouseButton>Left</MouseButton>

    </MouseAction>

    <MouseAction UIObjectName="CurrentUIMap.ApplicationCustom.Run1202PM24MicrosoftMenuItem">

      <ModifierKeys>None</ModifierKeys>

      <Location X="68" Y="15" />

      <WheelDirection>0</WheelDirection>

      <ActionType>Click</ActionType>

      <MouseButton>Left</MouseButton>

    </MouseAction>

    <TestStepMarker MarkerInformation="">

      <StepId>2147483647</StepId>

    </TestStepMarker>

</ExecuteActions>

Whenever a step is marked, a visual indiactor immediately appears in the same row as the step. Random marking of steps is allowed but markers are not inserted in all cases. If a step is marked in the forward direction without doing any AUT actions, the last inserted marker moves to this step. If a step is marked in backward direction, no marker movement occurs, only tooltips advising to mark steps in order, appear. An implicit start marker is inserted when a testcase is started and an implicit end marker is inserted on ending the testcase.

The tester will have a choice of switching on/off the background recorder, the video recorder and the action log and also to exclude applications to be recorded. Throughout the testcase execution, this information is displayed to her.

Fig 1: The tooltip shows the default recording settings. On clicking on the ‘Edit’ link, the Recording Settings dialog pops up

Fig 2: The Recording Settings showing available choices of loggers and AUTs

Fig 3: After the TC is started, hovering the mouse over the status bar shows which loggers are on and what all AUTs are being recorded

So when the same testcase is opened again, the UI shows a visual indication at the points where a marker was inserted. The tester has a choice of skipping some sections by playing back between a chosen set of markers to speed up her navigation through the testcase or overwrite the existing markers and recordings. In the end, the original and the new strips are merged and a choice is given to the tester to either keep the original recording or save the new merged recording.

Fig 4: On opening the TC again, the available markers are displayed which can be played back or re-recorded

Fig 5: After re-recording, when TC is ended, tester is given options to save/discard new markers

With so much functionality, MTR is sure to be a boon for all manual testers!

Microsoft ® Visual Studio® Team System code name “Rosario” November 2007 CTP: Team Test

What’s new from Team Test?
We’re excited to show some of the features in our Test Case Management Solution! It’s all about creating a simple product that will be easy to use and will make your life easier. If manual tests are part of your daily work, and the process includes writing your test cases in a word processor, excel, or any other tool, we have something cool to offer. If you want to keep track of your projects in an easy way, knowing what is left to do and what has been done, we have something for you!

How does it work?
You are a test manager. Your task today is to create a test plan for the following week. The test plan will contain the test cases that testers need to run in order to sign-off and release any given product that you’re currently working on. Your day could look like this:
You get to your office, and open your test case management solution:

You can see the several Activity Centers: Home, Planning, Testing, Defects, and Reports. You decide to go to Planning, in order to create a Test Suite and add some test cases to it. A Test Suite is just a hierarchy of folders in which you will host your test cases:

After you create your test suite and have added test cases to it, you create a test configuration, which will be the configuration to be used when running your test cases (in our example, it will be on a VISTA OS in the English language).

Now that you have your test suite with test cases in it and an actual test configuration, you can go ahead and create your test plan. A test plan will contain one or more test passes. A test pass is just a combination of a test suite with a test configuration.

The test plan is done, and your testers are ready to start running their manual tests! By the way, if you are curious about what a test case looks like, here it is:

Your  testers can now go to the Testing Activity Center, choose the test plan that you just created, and start running the test cases assigned to them by using the Manual Test Runner. Results can be published to the server and seen in the test case management solution right away!

We want to hear from you!
Please leave your comments or any kind of feedback. We are in the early stages of development, and we want to hear you!

Download the CTP here!

Thanks,
David Gorena Elizondo

Come chat with the Visual Studio Team System group – December 5th

Join members of the Visual Studio Team System product group to discuss features available in Team Foundation Server, Team Suite, Architecture Edition, Development Edition, Database Edition, and Test Edition. In addition, discuss what's new for these editions for Visual Studio 2008.

We will be holding two sessions:

Join the chat on Wednesday, December 5th, 2007 from 10:00am - 11:00am Pacific Time. Add to Calendar | Additional Time Zones

                -and-

Join the chat on Wednesday, December 5th, 2007 from 4:00pm - 5:00pm Pacific Time. Add to Calendar | Additional Time Zones