Greetings! It has been a while since the last post – I and my team have been preoccupied with work of course, wrapping up the Beta 1 release of VSTS 2010 and finishing up the remaining features for the Team Test and Lab Management products. The team also took a well deserved long weekend off to celebrate the accomplishments, and to gear up for the drive for the next Beta scheduled for later in the year. I am really looking forward to this phase – shipping world class products is always an exhilarating experience, and the end-game always pumps me up even more!
The weather too is shaping up nicely and that’s going to add to the zest as well. The two months of brutal summer is over in Hyderabad, and the monsoon is setting in. It is right now a bit humid, but the temperature on this Saturday morning is a nice 80 degrees, down from the 100+ even a few days back. Team members in the USA are enjoying the start of summer there, and the temperature in Redmond, WA is 73 degrees, and in Raleigh, NC is 76 degrees. So, that’s great weather all around!
The Beta 1 feedback is starting to come in. The feedback is positive. The value props, scenarios, and the feature set are resonating well. The team members are monitoring the feedback forums looking for cases where people might be having problems – those are areas we want to address and fix before we release.
I want to take this opportunity to point you to two specific blog posts from the product team.
The Lab Management team is running a sequence of tutorial like posts here. I talked to you about the high level concepts of the product in my last post. Since then, the team has posted two articles – one on creating Lab Environments, and another on automatically deploying bits on to the environments. Give those a read. The product is also getting good coverage at the www.virtualization.info site – one of most popular sites for the virtualization industry. Check out this post.
On the Team Test front, Brian Keller has just posted a video (about 25 minutes) where he talks about and demonstrates our Manual Testing solution. You can find the video here. You have read about the product offerings for the Generalist or Manual tester, and this video will give you a good view of those capabilities.
If you haven’t tried out the Beta 1 bits yet, I strongly recommend you do so and let us know your own opinion. You can find the information about the Beta here.
Do write back about your experience with the Beta 1 bits, and let me know if there are specific areas or capabilities that you would like to learn more about. I will be happy to talk about them in the future posts.
Enjoy the weather, where ever you are.
Cheers!
A lot is going on. The VSTS 2010 Beta 1 bits are out (as I had noted in my last post) and with that the interest in the product, both internally and externally, is peaking. The product team is busy interacting with customers and getting their feedback. There are also few feature modifications that are being worked on. So, while the Beta 1 is released, there is still a lot to be done. I will continue to walk you through the feature and functionality that’s in the product. As you install and play with the bits, do post your comments on areas of the product that you’d want additional information on.
The weather too is improving – the hot summer of Hyderabad is almost over and the monsoon is right around the corner. The clouds are building up and the oppressive heat is less now. The cooler climate will provide that extra burst of energy that we are all eager to put in on the product front and customer-connection front ;-)
One of the coolest products we are introducing with the VSTS 2010 release is the Lab Management product. This is an integrated solution that brings virtualization to the heart of application lifecycle management space. Here’s a glimpse of the capabilities that you get with the product:
- Create libraries of virtualized multi-tier test configurations really quickly
- Automatically deploy new builds of your application to these environments
- Seamless integration of our dev and test capabilities with the virtualized environments
- Take the “no-more no-repro” theme to the next level by leveraging snapshots
In this post, I will cover the high level concepts of the Lab Management product.
Lab Management High Level Architecture
The diagram below shows a high level architecture diagram for Lab Management.
On the server side, Lab Management service is one of the many services running inside Team Foundation Server (TFS). This is what makes the Lab Management solution unique for software testers and developers. Now you can map your lab resources, such as, hosts, virtual machines and storage to Team Project Collections and Team Projects; thus aligning lab hardware needs with the business needs for the projects you are working on.
The lab management service in TFS uses System Center Virtual Machine Manager (SCVMM) for management of lab infrastructure and provisioning of virtual machines across multiple virtualization platforms. You get a copy of SCVMM with Lab Management.
On the client side, the “Microsoft Test and Lab Manager” tool (earlier known as “Camano”) is still the tool to manage your virtualized assets. This is a Windows Presentation Foundation (WPF) based rich client that allows you to define test plans, test suites, test cases and run them in physical or virtual environments.
Basic Concepts
Similar to Internet, hardware virtualization is a disruptive technology that is changing the face of computing. Therefore, it is important to understand some of the basic concepts around virtualization and how these are used in Lab Management to understand this paradigm shift.
Virtual Machine
A virtual machine (VM) is a computer within a computer, implemented in software. A VM emulates a complete hardware system, from processor to network card, in a self-contained, isolated software environment, enabling the simultaneous operation of otherwise incompatible operating systems on a single physical computer. Each operating system runs in its own isolated software partition.
Virtual Machine Snapshot
A virtual machine snapshot is a file-based snapshot of the state, disk data, and configuration of a VM at a specific point in time. A VM snapshot is similar in functionality to laptop hibernation state with the additional flexibility that a VM supports multiple snapshots. You can roll back the VM to any of the previously taken snapshots and continue operating from there. The picture below shows a snapshot tree for a Hyper-V VM.
Host
A host is a physical computer that hosts one or more virtual machines.
Host Group
A host group is a custom group of virtual machine hosts, which an administrator can create in SCVMM for ease of monitoring and management. Host groups can be used to allocate and determine the resources reserved for various team projects. For example, an administrator could create a host group named “Global Bank Hosts” for a team that works on “Global Bank” project and bind it to the corresponding team project in Team Foundation Admin Console.
Library Share
One of the beauties of virtual machines is that you don’t need to tie up a host if you are not actively using a VM. You can store it on a disk and bring it back to life on a host in a few minutes. SCVMM supports the concept of a library share where you can store virtual machines and other resources, such as, ISO images. The library share is nothing but a file share that is accessible to all the hosts. Similar to host groups, you can create multiple library shares for ease of management. For example, you could have a library share for storing pristine or golden OS images. Another library share could be used for storing VMs that have various application software components installed.
Environment
A typical multi-tier application consists of multiple roles, such as, Database Server, Web Server, Client, etc. Each role could be running on one or more computer. You could also have multiple roles running on a single computer. An environment is a set of roles that are required to run a specific application and the lab machines to be used for each role.
Managing environments for multi-tier applications is an error prone task today. Replicating the same environment at same or another site is even a bigger problem.
Lab Management surfaces environments as a first class entity. See the picture below for an example. It shows a list of environments running in the lab. The “DinnerNow Integration Testing 2” environment consists for two virtual machines, named, DinnerNow-Web and Dinner-SQL.
Environment brings with it ‘strong’ group notion. That is when you do an operation on an environment, such as, start, stop, take snapshot, etc., that operation is applied on all the virtual machines that are part of the environment. In the next blog, we will show you how to create, start, stop, delete, save, take snapshot and interact with rich self-documenting virtual environments using Lab Management.
Online Documentation
As you install the products, please refer to our online documentation which will give you lot more detailed information on the concepts and the capabilities. Peter Bateman and our documentation team would love to hear your feedback on the docs too!
Next
The above covers the basic concepts that you need to understand in order to leverage the Lab Management product. I will take you through some of the high level features in my next few posts…
The VS 2010 Beta 1, the VSTS 2010 Beta1, and the .NET Framework Beta 1 is now available for all to download. Here are a few links that you will find useful:
The Beta 1 download site (the homage page this beta program)
The VSTS Beta 1 Suite Installer (includes the Team Test client components)
Team Foundation Server download
Lab Management Beta 1 download
Lab Agent download
Test Agent download
Note that the VSTS 2010 Suite Beta 1 bits includes all of the client products for Visual Studio Team Systems – that is, you will get the product for Developers, Architects, and Testers. The Team Foundation Server (TFS) bits provide the server side for the above client components, and the Lab Management product is for the physical machines which you want to devote to setting up a virtual lab. So, the above downloads will enable you to install and play with the Test Product and the Lab Management products that I have been talking to you about.
My colleague Brian Keller has created this nice video that will help you with the installation and configuration of the VSTS 2010 Suite Beta 1 bits.
I really encourage you to download and try out the product, and would love to hear your feedback. I am also including the official channels for getting your feedback and issues to the product development teams.
Beta 1 Forums
Beta 1 Connect Site (for bug reporting)
Finally, I will point to a couple of other important links that you will find useful as you try out the products:
Online documentation for Team Test (docs for the Team Test features)
Lab Management Setup Guide download (setup/config of Lab Management)
There – I think you now have all the information to get started with the bits. I will now return back to my series on testing, and in the new few posts focus on the Lab Management product. I hope you are enjoying this series, and stay tuned …
The Beta 1 download of VSTS 2010, along with Beta 1 bits for Visual Studio 2010 and .NET 4.0, are now available here.
MSDN subscribers can download the beta today, while the rest will have public access starting this Wednesday. I’ll post the public links for non-MSDN subscribers on Wednesday.
This is a very exciting milestone for us. A lot of innovative thinking and hard work has gone into these products. While we have released CTP bits earlier, nothing like getting a fully installable beta into your hands!
I am also very excited that now you will have an opportunity to try out the features as you follow along the topics I have been blogging about in the Testing and Lab Management areas. I’d love to hear about your own hands on experience with these features. I am happy to take feedback on my blog, but also do use http://connect.microsoft.com for bug reports, and visit the forums for detailed discussions.
Do also take a look at Jason Zander’s blog on this announcement – he gives you a highly level view of all of the features areas in the VS 2010 Beta!
Try out the bits and have fun!
You have been following along the innovative set of features and capabilities that we are providing with the Test product in the VSTS 2010 release. In this post, I want to explore with you the rich and flexible options you have for reporting on various aspects of test data and test results, and its management.
The richness and flexibility really comes from the fact that our test offerings are architected on top of the Team Foundation Server (TFS). One of the major value propositions of TFS has always been rich reporting across a broad set of data. In the 2005 and 2008 editions of VSTS we only included a minor set of data around your test artifacts, which primarily enabled you to report on the test results that you had run as part of your build. However, in the 2010 release, all of the great features you have read about in Microsoft Test and Lab Manager (earlier referred to with the code name “Camano”) are backed by the new Test Management services on Team Foundation Server 2010 and we have enabled you to report on the very rich set of data that you both capture and create during your testing.
One of the best ways to harness that data is to use the powerful features in Microsoft Excel in combination with SQL Server Analysis services to mine it. In this post I will help you get started with a couple off basic reports that you can view on a daily basis to track your testing effort.
Test Plan Progress
I will call the first report we will build, the Test Plan Progress. It will enable you to track the progress of one or more test plans across a period of time.
To get started you will need to insert a pivot table into a worksheet and connect it to your data warehouse. By default the warehouse will be installed on our TFS data tier.
Start by clicking on the Insert tab and selecting the PivotTable button:
In the Create PivotTable dialog you will select “Use an external data source” radio button and click on “Choose Connection”
From the “Existing Connections” dialog you will click the “Browse for more” button and from the “Select Data Source” browser you will select “New Source” at the bottom and will be presented with the “Data Connection Wizard”
Select “Microsoft SQL Server Analysis Services” as the type of data source you want and follow the wizard.
After you have entered the name of your analysis services server (which is a typical installation will be the same as your data tier), you will have the option to select the OLAP data base and cube that you want to connect to:
If you are using SQL Enterprise you will have the option to pick which perspective you want to use, however, if you are using SQL Standard edition you will only have the option to connect to the Team System cube. If you have the option to use a perspective you should select the Test Result perspective.
Once you have established the connection to the server you will need to add the following fields from the Pivot Table Field List to the Report Filter section of the Pivot Table editor.
- Team Project.Team Project Hierarchy
- Test Result.Iteration Hierarchy
- Test Result.Area Hierarchy
- Test Plan.Test Plan Name
(Also take the time to browse and view the rich set of fields that are exposed in pivot table field list!)
After you have added those fields to the Report Filter section, you will want to add the Test Result.Outcome field to the Column Labels section, the Cumulative Point Count measure to the Values section and the Date.Date field to the Row Labels section. After you initial setup your sheet will look something like this:
Once you have the basic data setup you can filter it down so it makes sense for you. In this example we are going to start by filtering out some of the Outcome’s that we don’t need to report on. In addition you might want to filter to only show results for a single iteration.
Select the dropdown next to the Column Labels field in the PivotTable and un-check the “None” and “Unknown” values from the dropdown.
Next right-click on the PivotTable and select “PivotTable Options” from the menu. In the “PivotTable Options” dialog select the “Totals & Filters” tab and un-check the “Grand Totals” options and press “OK”.
After you have setup your PivotTable with the data you want and the filters you want you can easily create a chart for our data.
Select your PivotTable and select the Area chart from the “Insert” tab
Excel will automatically generate a chart based on the style you have selected and after adding a title you have something that can be easily read by a variety of people
In the chart above we can see the progress of all of the test plans in our project collection. In an ideal trend we would see a steady increase in the number of Passed tests and a steady decrease in the number of Failed, Never Run or Blocked tests.
The data is organized in such a way that you are always looking at the most recent outcome for a given test case on a given day.
For example: If I run Test Case 1 on 4/30 and it passes and I don’t run it again, when I look at the data on 5/13 it will still show Test Case 1 as passing.
Test Results by Build
Another graph that you may be interested in seeing is your test results on a build by build basis.
To start out you will create a new worksheet in your workbook and insert a PivotTable with a connection to your warehouse in the same way you did at the beginning of this post.
For the Test Results by Build report you will select the following fields:
For the Report Filter, choose the following:
- Team Project.Team Project Hierarchy
- Test Result.Iteration Hierarchy
- Test Result.Area Hierarchy
- Build.Build Definition Name
For the Column Labels, select:
For Row Labels, select:
And finally for Values, choose:
- Cumulative Build Result Count
Once again you will want to filter out the “None” and “Unknown” outcomes by click in the “Column Labels” dropdown and un-checking the values.
After you get your PivotTable setup you are ready to generate your chart. For this particular report I think a stacked column chart works best.
In this report we can see the outcomes for the tests that were on a given build. In this case we have the full set of builds for a build definition. If this was our nightly build we could compare the results side by side to make sure we are not seeing any regressions.
Conclusion
Creating reports by simply dragging and dropping values into the PivotTable let you quickly explore your data. You can try different way for grouping and filtering the data as well as applying different type of charts to see what works best for you.
In my next post on reporting, I will discuss the dimensions we provide in greater detail so you can better understand the types of scenarios you can report on.
This is an exciting day for me and my team. We have just announced the official names for the 2010 Test Products that I have been talking about in this blog. You can see the announcement here by Jason Zander.
Let me elaborate on the announcement in the context of the component block diagram that I have shared with you earlier and am including here again for reference with a color coding to help you understand the product offerings.
All of the functionality above will be released as the following three products, which sum up a great line of offerings:
|
Visual Studio® Team Test 2010 Essentials |
Support for the generalist tester including the ability to manage test cases and manual/automated test execution. Installs as a scaled down product for easy access on test machines.
We expect that there will be plenty of cases where generalist testers will want to manage and execute test cases without installing the entire Visual Studio system on each machine. This then is the product for you.
The components included here are the Microsoft Test Runner, the Microsoft Test and Lab Manager application, Reporting, and Data Collectors (all of the components shown in blue in the above diagram). You also get a TFS Client Access License (CAL) |
|
Visual Studio® Team Test 2010 |
Support for the specialist tester including Web and load testing capabilities in addition to the ability to create automated test suites. Executes in the Visual Studio environment for test professionals and comes with Microsoft Test and Lab Manager.
In addition to all of the components from the “Test Essentials” product, this product will include Unit Testing, Coded UI Test, Web Test, and Load Test components (all of the components shown in green in the above diagram). This too of course comes with a TFS CAL. |
|
Visual Studio® Lab Management 2010 |
Support for creating virtualized environments with snapshot capabilities. You can now execute your tests using the lab capabilities and save the state later for both development and test usage. (The component shown in orange color). |
Please note that the main client side application for the products – which I have been referring to with the code name of “Camano” in this blog series, is now officially named as Microsoft Test and Lab Manager.
These components will all be part of the Visual Studio 2010 Beta 1 release that is just round the corner.
I am eager to talk to you about the few remaining components and capabilities I haven’t talked to you in great details yet. Next up is a post on our reporting capabilities. I will then follow up with a few posts on the Lab Management Product.
Stay tuned, and your feedback and comments are most welcome!
So I have been incommunicado for a while, though for a specific reason. I was out for a week, at Cape Code, Massachusetts attending the Power Lab – an executive development program. The program is organized as a fully immersive societal experience. The society has a class hierarchy – Elites you own almost all of the resources and jobs, Middles who manage the jobs for the Elites, and Immigrants who practically own nothing. I was “born” as an immigrant in the society, and it was an absolutely incredible experience. Mind you, this is not a regular gig where the program runs from 8 to 5 and you have nice cocktail dinners in the evenings. In this program you truly live your role 24 hours a day and for multiple days at a stretch! It was a great learning experience – literally living through the many experiences around issues of power and leadership, and confronting several situations which I’d normally consider off-limits for me. During the week I had no access to my laptop or cell phone, and hence was completely cut-off from work and my real-life society.
You can find out more about the Power Lab at www.powerandsystems.com.
This post was just a side-show. I will get back to the “center ring” and continue with the series on testing soon.
Next week is also TechEd2009 week – the main conference is in Los Angeles from May 11-15. There will be several sessions on Visual Studio Test Systems, Team Test, and the Lab Management products. In India, the main event will be in Hyderabad on those same dates. I will once again present the “Next Generation ALM Tools from Microsoft – A Lap around Visual Studio Team System 2010” session. We will also have a “Technology Tent” to show-off all of the cool stuff that the Test and Lab Management teams have been working on. Hope to see some of you there …
Tomorrow I am travelling to Bangalore, India to key note at the “Great Indian Developer Summit 2009”. You can find details about this conference here. This probably is the largest .NET event in this part of the world!
My key note is at 11:40am on April 22. Here’s a synopsis of my talk:
Next Generation ALM Tools from Microsoft – A Lap around Visual Studio Team System 2010
“You want to build great software, and to do so you need more than skills than just coding. A full team comprising of business analysts, architects, developer, testers, project managers, and operations manager needs to come together to create the magic of great software worthy for today’s enterprises. To make this happen you need a development platform and tools that seamlessly, transparently, and effectively bridge the requirements for all the roles and orchestrate a great development experience. Microsoft Visual Studio Team System 2010 introduces a new and improved tooling platform that addresses the needs of the entire application development lifecycle, enabling teams to build great software. In this demo filled session you will get a great overview of this platform and tools.”
I am excited about the opportunity of meeting with the developer community and demoing the VS2010 bits!
Hope to see some of you at this event!
This post finishes up the two part topic on Diagnostic Data Adapters in VS 2010, and how they help developers and testers to work together. In the last post I talked about the “Test Impact Collector”, and today I will talk about the “Diagnostic Trace Collector”.
In an earlier post, I had talked about the “No Repro” problem that plagues the Industry today. This is that frustrating, almost reflexive, resolution to bug reports by developers – or perhaps the only option left when the test team didn’t provide enough information in the bug report to make heads or tails of what they were trying to do with the product. The VSTS 2010 product provides a variety of mechanisms and dials to combat this problem. The Diagnostic Trace Collector is another one of those, and can turn any test case into a data record complete enough to support time travel debugging! Any tester can run a manual test with the Diagnostic Trace Collector enabled and file a bug. From that bug report the development team will have enough data about the state of the system under test to walk backwards and forwards from the point of test failure in a debugger located on their own machine.
This is a very powerful feature! Visualize this picture. The Developer simply clicks on a log file attached to a bug that’s assigned to him, and this launches Visual Studio into a debug like state that allows him to inspect call stacks, events, function parameters and the like. The magical part of this of course is that you get the experience of Visual Studio debugging an application that is no longer running on the developers work station or the testers machine, yet it literally is the state of the application at the time the defect was filed, and the developer can now trace backwards in time to see how the defect occurred!
Below is a screen shot that shows the configuration options for the Diagnostic Trace Collector.
This Collector is a natural to enable when the development team asks for more information regarding a specific bug report. While the default settings collect events raised by the applications under test, these applications can also be instrumented to facilitate tracing method level data to provide a more detailed picture of the state of the system during the test. The result file will be automatically included with any bug filed from the test runner when the test settings enable the collector, and will appear in the result file for any test run from Visual Studio with similar settings. Most of our diagnostic data adapters are available to tests executed through Microsoft Test and Lab Manager as well as the Visual Studio integrated development environment.
Double clicking the log file in a test result or a bug report will open the Trace Debugging Log viewer, shown below with a display of active threads running user code. This viewer is the access point to historical debugging based on the data collected during a manual test, for instance.
So, instead of a No Repro, any tester can now create a debuggable record of the test they performed with no special technical requirements necessary to collect this data!
Now that we’ve seen what a couple of these diagnostic data adapters can do we’re ready to go about authoring one of our own. We’ll tackle that challenge next time. The great part about our whole Agent-Controller and Diagnostic Data Adapter system is that teams can use the diagnostic data adapters we include or write their own to get all the utility they need out of their agents without installing the Visual Studio 2010 product on the agent or controller machines. It is as simple as dropping the new adapter assemblies into the correct folder and updating the test settings to make use of the new functionality – or simply using the out of the box adapters as they are. We’ll be going through these details and more next time, please do ask any more questions you may have as I want you to be as excited about this topic when you get your hands on the bits …
Last time I talked about remote execution and introduced the concept of Diagnostic Data Adapters in Visual Studio 2010. In this post and the next, we will jump right into working with two of our favorite Diagnostic Data Adapters: the Test Impact Collector and the Diagnostic Trace Collector. Both of these components have a part to play in how developers and testers interact in Visual Studio 2010, making it easier for each discipline to unlock the potential of the other. While a key pieces of the diagnostic data adapter story is the freedom for teams to create their own adaptors and tailor their data collection to better suit their needs, we plan to include several diagnostic data adapters that we feel will be useful to many teams and projects right out of the box. The remainder of this post (and the next one) will go into detail about these two great diagnostic data adapters and how they help address a couple perpetual sticking points between developers and testers: “What did you test?” and “No Repro.”
The first point we address is the eternal question “What did you test?” This question is often asked by developers of testers, testers of each other, and management of everyone whenever a bug makes it into your customer’s hands. There are many ways different groups attempt to answer this question, but the Test Impact Collector aims to bridge the gap between developers and testers to show you what needs to be tested and which tests need to be run. In the world of development and unit tests it is a fairly straightforward effort to enable code coverage and see what parts of the code have been called by your tests. The hassle of collecting coverage data, and providing consistent coverage through manual tests, is beyond the expectation for many testers. The development team often knows only by feature area what the test team has been up to, and vice versa for the test team to know what the development team has been trying to fix. The Test Impact Collector serves a vital role here by identifying for development which code changes will or will not be exercised by the test team’s existing test catalog, while it also helps testers identify which tests from that catalog they need to run in order to validate the new behavior their developers have included in a given build.
Configuration of the Test Impact Collector is very simple. Manual testers using Microsoft Test and Lab Manager (codename Camano) will simply run tests according to the plan configured by their test manager or a specialist tester. Configuring the test plan simply requires creating a Test Settings for the tests to use and marking the appropriate diagnostic data adapters for use during a test run, as seen above. Each diagnostic data adapter exposes configuration options. For the Test Impact Collector there are two lists that can be used as either white or black lists of processes and modules to either include or exclude from instrumentation during the test run. The Test Impact Collector will collect the names of methods exercised during the tests. This allows the team to focus their efforts on just their code and reduce the number of modules that will be instrumented at the start of the test case, or, to eliminate modules that are known to not be of interest during the test.
This diagnostic data adapter allows Visual Studio to inform developers when their code changes will not be covered by the test team’s existing tests. Knowing this before the change is made will allow the development team to take extra care in reviewing their work or warning the test team before new or unexpected code shows up in the application under test. From the perspective of the test team, the Test Impact Collector helps answer the question “Which tests do I need to run in order to test the new changes?” Collections of test cases may be grouped by feature areas, but these sets may be much larger than necessary to test small changes within those areas. Rather than expecting teams to run all the tests within a given area, the Test Impact Collector helps teams reduce the number of tests they must run without leaving out any tests that do exercise the newly changed product code.
The above battery of test cases to run can be reduced to the below list with the help of the Recommended Tests option within Microsoft Test and Lab Manager.
The Recommended Tests feature takes into account results from the Test Impact Collector as well as the changes to the product source between any known builds associated with that Team Foundation Server project. These configuration options help test teams keep pace with new development by always running the test cases that will put the new changes through their paces.
Gone then are the days, when the test teams mindlessly run an entire battery of tests for a new build, irrespective of the new changes that showed up there. Now testing will be much more efficient and leveraged. As each new build comes out, the system will know the exact changes that have occurred, and the Test Impact collector will flag for the testers which subset of their test cases they need to run to validate the new functionality.
That’s pretty neat!
(Staying on the topic of Diagnostic Data Adaptors, and how Developers and Testers work together, I will talk about the Diagnostic Trace Collector in the next post).
As promised, I will spend a few posts talking to you about remote execution and data collection. So far I have talked about specific features related to testing in the upcoming release of the VS 2010 product. While running tests on one’s own desktop is great for development of your product or your tests, but when it comes to testing your product we really need to be thinking about execution across multiple machines and leverage diagnostics information. This really enables testing at scale, as well as testing against real deployment configurations.
The first part of this story is remote execution. One of the great features of our 2010 offering is the ability to provision environments to satisfy the needs of the testing effort. In order to fully exploit that functionality we need to be able to schedule, deploy, and execute tests against an ever changing array of test systems. To pull this off we rely on a Test Agent Controller to schedule and coordinate testing, and Test Agents on each machine to handle the test execution and data collection. The simplest remote execution scenario requires an installation of Visual Studio, the Test Agent Controller and a Test Agent. While this can all be installed on the same machine, the differences between local and remote execution will really be highlighted when you have access to more than one agent across which to execute tests.
Enabling remote execution can be accomplished in Visual Studio by updating a test project’s testsettings file (Local.testsettings in a default test project) or by customizing a Test Settings associated with a Test Plan in Camano. In Visual Studio the test settings options are available from the Test Menu and the Edit Test Settings option, and in Camano there is a Test Settings wizard to streamline configuration. Since we’re working through a simplistic case we will edit the test setting in Visual Studio, select the Execution Criteria option in the dialog, check the radio button to Execute tests remotely, and provide the name of the Test Agent Controller we wish to schedule our test run against.
Prior to modifying the test settings, test execution would occur on the local machine but since we’ve enabled remote execution, when we start our test now the execution will occur on an agent selected by the Test Agent Controller. Initiating a test run will cause Visual Studio to contact the test agent controller, schedule the test against appropriate test agents, deploy our test code, and execute the test on the agent machine. After the test completes, the agent gathers the results and data to return to the test agent controller, which in turn passes that information back to the client that initiated the test run.
As we would expect from local execution, remote execution also populates the Test Results window with test run progress and test results. To see that we’re actually executing our tests remotely we can create a simple test method to print out the computer name and run the test both locally and remotely like so:
[TestMethod]
public void TestMethod1()
{
Console.WriteLine(SystemInformation.ComputerName);
}
And see that in the local case the test prints out the desktop client name (in this case “gstaneff2”) but in the remote case we get the name of one of the agents attached to the controller – for example, “MSL-2108091.”
As we scale out our test lab further we can queue several tests at once against a Test Agent Controller with several attached agents. It then is the responsibility of the controller to parcel out these pending tests based on a variety of criteria such as the properties of the agent machines, which roles are enabled on each agent (roles specify the diagnostic data collectors installed on a group of agents), the load already being carried by each agent and the agent status.
Agent properties are simply Name-Value pairs assigned to each agent in your test lab. If, for instance, some of your agents were running on an English OS and other agents were running on German OS a Language property would be a sensible way to filter available agents for language specific test executions. Agent properties can be added to agents via the Administer Test Agent Controllers dialog in the test menu. The following dialog shows setting of the Language property to the value ‘deu’ for the agent MSL-2108090.
Once our agents have properties we can then instruct the test agent controller to schedule our test run against only those agents that satisfy our requirements:
You’ll note the text under the agent properties list provides the number of agents that meet the specified criteria. The default filter in the testsettings will select agents that include the specified properties. Since by default there are no properties specified this filter will return all agents associated with the specified Test Agent Controller. Since we’ve added a Language property above, we have filtered the total number of agents associated with this controller down to the two agents that happen to have this property set to this value. Notably absent from this list, is the Agent MSL-2108090 for which we just set the value of ‘deu’ for the Language property.
The preview button shows all the agents that satisfy the specified agent properties, but the Agent Preview dialog shows additional information associated with each Agent – a list of Diagnostic Data Adapters that are currently installed on each agent. Diagnostic Data Adapters are like the sidebar gadgets of test execution. They provide functionality to collect information or impact the computer while a test is running that can be enabled or required as part of the test planning process. This frees each test case up from having to collect these information on their own, so they can concentrate on whatever it was they were actually testing. These adapter lists serve as an additional level of filtering, as the Test Agent Controller will not schedule a test run against an agent that does not belong to a role with the required diagnostic data adapter installed. At present the System Information data collector is enabled by default and that means these tests we’ve been running have been passing back a lot of free data about the environment during the test execution:
This, then, illustrates one of the key strengths of the Data Collector concept. It is not a stretch to think of a test case that depends on the Platform/OS/numProcs/Lang/etc. of the system running the test – that a defect may depend on the particular details of the environment. A Diagnostic Data Adapter eliminates the need for each test to implement or call routines required to collect this data (and to know a priori which data will be valuable). During creation of each test plan these adapters can be enabled or disabled as required by the testing effort with no change to the test code itself. By enabling data collectors on the project level in Visual Studio or the plan level in Camano it is possible to establish an organizational policy on what kinds of data to collect for any test case – and this helps us ensure that bug reports generates as a result of our tests contain high quality actionable data.
As the above image of the Agent Preview dialog shows, we have several data collectors planned for this release. We don’t, however, assume to know the particular requirements or needs of every test organization out there and understand our standard data collectors won’t fit all needs equally well. The adapter interface is public and we expect organizations to be able to easily customize their own adapters tailored specifically to their own needs. Once you’ve created and installed your adapter it will show up in the testsettings Execution Criteria options with the others:
While in this post I talked about the System Info data collector, in the next couple of posts I will share with you the details and power of couple of other collectors that we will ship in the box – the Test Impact data collector, and the Diagnostic Trace Data collector. I will also show you how to can add your own data collector and test type, leveraging the extensibility architecture in our product.
Since I’ll be on this topic for a while, please comment on what we’ve shown thus far so I can address it in subsequent posts.
I started the series on “Rethinking Software Testing” in December 2008, and in the past few months have covered several topics – Unit Testing, Load Testing, Test Planning, Manual Testing, and Coded UI. I hope you are finding this series useful and informative – it certainly has been an great experience for me and my team to talk about the product and features we are so passionate about, and in engaging with you around your comments.
I thought I’d take a moment to review how far we have come and what’s coming up next. I am including here the architecture block diagram of our products that I had shared with you earlier. So far, I have covered all of the areas marked in green. What’s remaining to cover are the blocks in blue. The In the next few posts,I will talk about remote execution and data collection (noted as the “Data Collectors” block in the diagram). Then will come reporting, and after that I will devote a few posts to the “Lab Management” product.
Each of these blocks of course are broad feature areas (or products in themselves) and there is a lot more to talk about their capabilities. Once I complete the overview posts on all of the blocks, I will go into more specific details. We should also be ready with the Beta 1 release of VS 2010 around then, and you will have an opportunity to play with the latest bits – I can’t wait for that to happen, and me and my team are hard at work to get ready for this important milestone :-)
Do please let me know if there are specific details about these feature areas that you’d like to hear about in more details, and I’d be happy happy to incorporate those.
Up next then - “Remote Execution and Data Collection” – stay tuned!
In the previous post, you saw how the Manual Testing has been made really easy with Camano. As I had mentioned, when you run through a manual test case for the first time, the Microsoft Test Runner (MTR) captures your UI actions as an “action log”. You have seen how this allows the tester to “fast forward” through her testing in subsequent runs, where MTR simply plays back the “action log”. This “action log” is also useful when it is time to turn a manual test into an automated test, which, for example, can be used as part of a build verification suite.
Let’s consider the case where you are testing the Windows Calculator and have a test case that checks some basic functionality of the application. You now want to automate the test, add validations, and run this regularly as part of a build cycle. Your goal is to ensure that there are no regressions in this part of the application. This is where the “Coded UI Test” feature of the VSTS 2010’s Test Edition comes into play.
Coded UI Test allows you to generate an automated test from a published test case in Team Foundation Server. You just point to the test case on a Team Foundation Server and code will be generated for the test. Magic!!! Not really. Visual Studio simply takes the action log stored for the manual test and converts it into code. (Coded UI Test is a new test type in Visual Studio). I really like this simplicity in converting any manual test into an automated test!
As an example, the “action log” for the manual test case noted above may look like:
![clip_image002[4]](http://blogs.msdn.com/blogfiles/amit_chatterjee/WindowsLiveWriter/CodedUITest_10081/clip_image002%5B4%5D_thumb.jpg "clip_image002[4]")
When you convert this into a Coded UI Test, the actions noted above are translated into the following snippet of code.
![clip_image004[4]](http://blogs.msdn.com/blogfiles/amit_chatterjee/WindowsLiveWriter/CodedUITest_10081/clip_image004%5B4%5D_thumb.jpg "clip_image004[4]")
Now, in the manual testing case, the tester of course was able to verify in person that the above test case generates the result 9. Coded UI Tests allow to simply add such validations directly into the code. This is as simple as right-clicking the position in the code where you want to add the validation, bringing up the “UI Control Locator”, selecting the control and it’s property in the application that you want to add validation for, and letting the tool add the assertion. Here are some screen shots for the “UI Control Locator”, and the screen to select the control property to validate.
![clip_image006[4]](http://blogs.msdn.com/blogfiles/amit_chatterjee/WindowsLiveWriter/CodedUITest_10081/clip_image006%5B4%5D_thumb.jpg "clip_image006[4]")
![clip_image008[4]](http://blogs.msdn.com/blogfiles/amit_chatterjee/WindowsLiveWriter/CodedUITest_10081/clip_image008%5B4%5D_thumb.jpg "clip_image008[4]")
You drag the crosshair of the control locator onto the result box of the application. Click on Add Control and Show Properties. Select the “DisplayText” property and hit OK. The following code gets generated for you.
![clip_image010[4]](http://blogs.msdn.com/blogfiles/amit_chatterjee/WindowsLiveWriter/CodedUITest_10081/clip_image010%5B4%5D_thumb.jpg "clip_image010[4]")
Presto! You have successfully created a Coded UI Test and you can now run it.
By the way, while in this case we have generated the automation code from an existing manual test, you can also start from scratch and use a action recorder to record the actions you want translate into code. Here’s a screen shot of the action recorder
![clip_image012[4]](http://blogs.msdn.com/blogfiles/amit_chatterjee/WindowsLiveWriter/CodedUITest_10081/clip_image012%5B4%5D_thumb.jpg "clip_image012[4]")
Of course, you may not want to use any of the tools, but want to create Coded UI Test right at the API level. For this, you can use the new Microsoft.VisualStudio.TestTools.UITesting API set to directly reference UI controls, perform actions, and validate properties.
It is also important to point out that code can be generated into any of the .Net languages. In the first version we will support code generation into C# and VB.Net. We hope this further helps break down the walls between developers and testers, where the automation engineers or developers can use the standard .Net languages for test automation code generation and there is no new scripting language to learn.
You will also be happy to know that, continuing with our theme of full integration of features across the life cycle, Coded UI Tests are fully integrated with Team Foundation Server (if you are using one).
- You can publish the results of a Coded UI test run to Team Foundation Server. A bug may be created in Team Foundation Server from the results of a Coded UI Test run.
- You can attach the Coded UI Test to a Test Case (see screenshot below). You can then run this automated test from Camano.
- You can create a data driven test case sourcing its data from a test case on Team Foundation Server.
![clip_image014[4]](http://blogs.msdn.com/blogfiles/amit_chatterjee/WindowsLiveWriter/CodedUITest_10081/clip_image014%5B4%5D_thumb.jpg "clip_image014[4]")
This then completes a quick overview of the “Coded UI Test” feature. To reiterate again, you will use this feature when:
- You want to add validations to your test
- You want to create an automated test and run it as part of the build cycle
- You want to customize your automated tests to improve its resilience
I am also including a link here for a step-by-step tutorial that walks you through the creation of a Coded UI Test - Tutorial on creating a Coded UI Test
Most of you are familiar with test automation and use it in some form or other in your projects. What you think of this new feature? Are there specific details you’d like me to elaborate in future posts? I am very interested in hearing from you – do leave a comment.
It’s a very busy phase for us at work. The entire team is busy stabilizing our products for the upcoming VS 2010 Beta release. This is exciting – the team is really proud of the new Test and Lab Management products that we have been designing and building for months, and can’t wait to get the Beta bits into your hands. I spent a significant amount of my time last week taking our bits for a test drive, and have been waiting to find the time to write to you more about them. It’s Saturday and I am just back from a round of golf. The weather here in Hyderabad, India is extremely hot already (95 degrees – and we are still in February) I am ensconced in my study now, and have the next post ready in the series of “re-thinking software testing”. I hope you enjoy learning about our new tools, and look forward to test driving them soon yourself!
In the last post, I talked about Camano, our new test case management solution. Once the test plans are in place, test execution is the next focus area for testers. In today’s post I will introduce MTR – the Microsoft Test Runner.
In an earlier post I had mentioned that our research shows that 70% of the testing done in the industry is manual testing done generalist testers. As we talk to the generalist testers though, we find that they today are frustrated by two aspects of their job:
1. They spend a lot of time in non-core tasks e.g. filing a detailed bug, reproducing the bug, reporting test status etc.
2. There is a lot of repetitiveness in manual testing e.g. they waste a lot of time navigating to a particular point in the application they are testing, and repeating same test with different test data and configurations.
MTR (Microsoft Test Runner) was designed to solve these types of problems. Below is a screen shot of MTR, and I then talk to about its key features.
1. Help manual testers by showing detailed steps & logging results: When one or more tests are launched from Camano’s testing activity center, they are loaded in the Test runner. You can look at the test case with its steps. The detailed steps provide information about what you are supposed to do next, any expected results and attachments etc. You can mark the results of the steps as passed/failed. You can also add any comments. When you are done with the testing, you can publish the results to the Test Case Management (TCM) server. Reports can be generated from the TCM server for any level of information about the tests. You don’t need to separately generate reports of what you have done.
2. Create actionable bugs: The problem of no-repro is a big issue between developers & testers. Testers spend enormous amount of time finding and reporting a bug, but many of the bugs are rejected by developers because the information in the bug report is not comprehensive enough, or they can’t reproduce the bugs on their systems. MTR goes a long way in solving this problem. While using Test runner, if you see an issue, you can, with one click of a button, get the screenshot of the area on the application and attach it to the step. To file a bug, simply click on the create-a-bug icon it populates the bug form. The bug form automatically includes all the information needed to reproduce the bug including what steps have been tested; what exact UI actions were performed on the application (yes each and every UI action), the video of the testing being done; detailed system information about the machine used to test; the comments & screenshots added during test etc. (Of course you control what to record in test settings). You may just type the title of the bug and fire away! The screen below shows how a part of the bug form looks like.
3. Fast forward for manual testing: When the developer fixes the bug , you will need to verify the fix. You may need to go through a lot of steps to reach the point where you want to pay more attention. With the fast forward feature, you can ask Test Runner to navigate the application to the right spot quickly. How does it do it? While testing the first time, the action recording engine in Test Runner captures your UI actions and can use these actions to fast forward quickly to the right spot. You can skip from any point in your test and then test manually and then skip again & so on. The two screen shots below show the tester being prompted to select the steps to fast forward, and the system playing back those actions to get the tester to specific test she wants to focus on.
4. Data driving tests: As mentioned earlier, it is a laborious process to test the same test case with different data sets. But it needs to be done to look for bugs that may come with different data (for example, try wrong data, no data …). Fast forward functionality of Test Runner not only works with the actions you performed, but is also intelligent enough to differentiate between normal UI actions and data entry. When you test the same test case with different data set, fast forward functionality will pick up the right data sets to be entered in the app while navigating. This helps you focus on the app and find bugs rather than typing all that data.
5. Shared steps: Many test cases share same steps to test various functionalities. Test Runner allows you to share these steps in sharable units. That is, you can create shared steps and then use then in other test cases. Writing manual tests just become easier. If application changes and you need to update the steps, just update it in the shared steps and all test cases benefit from the change. Similarly, if you had recorded the steps (for fast forward functionality), the recording is also shared among test cases. So if the application changes (which does so often) its impact can be contained to just one shared component.
Can you believe we are still talking about manual testing?
I hope you like what you have seen so far – I’d love to hear from you, so please do leave a comment with your opinions, comments, and questions.
In a subsequent post I look under the covers of the MTR with you and talk about the innovative way it works. I will also discuss about its extensible architecture. But before I do that, I will introduce our support for automation in the next post. Talk to you soon!
The concept of a test plan
Just like architects plan and document the design and requirements of software products, and just like developers plan and document how they will be implementing it, testers also plan and document their strategy to verify that such products meet those specifications and requirements. This documented strategy created by testers is called a test plan. Great test plans are those that are able to effectively verify the quality of a product (given the requirements and specifications) through the use of effective manual and/or automated test cases.
Test plans in Camano
Camano is the internal codename for the test case management and test runner solution that we have been working on as part of our next major release (Visual Studio Team System 2010 Test Edition).
Test plans in Camano fit the definition that we already talked about, by providing testers the tools they need to document their strategy. But it also go beyond that - letting them organize, execute and track the progress of the different artifacts on their plans (test cases, configurations, runs, etc). This is how a test plan looks like: 
I will throw in some questions for you along the way. Are these test plan properties (name, description, iteration, etc) shown here enough? Leave a comment if you think there are key attributes missing?
When and why does it make sense to use a test plan?
A test plan makes sense whenever there is an application, specific feature, or specific goal (such as testing localization) within a context of an application. Test plans in Camano will help test managers track testing from its initial design to its completion, and will help testers track their test cases, manual and/or automated runs, analysis, and defects.
Test design: adding/organizing test cases to/on a test plan
The testing design starts whenever testers start thinking and documenting test cases that will help them uncover defects throughout the whole lifecycle of a product. Camano enables this design process in a very simple way.
A tester can add existing test cases (maybe there are test cases that can be reused from other projects or features) or just create new ones on any test plan. These test cases can also be grouped in what we call static or query-based suites. This is what an organized test plan with test cases could look like:
As you can see in the left pane, suites are used to organize tests (which appear in the right pane), and progress on the test plan can be seen right there as well. Does this seem easy and straightforward to use?
Test cases are ready to be run either manually or in an automated fashion right away, and the progress of plans can be tracked in a straightforward way as well. A usually complex task turned into a simple and fun effort!
Test organization
Test organizations can be as small as one tester doing the whole job, and up to tens or hundreds of people working on a project. Roles go from test managers, to test leads, and individual testers. Camano makes it easy for the whole team to work together, and at the same time enables each person on the team to focus on stuff that he/she needs to get done.
Test managers or leads will be interested in knowing how the testing effort is going. They might want to know, for example, how many test cases exist for a given feature, and how many have been completed so far. They might also want to know the owner for each of these test cases. They can easily view all this information from any given test plan:
Testers may want to track planned test cases, and may want to filter based on those that are assigned to them. They may also want to specify different configurations for them. Camano will let them do all these and more…
How to know when testing is done?
Saying that we’re done testing is never an easy task. In order to have the right answer, the correct test design will need to be figured out, and the actual implementation of that test design will need to be completed. Camano will not test for you, but it will give you the right and simple-to-use tools to accomplish this goal: Easy test design, easy tracking of the effort, and easy reporting on the status of test cases; all these things right out of the test plan. If the right design is in place, the answer will be a straightforward one: “We’re done testing”.
What else is there in Camano?
This post has focused on test planning; but there is much more in Camano. As soon as test planning is done, execution is what makes a test plan look good. Camano makes running tests a simple and efficient task. Afterwards, there is a whole integrated story that includes analyzing runs, linking and tracking bugs, viewing reports, looking at recommended tests to run on new builds, running tests remotely and collecting data such as code coverage, and even creating test environments that testers can share with their developers. Are we missing any features that you as a tester think we have to ship with this version?
When we go out to the field and demo the product to customers, we’ve heard that they love the whole integrated story that lets testers easily test a product and create bugs against it; also the fact that developers get all the information they need in order to be able to repro the bugs quickly and fix them accordingly.
Stay tuned for further blog posts describing all these features in Camano that integrate the tester and developer stories together!
Can’t wait for future blog posts?
Then go ahead and try the latest bits! You can get them here:
https://connect.microsoft.com/VisualStudio/content/content.aspx?ContentID=9790
Please let us know if you love our product, and if you think we’re missing anything.
