Amit Chatterjee's Blog

Have you tried out the Lab Management Product in the VS 2010 Beta 2 release?

2009-11-29T01:12:00Z

Have you tried out the Lab Management product that is part of the Visual Studio 21010 Beta 2 release? It is a really cool product! I have blogged about its capabilities earlier, and would really encourage you to try out the product. The product development team has recently posted a four part series on getting started with Lab Management, and you can find the start of the series here.

This is a great product for:

Test Managers or leads who manage a lab for their development team, and have faced the many challenges is setting up machines, and managing test runs on them
Or, for individual testers who need access to multiple test configurations
Or, for developers who want to test out their code on product like environments before they check their code in, and want to do so just from their decktop!
Or, Managers who want to have a view of how their development labs are running and being utilized

You can get started with the product just with a couple of machines (two Hyper-V enabled 4GB boxes will do, or a single Hyper-V enabled 8GB box will also be fine). The blog post above guides you throug step-by-step instructions about getting a virtual lab started, creating multi-machine environments, bringing up the environments, deploying the application you want to test on to the environment, and running validation tests.

I personally have been playing with the product and it is very cool. Some of my favorites are:

Ability to easily deploy different configurations from my library of stored environment
Checkpointing multi-machine environments to a known clean state
The integration of the Lab Manamgent capability with the rest of the Visual Studio 2010 capabilities

I can run tests on the Lab environment right from my desktop by using the Microsoft Test and Lab Manager tool
I can bring up the "Lab Enviroment Viewer" to access the different machines in the environment
I can integrate with team build definitions to deploy the app I am building automatically onto the the Lab environment
I can run unit tests on my lab environment
I can also run Coded UI of functional automation tests automatically ont he lab environment - I can tell you, it is an absolute joy to visually see the tests run remotely
Restore back to a known checkpoint to do further testing
Appending links to the environments being tested in bug reports, which I can access later
Etc.

My team is also using the product for all of their product testing as well. The tool gives me a great view of seeing the entire lab in "action". I will blog more about this.

You have to try this out to see the benefits. Once again, the team has a detailed post here ( http://blogs.msdn.com/lab_management/archive/2009/11/18/Getting-started-with-Lab-Management-_2800_Part-1_2900_.aspx) to get you started.

You will enjoy the product. Do post back a comment if you have used the product, or if you need further clarifications from us.

Cheers!

In Stockholm the next few days ...

2009-11-28T12:41:00Z

I am travelling to Stockholm and will spend the week there meeting customers and talking about the Visual Studio 2010 product, and especially the Test and Lab Management capabilities. I will also attend the Eurostar conference. Write me a note if you are in the region and would like to discuss the adoption of VS 2010 test products with me.

Cheers!

What’s new in Test and Lab Management in VS 2010 Beta 2

2009-10-23T13:05:00Z

Visual Studio 2010 Beta 2 is now live, and we have updated the Beta2 landing page to point to the publicly available downloads. Early reaction from our MSDN customers and MVPs has been very positive. The Beta 2 release is also a “go-live” release – that it, the product is ready enough that we are licensing the product for developing and deploying production applications. You can read more about what it means to go-live from Jeff Beehler’s post on the topic.

In my product team, we are actively working with a set of customers who have chosen to go-live with our Test and Lab Management Products. Please post a comment on this blog if you are interested in going live with the VS 2010 Beta 2 product.

In this post I want to highlight the changes that you will see in Beta 2, focusing on the Test and Lab Management capabilities.

One of the most significant changes is on the packing front. We have finalized the SKU structure for the VS 2010 release, and it is a marked simplification from the many different SKUs we had in VS 2008 which made the buying decision quite complex. With the 2010 release, we are moving to a simple nested-model of capabilities which you can think of as three variants – good, better, and best. The picture below shows the three variants.

At the lowest level you have the good – the VS 2010 Professional SKU with or without MSDN. These are the tools for basic development tasks. The “better” is the VS 2010 Premium SKU which offers a complete toolset that allows developers to deliver scalable, high quality applications. Finally, the “best” builds upon this theme and provides a comprehensive suite of application lifecycle management tools for teams. The next diagram below gives you a sense of how the capabilities build up from the good to the best. I have marked off those feature sets that are targeted for testers (or valuable for testers) with red boxes.

There are some additional products in the VS 2010 family that are shown below – these are particularly important for testers. The Visual Studio Test Elements SKU is essential for anyone engaging in test case management or manual test execution in the development cycle. All of the functionality of this SKU, by the way, is also included in the VS 2010 Ultimate SKU as well. The Lab Management product enables teams to configure and manage virtual lab environments. All of these capabilities are of course built on top of the Team Foundation Server product.

A high level diagram which shows were the features land in the different client side SKUs is shown below

In the backdrop of the above changes in the SKU and packaging, let me now tell you about the key changes you will see in the Test and Lab Management products in Beta 2 that are new or improved over the functionality we had in the Beta 1 release.

The table below shows all of the features and capabilities for our Test product that is new or changed in Beta 2

The next table shows the improvements in the Lab Management product for Beta 2

You will see a lot of blog activity from team members soon, as they talk about their favorite features and scenarios. I will complete this post with a set of screen shots from the Beta 2 builds.

A view of the plan activity center showing the hierarchy of suites and test cases in a suite

Test Plan properties showing various settings and results overview

Test Results from a Data Driven Test

The Lab Activity center showing Lab Environments in various states

Releasing Beta 2 is a very significant milestone for us. We are now in the home stretch for the product, and are ready for you to go live with this release. I look forward to you deploying the products in your teams, and will be happy to answer any questions and clarification you may have.

I will be back soon with future posts on value props and scenarios we are enabling with this release.

Visual Studio 2010 Beta 2 is now available!

2009-10-20T16:49:00Z

I am very excited that Visual Studio 2010 Beta 2 is now available to our customers for testing and deployment. MSDN subscribers have access to download the bit immediately, and general availability is tomorrow (October 21). The Beta can be downloaded from here

Beta 1 was shipped in May 2009, and you will be amazed at the level of improvements we have made in the feature set, performance and stability, fit and finish, and simplification of install and use that we are bringing to you with this Beta 2 release!

At the overall Visual Studio level, there are several key changes. The biggest that you will see is a very significant simplification of the SKU and packaging.

Jason Zander articulates this well in his blog (and here is the new cool Logo as well)

Visual Studio Express: the free Express SKUs for C++, C#, VB, and Web have been updated and released with this version and give you the basics for writing applications

Microsoft Visual Studio 2010 Professional with MSDN: professional development tools as you are used to today with the addition of source control integration, extensibility, etc.

Microsoft Visual Studio 2010 Premium with MSDN: Premium has everything in Professional plus advanced development (including profiling and debugging), advanced database support, coded UI testing, etc. Rather than buying multiple “Team” SKUs, you can now get this combination of features in one box.

Microsoft Visual Studio 2010 Ultimate with MSDN: Ultimate has everything in Premium plus additional advanced features for developers, testers, and architects including features like Intellitrace (formerly Historical Debugging), the new architecture tools (UML, discovery), etc. All of the scenarios we’ve talked about are supported with this version of the product.

In addition to these changes, Team Foundation Server (TFS) is now available with all versions of Visual Studio 2010. You can get started with TFS Basic and work up to full multi-server support, SharePoint integration, etc. as your needs grow.

The new “Test Elements” SKU is designed for testers who don’t need to write code. You can manage your test cases, work items, and do manual testing using the SKU. Because everything is integrated through TFS, you can continue to collaborate with the entire team

You have the “Team Lab Management” SKU for managing your dev and test lab using Virtualization – allowing you to create multi-tier virtual environments, and to run build-deploy-test workflows on those environments.

Beta 2 is also comes with a “go-live” license, which implies that you can deploy the tools for your “live” projects in production (view the license agreements for more details). This is a really a great opportunity for you to get going with this Visual Studio release!

Also check out the Microsoft PressPass news about the announcement, including the promotional “Ultimate Offer”. I will let you soak in the top level messages around the announcement for a couple of days. You can also get a feel of installing the product and the top level features from this video by Brian Keller.

I will be back soon to talk to you about the many improvements and changes that we have made in the Testing related products.

Cheers!

Is this world a giant Load Test in action?

2009-09-20T10:52:00Z

Next week I am on the road to meet a few customers and discuss with them the Test offerings that my team has been working on. In particular, most of the conversations on this trip are going to revolve around our Load Test offering. I have talked about our Load Testing feature earlier in my blog, and you can find a lot more details about the product and it’s capabilities at Ed Glas’ blog. Ed and his team have done a superb job in building this product, and I am really proud of having this product as part of our repertoire for testing organizations.

The diagram below shows the architecture of our Load Test product. The load testing is orchestrated from Visual Studio. You set up the scenarios (groups of tests), their characteristics (mixes) for browser and network connections, the load pattern (constant, step, or goal based), the run settings, the data binding for individual tests (to ensure that the tests drive different behavior on different run), the time of the run, etc. Visual studio then, through a controller and one or more agents, simulates a real-life load situation for the target server (each agent simulates multiple users of the server). The target server itself is invoked through its public “http”, web-services, or other protocol interfaces. Furthermore, on the target server you can have collector agents running which can “observe” the server under load, collecting various profiling, timing, and resource usage metrics which can be sent to visual studio for analysis of how the serve is holding up.

This is a neat product and a must for any team that is shipping a server product and wants to make sure that the server will hold up in real life with millions of users pounding on it.

This post, though, isn’t so much about the product as it is about the parallels of load testing in real life.

A thought occurred to me that this world, and all of its inhabitants, is enacting nothing but a giant load testing run!

Let start with us humans. We are the individual “servers” that are being load tested. I like this analogy with a server or service – a higher goal in life is really to be of service to this web of life that we call the world. This server, at its core is made up of basic ingredients – earth, water, fire, space, and air – and has its own interfaces – the five organs of perception (sight, hearing, smell, touch, and taste), five organs of action (hands, feet etc.), and the four inner instruments of mind, intellect, memory, and ego. The server has a core set of functions (breathing, eating, waking and sleeping, etc.), and then throughout its life, being driven by the inner instruments and the organs of action and perception, indulgences in a bevy of activities that we are familiar with.

The ego perceives all of these activities as its own and thinks it is the real doer. But perhaps that isn’t the real truth and there is a “Controller” somewhere that is orchestrating all of these behind the scenes? Perhaps this giant run has been set in motion, with “data binding and data access” leading to the seemingly random and unique events of life? The server doesn’t know how long it has been scheduled to run for, and even as individual servers fall over and leave the run, newer ones are added. Moreover, are individual servers, which are being load-tested, in turn “generating load” for other servers? How complicated does this get if you throw in the other 8.4 million species in this planet into this test – are these “smaller servers” in the mix?

Is it possible that a part of the “Orchestrator” resides in each individual server as a “silent witness” simply observing what’s going on and perhaps collecting “data” that will be useful in the “final analysis”?

The customer visit next week is a good opportunity, I will be thinking deeply about our load testing product :-)

Lab Management Tutorials and Manual Testing Video

2009-06-13T07:38:00Z

Greetings! It has been a while since the last post – I and my team have been preoccupied with work of course, wrapping up the Beta 1 release of VSTS 2010 and finishing up the remaining features for the Team Test and Lab Management products. The team also took a well deserved long weekend off to celebrate the accomplishments, and to gear up for the drive for the next Beta scheduled for later in the year. I am really looking forward to this phase – shipping world class products is always an exhilarating experience, and the end-game always pumps me up even more!

The weather too is shaping up nicely and that’s going to add to the zest as well. The two months of brutal summer is over in Hyderabad, and the monsoon is setting in. It is right now a bit humid, but the temperature on this Saturday morning is a nice 80 degrees, down from the 100+ even a few days back. Team members in the USA are enjoying the start of summer there, and the temperature in Redmond, WA is 73 degrees, and in Raleigh, NC is 76 degrees. So, that’s great weather all around!

The Beta 1 feedback is starting to come in. The feedback is positive. The value props, scenarios, and the feature set are resonating well. The team members are monitoring the feedback forums looking for cases where people might be having problems – those are areas we want to address and fix before we release.

I want to take this opportunity to point you to two specific blog posts from the product team.

The Lab Management team is running a sequence of tutorial like posts here. I talked to you about the high level concepts of the product in my last post. Since then, the team has posted two articles – one on creating Lab Environments, and another on automatically deploying bits on to the environments. Give those a read. The product is also getting good coverage at the www.virtualization.info site – one of most popular sites for the virtualization industry. Check out this post.

On the Team Test front, Brian Keller has just posted a video (about 25 minutes) where he talks about and demonstrates our Manual Testing solution. You can find the video here. You have read about the product offerings for the Generalist or Manual tester, and this video will give you a good view of those capabilities.

If you haven’t tried out the Beta 1 bits yet, I strongly recommend you do so and let us know your own opinion. You can find the information about the Beta here.

Do write back about your experience with the Beta 1 bits, and let me know if there are specific areas or capabilities that you would like to learn more about. I will be happy to talk about them in the future posts.

Enjoy the weather, where ever you are.

Cheers!

The Lab Management Product – An Overview …

2009-05-24T14:43:00Z

A lot is going on. The VSTS 2010 Beta 1 bits are out (as I had noted in my last post) and with that the interest in the product, both internally and externally, is peaking. The product team is busy interacting with customers and getting their feedback. There are also few feature modifications that are being worked on. So, while the Beta 1 is released, there is still a lot to be done. I will continue to walk you through the feature and functionality that’s in the product. As you install and play with the bits, do post your comments on areas of the product that you’d want additional information on.

The weather too is improving – the hot summer of Hyderabad is almost over and the monsoon is right around the corner. The clouds are building up and the oppressive heat is less now. The cooler climate will provide that extra burst of energy that we are all eager to put in on the product front and customer-connection front ;-)

One of the coolest products we are introducing with the VSTS 2010 release is the Lab Management product. This is an integrated solution that brings virtualization to the heart of application lifecycle management space. Here’s a glimpse of the capabilities that you get with the product:

Create libraries of virtualized multi-tier test configurations really quickly
Automatically deploy new builds of your application to these environments
Seamless integration of our dev and test capabilities with the virtualized environments
Take the “no-more no-repro” theme to the next level by leveraging snapshots

In this post, I will cover the high level concepts of the Lab Management product.

Lab Management High Level Architecture

The diagram below shows a high level architecture diagram for Lab Management.

On the server side, Lab Management service is one of the many services running inside Team Foundation Server (TFS). This is what makes the Lab Management solution unique for software testers and developers. Now you can map your lab resources, such as, hosts, virtual machines and storage to Team Project Collections and Team Projects; thus aligning lab hardware needs with the business needs for the projects you are working on.

The lab management service in TFS uses System Center Virtual Machine Manager (SCVMM) for management of lab infrastructure and provisioning of virtual machines across multiple virtualization platforms. You get a copy of SCVMM with Lab Management.

On the client side, the “Microsoft Test and Lab Manager” tool (earlier known as “Camano”) is still the tool to manage your virtualized assets. This is a Windows Presentation Foundation (WPF) based rich client that allows you to define test plans, test suites, test cases and run them in physical or virtual environments.

Basic Concepts

Similar to Internet, hardware virtualization is a disruptive technology that is changing the face of computing. Therefore, it is important to understand some of the basic concepts around virtualization and how these are used in Lab Management to understand this paradigm shift.

Virtual Machine

A virtual machine (VM) is a computer within a computer, implemented in software. A VM emulates a complete hardware system, from processor to network card, in a self-contained, isolated software environment, enabling the simultaneous operation of otherwise incompatible operating systems on a single physical computer. Each operating system runs in its own isolated software partition.

Virtual Machine Snapshot

A virtual machine snapshot is a file-based snapshot of the state, disk data, and configuration of a VM at a specific point in time. A VM snapshot is similar in functionality to laptop hibernation state with the additional flexibility that a VM supports multiple snapshots. You can roll back the VM to any of the previously taken snapshots and continue operating from there. The picture below shows a snapshot tree for a Hyper-V VM.

Host

A host is a physical computer that hosts one or more virtual machines.

Host Group

A host group is a custom group of virtual machine hosts, which an administrator can create in SCVMM for ease of monitoring and management. Host groups can be used to allocate and determine the resources reserved for various team projects. For example, an administrator could create a host group named “Global Bank Hosts” for a team that works on “Global Bank” project and bind it to the corresponding team project in Team Foundation Admin Console.

Library Share

One of the beauties of virtual machines is that you don’t need to tie up a host if you are not actively using a VM. You can store it on a disk and bring it back to life on a host in a few minutes. SCVMM supports the concept of a library share where you can store virtual machines and other resources, such as, ISO images. The library share is nothing but a file share that is accessible to all the hosts. Similar to host groups, you can create multiple library shares for ease of management. For example, you could have a library share for storing pristine or golden OS images. Another library share could be used for storing VMs that have various application software components installed.

Environment

A typical multi-tier application consists of multiple roles, such as, Database Server, Web Server, Client, etc. Each role could be running on one or more computer. You could also have multiple roles running on a single computer. An environment is a set of roles that are required to run a specific application and the lab machines to be used for each role.

Managing environments for multi-tier applications is an error prone task today. Replicating the same environment at same or another site is even a bigger problem.

Lab Management surfaces environments as a first class entity. See the picture below for an example. It shows a list of environments running in the lab. The “DinnerNow Integration Testing 2” environment consists for two virtual machines, named, DinnerNow-Web and Dinner-SQL.

Environment brings with it ‘strong’ group notion. That is when you do an operation on an environment, such as, start, stop, take snapshot, etc., that operation is applied on all the virtual machines that are part of the environment. In the next blog, we will show you how to create, start, stop, delete, save, take snapshot and interact with rich self-documenting virtual environments using Lab Management.

Online Documentation

As you install the products, please refer to our online documentation which will give you lot more detailed information on the concepts and the capabilities. Peter Bateman and our documentation team would love to hear your feedback on the docs too!

Next

The above covers the basic concepts that you need to understand in order to leverage the Lab Management product. I will take you through some of the high level features in my next few posts…

VSTS 2010 Beta 1 download now available for everyone …

2009-05-21T12:27:00Z

The VS 2010 Beta 1, the VSTS 2010 Beta1, and the .NET Framework Beta 1 is now available for all to download. Here are a few links that you will find useful:

The Beta 1 download site (the homage page this beta program)

The VSTS Beta 1 Suite Installer (includes the Team Test client components)

Team Foundation Server download

Lab Management Beta 1 download

Lab Agent download

Test Agent download

Note that the VSTS 2010 Suite Beta 1 bits includes all of the client products for Visual Studio Team Systems – that is, you will get the product for Developers, Architects, and Testers. The Team Foundation Server (TFS) bits provide the server side for the above client components, and the Lab Management product is for the physical machines which you want to devote to setting up a virtual lab. So, the above downloads will enable you to install and play with the Test Product and the Lab Management products that I have been talking to you about.

My colleague Brian Keller has created this nice video that will help you with the installation and configuration of the VSTS 2010 Suite Beta 1 bits.

I really encourage you to download and try out the product, and would love to hear your feedback. I am also including the official channels for getting your feedback and issues to the product development teams.

Beta 1 Forums

Beta 1 Connect Site (for bug reporting)

Finally, I will point to a couple of other important links that you will find useful as you try out the products:

Online documentation for Team Test (docs for the Team Test features)

Lab Management Setup Guide download (setup/config of Lab Management)

There – I think you now have all the information to get started with the bits. I will now return back to my series on testing, and in the new few posts focus on the Lab Management product. I hope you are enjoying this series, and stay tuned …

Announcement MSDN subscribers: VSTS 2010 Beta 1 now available!

2009-05-19T10:28:00Z

The Beta 1 download of VSTS 2010, along with Beta 1 bits for Visual Studio 2010 and .NET 4.0, are now available here.

MSDN subscribers can download the beta today, while the rest will have public access starting this Wednesday. I’ll post the public links for non-MSDN subscribers on Wednesday.

This is a very exciting milestone for us. A lot of innovative thinking and hard work has gone into these products. While we have released CTP bits earlier, nothing like getting a fully installable beta into your hands!

I am also very excited that now you will have an opportunity to try out the features as you follow along the topics I have been blogging about in the Testing and Lab Management areas. I’d love to hear about your own hands on experience with these features. I am happy to take feedback on my blog, but also do use http://connect.microsoft.com for bug reports, and visit the forums for detailed discussions.

Do also take a look at Jason Zander’s blog on this announcement – he gives you a highly level view of all of the features areas in the VS 2010 Beta!

Try out the bits and have fun!

Reporting on Test Management Data

2009-05-18T20:42:00Z

You have been following along the innovative set of features and capabilities that we are providing with the Test product in the VSTS 2010 release. In this post, I want to explore with you the rich and flexible options you have for reporting on various aspects of test data and test results, and its management.

The richness and flexibility really comes from the fact that our test offerings are architected on top of the Team Foundation Server (TFS). One of the major value propositions of TFS has always been rich reporting across a broad set of data. In the 2005 and 2008 editions of VSTS we only included a minor set of data around your test artifacts, which primarily enabled you to report on the test results that you had run as part of your build. However, in the 2010 release, all of the great features you have read about in Microsoft Test and Lab Manager (earlier referred to with the code name “Camano”) are backed by the new Test Management services on Team Foundation Server 2010 and we have enabled you to report on the very rich set of data that you both capture and create during your testing.

One of the best ways to harness that data is to use the powerful features in Microsoft Excel in combination with SQL Server Analysis services to mine it. In this post I will help you get started with a couple off basic reports that you can view on a daily basis to track your testing effort.

Test Plan Progress

I will call the first report we will build, the Test Plan Progress. It will enable you to track the progress of one or more test plans across a period of time.

To get started you will need to insert a pivot table into a worksheet and connect it to your data warehouse. By default the warehouse will be installed on our TFS data tier.

Start by clicking on the Insert tab and selecting the PivotTable button:

In the Create PivotTable dialog you will select “Use an external data source” radio button and click on “Choose Connection”

From the “Existing Connections” dialog you will click the “Browse for more” button and from the “Select Data Source” browser you will select “New Source” at the bottom and will be presented with the “Data Connection Wizard”

Select “Microsoft SQL Server Analysis Services” as the type of data source you want and follow the wizard.

After you have entered the name of your analysis services server (which is a typical installation will be the same as your data tier), you will have the option to select the OLAP data base and cube that you want to connect to:

If you are using SQL Enterprise you will have the option to pick which perspective you want to use, however, if you are using SQL Standard edition you will only have the option to connect to the Team System cube. If you have the option to use a perspective you should select the Test Result perspective.

Once you have established the connection to the server you will need to add the following fields from the Pivot Table Field List to the Report Filter section of the Pivot Table editor.

Team Project.Team Project Hierarchy
Test Result.Iteration Hierarchy
Test Result.Area Hierarchy
Test Plan.Test Plan Name

(Also take the time to browse and view the rich set of fields that are exposed in pivot table field list!)

After you have added those fields to the Report Filter section, you will want to add the Test Result.Outcome field to the Column Labels section, the Cumulative Point Count measure to the Values section and the Date.Date field to the Row Labels section. After you initial setup your sheet will look something like this:

Once you have the basic data setup you can filter it down so it makes sense for you. In this example we are going to start by filtering out some of the Outcome’s that we don’t need to report on. In addition you might want to filter to only show results for a single iteration.

Select the dropdown next to the Column Labels field in the PivotTable and un-check the “None” and “Unknown” values from the dropdown.

Next right-click on the PivotTable and select “PivotTable Options” from the menu. In the “PivotTable Options” dialog select the “Totals & Filters” tab and un-check the “Grand Totals” options and press “OK”.

After you have setup your PivotTable with the data you want and the filters you want you can easily create a chart for our data.

Select your PivotTable and select the Area chart from the “Insert” tab

Excel will automatically generate a chart based on the style you have selected and after adding a title you have something that can be easily read by a variety of people

In the chart above we can see the progress of all of the test plans in our project collection. In an ideal trend we would see a steady increase in the number of Passed tests and a steady decrease in the number of Failed, Never Run or Blocked tests.

The data is organized in such a way that you are always looking at the most recent outcome for a given test case on a given day.

For example: If I run Test Case 1 on 4/30 and it passes and I don’t run it again, when I look at the data on 5/13 it will still show Test Case 1 as passing.

Test Results by Build

Another graph that you may be interested in seeing is your test results on a build by build basis.

To start out you will create a new worksheet in your workbook and insert a PivotTable with a connection to your warehouse in the same way you did at the beginning of this post.

For the Test Results by Build report you will select the following fields:

For the Report Filter, choose the following:

Team Project.Team Project Hierarchy
Test Result.Iteration Hierarchy
Test Result.Area Hierarchy
Build.Build Definition Name

For the Column Labels, select:

Test Result.Outcome

For Row Labels, select:

Build.Build Name

And finally for Values, choose:

Cumulative Build Result Count

Once again you will want to filter out the “None” and “Unknown” outcomes by click in the “Column Labels” dropdown and un-checking the values.

After you get your PivotTable setup you are ready to generate your chart. For this particular report I think a stacked column chart works best.

In this report we can see the outcomes for the tests that were on a given build. In this case we have the full set of builds for a build definition. If this was our nightly build we could compare the results side by side to make sure we are not seeing any regressions.

Conclusion

Creating reports by simply dragging and dropping values into the PivotTable let you quickly explore your data. You can try different way for grouping and filtering the data as well as applying different type of charts to see what works best for you.

In my next post on reporting, I will discuss the dimensions we provide in greater detail so you can better understand the types of scenarios you can report on.

Official Names for the 2010 Test Products now announced!

2009-05-14T09:18:00Z

This is an exciting day for me and my team. We have just announced the official names for the 2010 Test Products that I have been talking about in this blog. You can see the announcement here by Jason Zander.

Let me elaborate on the announcement in the context of the component block diagram that I have shared with you earlier and am including here again for reference with a color coding to help you understand the product offerings.

All of the functionality above will be released as the following three products, which sum up a great line of offerings:

Visual Studio® Team Test 2010 Essentials

Support for the generalist tester including the ability to manage test cases and manual/automated test execution. Installs as a scaled down product for easy access on test machines.

We expect that there will be plenty of cases where generalist testers will want to manage and execute test cases without installing the entire Visual Studio system on each machine. This then is the product for you.

The components included here are the Microsoft Test Runner, the Microsoft Test and Lab Manager application, Reporting, and Data Collectors (all of the components shown in blue in the above diagram). You also get a TFS Client Access License (CAL)

Visual Studio® Team Test 2010

Support for the specialist tester including Web and load testing capabilities in addition to the ability to create automated test suites. Executes in the Visual Studio environment for test professionals and comes with Microsoft Test and Lab Manager.

In addition to all of the components from the “Test Essentials” product, this product will include Unit Testing, Coded UI Test, Web Test, and Load Test components (all of the components shown in green in the above diagram). This too of course comes with a TFS CAL.

Visual Studio® Lab Management 2010

Support for creating virtualized environments with snapshot capabilities. You can now execute your tests using the lab capabilities and save the state later for both development and test usage. (The component shown in orange color).

Please note that the main client side application for the products – which I have been referring to with the code name of “Camano” in this blog series, is now officially named as Microsoft Test and Lab Manager.

These components will all be part of the Visual Studio 2010 Beta 1 release that is just round the corner.

I am eager to talk to you about the few remaining components and capabilities I haven’t talked to you in great details yet. Next up is a post on our reporting capabilities. I will then follow up with a few posts on the Lab Management Product.

Stay tuned, and your feedback and comments are most welcome!

A Side Show: Back from the Power Lab at Cape Cod, MA

2009-05-09T07:36:00Z

So I have been incommunicado for a while, though for a specific reason. I was out for a week, at Cape Code, Massachusetts attending the Power Lab – an executive development program. The program is organized as a fully immersive societal experience. The society has a class hierarchy – Elites you own almost all of the resources and jobs, Middles who manage the jobs for the Elites, and Immigrants who practically own nothing. I was “born” as an immigrant in the society, and it was an absolutely incredible experience. Mind you, this is not a regular gig where the program runs from 8 to 5 and you have nice cocktail dinners in the evenings. In this program you truly live your role 24 hours a day and for multiple days at a stretch! It was a great learning experience – literally living through the many experiences around issues of power and leadership, and confronting several situations which I’d normally consider off-limits for me. During the week I had no access to my laptop or cell phone, and hence was completely cut-off from work and my real-life society.

You can find out more about the Power Lab at www.powerandsystems.com.

This post was just a side-show. I will get back to the “center ring” and continue with the series on testing soon.

Next week is also TechEd2009 week – the main conference is in Los Angeles from May 11-15. There will be several sessions on Visual Studio Test Systems, Team Test, and the Lab Management products. In India, the main event will be in Hyderabad on those same dates. I will once again present the “Next Generation ALM Tools from Microsoft – A Lap around Visual Studio Team System 2010” session. We will also have a “Technology Tent” to show-off all of the cool stuff that the Test and Lab Management teams have been working on. Hope to see some of you there …

Upcoming VSTS 2010 Presentation at the Great Indian Developer Summit …

2009-04-21T10:52:00Z

Tomorrow I am travelling to Bangalore, India to key note at the “Great Indian Developer Summit 2009”. You can find details about this conference here. This probably is the largest .NET event in this part of the world!

My key note is at 11:40am on April 22. Here’s a synopsis of my talk:

Next Generation ALM Tools from Microsoft – A Lap around Visual Studio Team System 2010

“You want to build great software, and to do so you need more than skills than just coding. A full team comprising of business analysts, architects, developer, testers, project managers, and operations manager needs to come together to create the magic of great software worthy for today’s enterprises. To make this happen you need a development platform and tools that seamlessly, transparently, and effectively bridge the requirements for all the roles and orchestrate a great development experience. Microsoft Visual Studio Team System 2010 introduces a new and improved tooling platform that addresses the needs of the entire application development lifecycle, enabling teams to build great software. In this demo filled session you will get a great overview of this platform and tools.”

I am excited about the opportunity of meeting with the developer community and demoing the VS2010 bits!

Hope to see some of you at this event!

Diagnostic Data Adapters: Changing how Developers and Testers work together (Part 2 of 2 - The Diagnostic Trace Collector)

2009-04-21T10:31:00Z

This post finishes up the two part topic on Diagnostic Data Adapters in VS 2010, and how they help developers and testers to work together. In the last post I talked about the “Test Impact Collector”, and today I will talk about the “Diagnostic Trace Collector”.

In an earlier post, I had talked about the “No Repro” problem that plagues the Industry today. This is that frustrating, almost reflexive, resolution to bug reports by developers – or perhaps the only option left when the test team didn’t provide enough information in the bug report to make heads or tails of what they were trying to do with the product. The VSTS 2010 product provides a variety of mechanisms and dials to combat this problem. The Diagnostic Trace Collector is another one of those, and can turn any test case into a data record complete enough to support time travel debugging! Any tester can run a manual test with the Diagnostic Trace Collector enabled and file a bug. From that bug report the development team will have enough data about the state of the system under test to walk backwards and forwards from the point of test failure in a debugger located on their own machine.

This is a very powerful feature! Visualize this picture. The Developer simply clicks on a log file attached to a bug that’s assigned to him, and this launches Visual Studio into a debug like state that allows him to inspect call stacks, events, function parameters and the like. The magical part of this of course is that you get the experience of Visual Studio debugging an application that is no longer running on the developers work station or the testers machine, yet it literally is the state of the application at the time the defect was filed, and the developer can now trace backwards in time to see how the defect occurred!

Below is a screen shot that shows the configuration options for the Diagnostic Trace Collector.

This Collector is a natural to enable when the development team asks for more information regarding a specific bug report. While the default settings collect events raised by the applications under test, these applications can also be instrumented to facilitate tracing method level data to provide a more detailed picture of the state of the system during the test. The result file will be automatically included with any bug filed from the test runner when the test settings enable the collector, and will appear in the result file for any test run from Visual Studio with similar settings. Most of our diagnostic data adapters are available to tests executed through Microsoft Test and Lab Manager as well as the Visual Studio integrated development environment.

Double clicking the log file in a test result or a bug report will open the Trace Debugging Log viewer, shown below with a display of active threads running user code. This viewer is the access point to historical debugging based on the data collected during a manual test, for instance.

So, instead of a No Repro, any tester can now create a debuggable record of the test they performed with no special technical requirements necessary to collect this data!

Now that we’ve seen what a couple of these diagnostic data adapters can do we’re ready to go about authoring one of our own. We’ll tackle that challenge next time. The great part about our whole Agent-Controller and Diagnostic Data Adapter system is that teams can use the diagnostic data adapters we include or write their own to get all the utility they need out of their agents without installing the Visual Studio 2010 product on the agent or controller machines. It is as simple as dropping the new adapter assemblies into the correct folder and updating the test settings to make use of the new functionality – or simply using the out of the box adapters as they are. We’ll be going through these details and more next time, please do ask any more questions you may have as I want you to be as excited about this topic when you get your hands on the bits …

Diagnostic Data Adapters: Changing how Developers and Testers work together (Part 1 of 2 - The Test Impact Collector)

2009-04-11T10:05:00Z

Last time I talked about remote execution and introduced the concept of Diagnostic Data Adapters in Visual Studio 2010. In this post and the next, we will jump right into working with two of our favorite Diagnostic Data Adapters: the Test Impact Collector and the Diagnostic Trace Collector. Both of these components have a part to play in how developers and testers interact in Visual Studio 2010, making it easier for each discipline to unlock the potential of the other. While a key pieces of the diagnostic data adapter story is the freedom for teams to create their own adaptors and tailor their data collection to better suit their needs, we plan to include several diagnostic data adapters that we feel will be useful to many teams and projects right out of the box. The remainder of this post (and the next one) will go into detail about these two great diagnostic data adapters and how they help address a couple perpetual sticking points between developers and testers: “What did you test?” and “No Repro.”

The first point we address is the eternal question “What did you test?” This question is often asked by developers of testers, testers of each other, and management of everyone whenever a bug makes it into your customer’s hands. There are many ways different groups attempt to answer this question, but the Test Impact Collector aims to bridge the gap between developers and testers to show you what needs to be tested and which tests need to be run. In the world of development and unit tests it is a fairly straightforward effort to enable code coverage and see what parts of the code have been called by your tests. The hassle of collecting coverage data, and providing consistent coverage through manual tests, is beyond the expectation for many testers. The development team often knows only by feature area what the test team has been up to, and vice versa for the test team to know what the development team has been trying to fix. The Test Impact Collector serves a vital role here by identifying for development which code changes will or will not be exercised by the test team’s existing test catalog, while it also helps testers identify which tests from that catalog they need to run in order to validate the new behavior their developers have included in a given build.

Configuration of the Test Impact Collector is very simple. Manual testers using Microsoft Test and Lab Manager (codename Camano) will simply run tests according to the plan configured by their test manager or a specialist tester. Configuring the test plan simply requires creating a Test Settings for the tests to use and marking the appropriate diagnostic data adapters for use during a test run, as seen above. Each diagnostic data adapter exposes configuration options. For the Test Impact Collector there are two lists that can be used as either white or black lists of processes and modules to either include or exclude from instrumentation during the test run. The Test Impact Collector will collect the names of methods exercised during the tests. This allows the team to focus their efforts on just their code and reduce the number of modules that will be instrumented at the start of the test case, or, to eliminate modules that are known to not be of interest during the test.

This diagnostic data adapter allows Visual Studio to inform developers when their code changes will not be covered by the test team’s existing tests. Knowing this before the change is made will allow the development team to take extra care in reviewing their work or warning the test team before new or unexpected code shows up in the application under test. From the perspective of the test team, the Test Impact Collector helps answer the question “Which tests do I need to run in order to test the new changes?” Collections of test cases may be grouped by feature areas, but these sets may be much larger than necessary to test small changes within those areas. Rather than expecting teams to run all the tests within a given area, the Test Impact Collector helps teams reduce the number of tests they must run without leaving out any tests that do exercise the newly changed product code.

The above battery of test cases to run can be reduced to the below list with the help of the Recommended Tests option within Microsoft Test and Lab Manager.

The Recommended Tests feature takes into account results from the Test Impact Collector as well as the changes to the product source between any known builds associated with that Team Foundation Server project. These configuration options help test teams keep pace with new development by always running the test cases that will put the new changes through their paces.

Gone then are the days, when the test teams mindlessly run an entire battery of tests for a new build, irrespective of the new changes that showed up there. Now testing will be much more efficient and leveraged. As each new build comes out, the system will know the exact changes that have occurred, and the Test Impact collector will flag for the testers which subset of their test cases they need to run to validate the new functionality.

That’s pretty neat!

(Staying on the topic of Diagnostic Data Adaptors, and how Developers and Testers work together, I will talk about the Diagnostic Trace Collector in the next post).