VSTS Quality Tools

Using Instrumented Builds to Analyze Testing and Find Dead Code

As part of my weekly duties since joining the Quality Tools team, I have been providing code coverage analysis to the team, ensuring that our automated tests cover a required percentage of our shipping code.  This has been a fun exercise for me, allowing me to both gain a better understanding of what code coverage means from a quality standpoint, as well as flexing my knowledge of PivotTables and charts in Excel to provide user-friendly results.  Performing basic code coverage of automated tests (such as unit tests) can be done by following the instructions at the Quality Tools blog at http://blogs.msdn.com/vstsqualitytools/archive/2005/06/08/426979.aspx.  A further dive into the details of Code Coverage can be found at https://blogs.msdn.com/ms_joc/articles/406608.aspx.

For this particular exercise, rather than analyzing the coverage of automation tests, we opted for a different approach: instrument a build and collect coverage information as the tester performs ad-hoc/manual testing.  The process (in a nutshell) looks like:

·         Build the Application Under Test (AUT)’s binaries.

·         Instrument the binaries that you wish to collect coverage on. (using vsinstr –coverage)

·         Create a Setup Package containing the instrumented assemblies.

·         Each Tester installs the Setup Package, and turns on data collection (using vsperfcmd)

·         Each Tester performs a pass through the AUT.

·         When vsperfcmd is shut down, a .coverage file is generated.

·         Analyze the results of all of the .coverage files.

·         Merge the separate .coverage files into a single result.

Now, once you’ve accomplished the work above, you’ve got yourself two important artifacts:

·         One coverage file for each tester’s work.

·         One coverage file that merges all tester’s work.

The two of these can used to be answer some interesting questions about both the testing process, and the application under test.

·         What classes/methods were never covered?

o   How can you tell?  Look for classes/methods with 0% coverage in the merged result.

o   What does this mean?  This generally indicates either scenarios that aren’t being tested, or “dead code” that, while covered by the automated tests that were created when it was checked in, is no longer used by the application.

o   What else can I tell from this?  What kind of classes are getting 0% coverage?  If there are lots of exception classes that are uncovered, this may indicate that testers aren’t evaluating error conditions (ex: opening a file from the application with invalid data).  You may also find that there was an application state (such as docked, minimized, or placed in the system tray) that is required to test certain code paths.

·         Is each tester testing the same things?

o   How can you tell?  Look for classes where the coverage for each tester is reasonably similar to each other and the overall merged result. 

o   What does this mean?  If the merged result is higher than the top individual tester coverage, this indicates that each tester exercised different code.  On the other hand, a class where all testers cover ~70%, and the merged result is also ~70%, this indicates that all testers were covering the same scenarios and code paths.

Workaround to upgrade VSTS 2005 Test Projects in Visual Studio Team System 2008 – Development Edition

Some users report an issue in Visual Studio Team System (VSTS) 2008 – Development Edition. In the Development Edition, they failed to upgrade VSTS 2005 Test Projects. We have verified this is a product defect only in  VSTS Development Edition and Visual Studio Professional Edition. We understand the impact of this issue, we are working to produce an update to resolve it. In the meantime, if you are looking for a workaround, please look at this article that we posted to our MSDN forum: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2534717&SiteID=1

We encourage you visit our forum often – ask questions and give us feedback, we will make our product better.

Thanks,

Christine Zhao

Visual Studio Team System Test

What's new in November 2007 CTP: Automation for Navigation

Manual Test Runner (MTR) is one of the key portions of Rosario with which we are hoping to capture the imagination of the manual tester. Read more...

AfN-Automation for Navigation is a feature that we are adding to MTR to enrich the testing experience, to facilitate the tedious manual testing! It helps the tester to automate the repetitive and time consuming parts of a manual testcase so that more focus can be laid on testing the functionality of the application, thus increasing efficiency.

What happens exactly is that when the tester does manual testing for the first time, we start a background recorder that captures all the manual actions and creates a recording strip which is essentially an XML file containing all the actions performed and is exposed to the tester. The strip has markers inserted at all those points where a step was marked in the middle of the recording. Here is a snippet of the XML file that’s generated:

<?xml version="1.0" encoding="utf-8"?><UITest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" Name="" Id="83a5f887-245b-4a71-9d5e-20def6ebfd6d" Version="1.0">

<InitializeActions />

  <ExecuteActions>

    <TestStepMarker MarkerInformation="">

      <StepId>0</StepId>

    </TestStepMarker>

    <MouseAction UIObjectName="CurrentUIMap.ItemWindow.RunningApplicationsToolBar.VshostexeButton">

      <ModifierKeys>None</ModifierKeys>

      <Location X="53" Y="13" />

      <WheelDirection>0</WheelDirection>

      <ActionType>Click</ActionType>

      <MouseButton>Left</MouseButton>

    </MouseAction>

    <MouseAction UIObjectName="CurrentUIMap.ApplicationCustom.Run1202PM24MicrosoftMenuItem">

      <ModifierKeys>None</ModifierKeys>

      <Location X="68" Y="15" />

      <WheelDirection>0</WheelDirection>

      <ActionType>Click</ActionType>

      <MouseButton>Left</MouseButton>

    </MouseAction>

    <TestStepMarker MarkerInformation="">

      <StepId>2147483647</StepId>

    </TestStepMarker>

</ExecuteActions>

Whenever a step is marked, a visual indiactor immediately appears in the same row as the step. Random marking of steps is allowed but markers are not inserted in all cases. If a step is marked in the forward direction without doing any AUT actions, the last inserted marker moves to this step. If a step is marked in backward direction, no marker movement occurs, only tooltips advising to mark steps in order, appear. An implicit start marker is inserted when a testcase is started and an implicit end marker is inserted on ending the testcase.

The tester will have a choice of switching on/off the background recorder, the video recorder and the action log and also to exclude applications to be recorded. Throughout the testcase execution, this information is displayed to her.

Fig 1: The tooltip shows the default recording settings. On clicking on the ‘Edit’ link, the Recording Settings dialog pops up

Fig 2: The Recording Settings showing available choices of loggers and AUTs

Fig 3: After the TC is started, hovering the mouse over the status bar shows which loggers are on and what all AUTs are being recorded

So when the same testcase is opened again, the UI shows a visual indication at the points where a marker was inserted. The tester has a choice of skipping some sections by playing back between a chosen set of markers to speed up her navigation through the testcase or overwrite the existing markers and recordings. In the end, the original and the new strips are merged and a choice is given to the tester to either keep the original recording or save the new merged recording.

Fig 4: On opening the TC again, the available markers are displayed which can be played back or re-recorded

Fig 5: After re-recording, when TC is ended, tester is given options to save/discard new markers

With so much functionality, MTR is sure to be a boon for all manual testers!

Microsoft ® Visual Studio® Team System code name “Rosario” November 2007 CTP: Team Test

What’s new from Team Test?
We’re excited to show some of the features in our Test Case Management Solution! It’s all about creating a simple product that will be easy to use and will make your life easier. If manual tests are part of your daily work, and the process includes writing your test cases in a word processor, excel, or any other tool, we have something cool to offer. If you want to keep track of your projects in an easy way, knowing what is left to do and what has been done, we have something for you!

How does it work?
You are a test manager. Your task today is to create a test plan for the following week. The test plan will contain the test cases that testers need to run in order to sign-off and release any given product that you’re currently working on. Your day could look like this:
You get to your office, and open your test case management solution:

You can see the several Activity Centers: Home, Planning, Testing, Defects, and Reports. You decide to go to Planning, in order to create a Test Suite and add some test cases to it. A Test Suite is just a hierarchy of folders in which you will host your test cases:

After you create your test suite and have added test cases to it, you create a test configuration, which will be the configuration to be used when running your test cases (in our example, it will be on a VISTA OS in the English language).

Now that you have your test suite with test cases in it and an actual test configuration, you can go ahead and create your test plan. A test plan will contain one or more test passes. A test pass is just a combination of a test suite with a test configuration.

The test plan is done, and your testers are ready to start running their manual tests! By the way, if you are curious about what a test case looks like, here it is:

Your  testers can now go to the Testing Activity Center, choose the test plan that you just created, and start running the test cases assigned to them by using the Manual Test Runner. Results can be published to the server and seen in the test case management solution right away!

We want to hear from you!
Please leave your comments or any kind of feedback. We are in the early stages of development, and we want to hear you!

Download the CTP here!

Thanks,
David Gorena Elizondo

Come chat with the Visual Studio Team System group – December 5th

Join members of the Visual Studio Team System product group to discuss features available in Team Foundation Server, Team Suite, Architecture Edition, Development Edition, Database Edition, and Test Edition. In addition, discuss what's new for these editions for Visual Studio 2008.

We will be holding two sessions:

Join the chat on Wednesday, December 5th, 2007 from 10:00am - 11:00am Pacific Time. Add to Calendar | Additional Time Zones

                -and-

Join the chat on Wednesday, December 5th, 2007 from 4:00pm - 5:00pm Pacific Time. Add to Calendar | Additional Time Zones

Running Unit tests as MTA rather than STA

Ever done complex interop testing where you wanted to run as MTA? Needed to call WaitHandle.WaitAll? Checkout this post from Mark Seemann, which has the gory details.

Test Edition 2008 Documentation Posted

The Visual Studio 2008 Beta 2 documentation is live. Team System Test Edition can now do a lot more for you. (For example, you can now call one Web test from another Web test, and run unit tests in Visual Studio Pro.) To see a full list of what's new in Test Edition, look here.

When you've used any of these new areas, and turned to the Help topics to learn more, please let us know how well the online Help worked for you! To do that, use the Click to Rate and Give Feedback tool at the top of any Help topic. We very much appreciate getting your feedback -- thanks!

Right-Click Test Execution

When working with unit tests it is often useful to be able to run the current test method or all tests in the current test class.  This can be accomplished via the new Run Tests menu option on the unit test context menu (Fig. 1).  As you can see, the Run Tests menu option sits at the top of the unit test context menu, below the refactoring options.  Clicking this menu option will execute your tests in the current scope—where you are in your test project.  For example, if you right-click inside a test method, Run Tests will run just that test method.   If you right-click inside a test class, but not inside an actual test method—i.e. a method with the TestMethodAttribute attribute—Run Tests will run all the tests in the test class.  Last but not least, if you right-click inside a test project but neither inside a test method nor inside a test class, Run Tests will run all tests in the test project.

Figure 1: Unit Test Context Menu

Test Generation from a Compiled Assembly

August 2007 Rosario CTP: Test Case Management

Test Case Manager is a new feature in the Microsoft^® Visual Studio^® Team System code name “Rosario” August 2007 CTP. TCM allows you to do some exciting new things. One of these is creating a Test Case with test steps. These steps let the test author tell the manual tester how to perform the test. Below is a basic walk through of adding steps and parameters to a Test Case. I also included a few questions for you at the end. They will help us know what you think about TCM. Plus it’s exciting to get feedback from customers that are using our product.

Open Test Case Manager:

1.       Connect to your Team Foundation Server

2.       Click on your project in the Team Explorer

3.       On the main VS Menu bar click on Test

4.       Go to the bottom of the drop down box to Windows

5.       Then click on Test Case Manager

Create a new Test Case:

1.       Click on the Add New Test Case button on the Test Case Manager tool bar

Add steps to the Test Case:

1.       Click on the Steps tab

2.       Click <click here to add a step> in the Title column

3.       Type in what the tester needs to do for the step

4.       In the Expected column type the expected results

You can also add two different types of parameters to a test step. Both types of parameters can be created in either the “Title” or “Expected” columns of the test step.

Data Bound parameters – Bound parameters created in the Title column provide the data that needs to be used during the execution of the step. An example would be “Login using @username @password”. Bound parameters created in the Expected column provide the tester the expected result that needs to be verified. An Example would be “User’s @emailaddress”.

Unbound parameters – Unbound parameters created in the Title column let the tester know that a value needs to be used but was not known at the time the test was created. An example of this would be “Enter in current @?date”. Unbound parameters created in the Expected column let the tester know a result needs to be validated during the execution of this test. But the value was not known at the time of the creation of the parameter. An example of this would be “Number of Today’s @unreademails”.

After that explanation are parameters making sense? How do you feel about the concept of Data bound and Unbound parameters?

Basic rules of parameters:

·         Use @ for Data bound and @? for Unbound

·         The name of the parameter cannot contain spaces

·         The first character of the name of the parameter needs to be a letter

·         You can use a parameter in more than one step

·         You can also use a parameter in both columns

Add data to the parameter:

1.       Click on the Data button

2.       In each row add the data for one iteration of the test

Now that you have seen a basic walk through of creating new Test Case with steps and parameters, what do you think of the new feature?

What else do you think a manual tester would need available in a Test Case?

Is there any part of the Test Case that you don’t think you would use, and what would be your reason for not using it?

Finally what other features of TCM would you like us to blog about?

You can find more information on “Rosario” and the August 2007 CTP at http://go.microsoft.com/fwlink?LinkID=96097

Thanks, 

Rusty Scrivens

First Rosario CTP now available

Jeff Beehler wrote about the release of Microsoft^® Visual Studio^® Team System code name “Rosario” August 2007 CTP.  I want to call your attention to what Team Test has added in this preview that are worth checking out.  If you have been asking us to do more around test management and manual tests, you'll definitely want to take a look at this.

From Jeff's blog:

The download is available here.  We're looking forward to your feedback!

Cheers,

David R. Williamson, Engineering Lead

TechEd 2007 sessions now available!

You can find the 2007 TechEd sessions posted at http://msdn2.microsoft.com/en-us/teamsystem/bb676080.aspx, including my talk on Web and Load testing in VS2008 (codenamed Orcas).

 Cheers,

David R. Williamson

Rosario: Next Generation of Team Test (almost) out in the wild - Rosario Aug CTP

Visual Studio Team System Chat – August 1st

Join members of the Visual Studio Team System product group to discuss features available in Visual Studio Team Foundation Server, Team Editions for Architects, Developers, Database Pros, and Testers. In addition, discuss what's new in the latest Visual Studio 2008 CTP.

We will be holding two sessions:

Join the chat on Wednesday, August 1^st, 2007 from 10:00am - 11:00am Pacific Time. Add to Calendar | Additional Time Zones

            -or-

Join the chat on Wednesday, August 1^st, 2007 from 4:00pm - 5:00pm Pacific Time. Add to Calendar | Additional Time Zones

New Team System Case Study: Getty Images

We have published a new case study

http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=4000000157

"Imagery, film, and digital services provider Getty Images was losing thousands of hours of software development productivity per month due to a slow and unreliable source code management system. The company switched from MKS Integrity to a software life cycle management solution based on Microsoft® Visual Studio® 2005 Team Foundation Server, a component of Visual Studio Team System, which eliminated all major problems faced in the past. Better performance, reliability, and ease of use have improved software quality and made the development group 10 to 15 percent more productive. New collaborative abilities enable people to work more closely together, and improved reporting allows them to monitor project status and progress at a glance. Moreover, the move to Team Foundation Server cost less than it would have to stay with MKS Integrity."