CodedUITest can be executed in many ways, from VS IDE, MSTest.exe and from Microsoft Test Manager. If the user has run the tests from VS IDE or MSTest, he can publish the results to TCM Server using the publish button in the test results viewer in VS IDE. Once he publishes the results to TCM Server, he can use Microsoft Test Manager (MTM) to analyze the results. In this blog post we will look into how to analyze a test run using MTM.
To understand the MTM UI workflow and the features please take a look at Activity Center and Testing vs Planning blogs. These blogs talk about how MTM UI is laid out and how users can navigate to various views. Please read these blogs before proceeding further if you are new to MTM.
All Test runs are stored under Testing Center -> Test -> Analyze Test Runs in MTM. By default this view shows all the automated test runs that are executed in the last 14 days.
User can customize this view by clicking on â€œShow manual runsâ€ to show manual test runs executed from Test Runner or change the start date range value to expand or narrow their search. Below screenshot shows how to customize the Test run View.
User can customize this view by clicking on â€œShow manual runsâ€ or change the start date range value
First letâ€™s look at how to find the run that the user just started or the run just completed. User can find the run based on run build number, Owner and creation date. Like all the views in MTM user can filter column values and group runs using columns names. User can even group runs based on column. Once the user find the test run user can click on Open button on the top or double-click on the run to view the results. Below screenshot shows how to filter the test runs, owner column is user for grouping and build number is used for filtering.
The run details that you are seeing in the below screenshot shows all the aspects of the run like test settings, Controller, lab environment, run type and owner. It also shows the charts on the analysis information and users can also open the associated test case if it needs to be updated. Now letâ€™s analyse this run.
In the above sample run there are 11 inconclusive results which the test has thrown based on the configuration it is run on. These tests were run on this config by mistake so letâ€™s analyze these tests with â€œFailure typeâ€ as â€œKnown issueâ€ and â€œResolutionâ€ as â€œConfiguration issueâ€. Notice that you can select multiple items and analyze them together.
Users can choose one of the 5 options as failure types based on the reason of the failure and â€œNoneâ€ is the default value. Failure type can be a Regression if the same issue was found and fixed earlier and it has regressed again. If the bug is not a known issue not it is a regression, we shall mark the failure as New issue. In this case this is known issue where the test should not have been run, so letâ€™s select â€œknown issueâ€
Users can choose one of the 5 options as resolution and â€œNoneâ€ is the default value. In the below example we have set the resolution as â€œConfiguration issueâ€ where this problem happens in this particular config. If we have analyzed the failure as a code defect we shall mark the resolution as Product issue or if the failure happens because of a test defect, then we shall mark the resolution as test issue. We shall mark the resolution as â€œNeeds investigationâ€ if we are just browsing through the failures and analyzing the simple and easy ones first and look at the hard ones later, users can select this option .
The button highlighted in yellow in the below snapshot will reset the test result and the related test will run again on the controller agent. If there was any mass failures, users can reset all the failures so these tests can rerun again. MTM will reset the test results to active so that test controller can pick them up. Users can even reset a passed test case, if there is a need.
â€œNotesâ€ is where users can details on the investigation and why the test failed. This data will be very handy when the user has found a similar issue in later builds or when the user files a bug this details will be moved to the bug for the developer to take a look.
Once the failures are analyzed, users can file bugs or link existing bugs to the test results by clicking on â€œCreate bugâ€ or â€œLink to Bugâ€ buttons.
In this blog on Analyzing Test Results in MTM â€“ Part 1 we saw the following in detail
Â· Find the run from â€œAnalyze Test runsâ€ view
Â· Features of Test run analysis
Â· Analyze some of the failures from Test run view itself
In the part-II blog we shall see the following
Â· Analyzing a test failures using Test case result view
Â· Analysis complete indicator and final report
Author – Venkatesh Sargunesan
Test Lead – CodedUITest