In the previous blog post Analyzing Test Results in MTMâ€“ Part 1 we have seen how to find the run and a snapshot of various test run analysis features. In this blog post we will look at Analyzing a test failures using Test case result view and how to get a final report. In the sample run provided in the blog post there were 11 inconclusive results and 5 failures. We have analyzed 11 inconclusive results in the previous post using Test run view. Mow letâ€™s analyze the 5 failures.
There are 5 failures in the run and letâ€™s analyze each one of them using the above features. Since these are genuine failures in the run we need to open each test result to get more information on the failure from the logs. As usual user can double-click on the test result or Click â€œView Resultsâ€ Button to view the test case results. The below analysis screen comes up where it displays all the data that the user is looking for. Browse through the below screenshot and take a look at the data displayed in each of these fields.
In the above screenshot you can see related items Combo Box which will let you open the Test plan, Test run and the test case that the test result belongs to.
Based on the data collectors turned on in the test settings, user will find all the log files associated to the run in the attachment section. If you have enabled video collector, you shall even look at what exactly happened during the test execution. Users can also write custom data collectors to gather specific information that they wanted from the test machine which will help them analyze the results. Note that if the attachments are huge and coming from multiple agents there may a time lag for all the attachments to get into TFS. In these cases you will attachments with 0KB of size, this means the files are getting uploaded to the server.
Even the TRX file has been saved here. Opening the TRX file will open it VS IDE. User can investigate the failure and then update the analysis information in the results individually.
Users can view the past test result history in the same view where they can find if the same issue has happened earlier. \
Based on the investigation users can update the test result with analysis information. The below screenshot shows updated fields. Like this we shall analyze all the other 4 failures.
Notice that unless there are no failures with failure type value â€œNoneâ€ or Resolution type of â€œNoneâ€ or â€œNeeds Investigationâ€, the analysis is not complete. Since all this analysis information is preserved, it is meaningful to analyze all run failures and update all the fields with appropriate details so that if the same issue happens in future it can easily be auto-analyzed. Also all the analysis data gets moved to warehouse where users can get historical reports about the past test runs and can get cool and meaning reports using excel reporting feature.
After the analysis of the tests the final results looks like this. The user get to see some pie charts which gives a snapshot of the results based on failed test and analysis of those tests
With this blog post Analyzing Test Results in MTM series is complete. In this blog we have seen following in detail
- Analyzing a test failures using Test case result view
- Analysis complete indicator and final report
Click here to get to Analyzing Test Results in MTM - Part I
Author – Venkatesh Sargunesan
Test Lead – CodedUITest