A custom data diagnostic adapter for Visual Studio Coded UI Test using Windows PSR

A custom data diagnostic adapter for Visual Studio Coded UI Test using Windows PSR

Rate This
  • Comments 4

 

This is a guest blog from the Windows Azure AppFabric team. They are using Visual Studio Coded UI Test and want to share an exciting idea for generating rich troubleshooting logs.

Hello, my name is Jerrin Elanjikal and I’m a test developer with the Windows Azure AppFabric product group at Microsoft IDC. My team develops automated UI test frameworks and tools to ensure the quality of our user interface components. We deal a unique set of technical challenges around UI test automation and debugging, which is very different from both API test automation and manual UI testing.

 

Our product has multiple user interfaces including a Silverlight web portal, Visual Studio designer surfaces, WPF dialogs and a few legacy WinForms dialogs. We target 85% code coverage through automated UI test cases. And most of our UI automation is done using Coded UI, along with a mix of internal frameworks for specialized functions.

 

When one of these test fail and you have to debug or reproduce the failure, we run into a few common pain points that are unique to UI test automation: 

1. The average test takes a lot more time to run as compared to your typical API unit test. Our tests on average take about 10 minutes each.

2. And while a UI test is running, you cannot use your computer for anything else that involves user interaction. This means that you either keep a dedicated second machine for debugging UI test failures, or just sit back and watch.

3. UI tests can fail randomly due to a number of external reasons - a pop up thrown by another application, someone accidently logging into the machine, flaky internal test frameworks and the list goes on. It’s almost impossible to reproduce or debug such failures on your development machine.

 

The standard solution to these problems involves diagnostic logs and video recording of the test automation, so that you can debug without having to re-run the test. Often logs alone are not sufficient. Video recordings take up a lot of disk space, typically a few MB per test case. And you spend a lot of time watching the video and looking out for something interesting to happen.

 

Fortunately, there is a better alternative. The 'Problem Steps Recorder' is an awesome little tool that comes built in with Windows 7 and later. PSR was originally intended to help customers explain their problem scenario to tech support. What it does is take screenshots of each UI action and generate a zipped html report with images and descriptions. Looking at a PSR report is like watching the highlights of a match - short & interesting. It’s every UI developer's dream come true!

 

So, what we did is plug PSR into our CodedUI tests by writing a custom diagnostics data adapter that essentially just invokes psr.exe with the right parameters.  This MSDN article < http://msdn.microsoft.com/en-us/library/dd286727.aspx > describes how to write and install a custom diagnostics data adapter for Visual Studio. And the command line parameters for PSR are as follows:

 

psr.exe [/start |/stop][/output <fullfilepath>] [/sc (0|1)] [/maxsc <value>]

    [/sketch (0|1)] [/slides (0|1)] [/gui (o|1)]

    [/arcetl (0|1)] [/arcxml (0|1)] [/arcmht (0|1)]

    [/stopevent <eventname>] [/maxlogsize <value>] [/recordpid <pid>]

 

/start                        :Start Recording. (Outputpath flag SHOULD be specified)

/stop                        :Stop Recording.

/sc                          :Capture screenshots for recorded steps.

/maxsc                        :Maximum number of recent screen captures.

/maxlogsize                :Maximum log file size (in MB) before wrapping occurs.

/gui                        :Display control GUI.

/arcetl                        :Include raw ETW file in archive output.

/arcxml                        :Include MHT file in archive output.

/recordpid                :Record all actions associated with given PID.

/sketch                        :Sketch UI if no screenshot was saved.

/slides                        :Create slide show HTML pages.

/output                        :Store output of record session in given path.

/stopevent                :Event to signal after output files are generated.

 

To start PSR recording at the beginning of each test case, we invoke psr.exe with the parameters /start /output <PSRFileName> /gui 0 /sc 1 /sketch 1 /maxsc 100 in the OnTestCaseStart event handler of our custom diagnostics data adapter.

 

To complete the PSR recording and generate the zipped report we invoke psr.exe /stop in the OnTestCaseEnd event handler, with an appropriate delay to complete the report file creation.

 

But PSR records a screenshot only when a UI action is performed. If the test fails because a UI element was not found on screen, as is often the case, PSR will not record a screenshot at that point. So we add the following code to the OnTestCaseEnd event handler to take an extra screenshot at the point of failure.

 

Playback.Initialize();

Image img = UITestControl.Desktop.CaptureImage();

Playback.Cleanup();

img.Save(ScreenshotFileName, ImageFormat.Jpeg);

 

We then attach the PSR report and the final screenshot to the test case.

 

dataSink.SendFileAsync(e.Context, ScreenshotFileName, false);

dataSink.SendFileAsync(e.Context, PSRLogFile, false);

 

The complete code for the PSR diagnostics data collector is given below:

 

using System;
using
System.Diagnostics;
using
System.Drawing;
using
System.Drawing.Imaging;
using
System.IO;
using
System.Threading;
using
System.Xml;
using
Microsoft.VisualStudio.TestTools.Common;
using
Microsoft.VisualStudio.TestTools.Execution;
using
Microsoft.VisualStudio.TestTools.UITesting;
 
namespace
PSRDataCollector
{
   
// Problem Screen Recorder diagnostic data adapter

    [
DataCollectorTypeUri("datacollector://Microsoft/PSRDataCollector/1.0")]
    [
DataCollectorFriendlyName("Problem Screen Recorder", false
)]
   
public class PSRDataCollector : DataCollector

    {
        #region Constants
 
       
private const string PSRExe = "psr.exe"
;
       
private const int
SaveDelay = 5000;
 
        #endregion
Constants
 
        #region
Fields
 
       
private DataCollectionEvents
dataEvents;
       
private DataCollectionLogger
dataLogger;
       
private DataCollectionSink
dataSink;
       
private XmlElement
configurationSettings;
       
private string
PSRLogFile;
 
        #endregion
Fields
 
        #region
DataCollector
 
       
// Required method called by the testing framework

       
public override void Initialize(XmlElement configurationElement,
                                       
DataCollectionEvents
events,
                                       
DataCollectionSink
sink,
                                       
DataCollectionLogger
logger,
                                       
DataCollectionEnvironmentContext
environmentContext)
        {
            dataEvents = events;
// The test events

            dataLogger = logger;
// The error and warning log
            dataSink = sink;    
// Saves collected data
           
// Configuration from the test settings
            configurationSettings = configurationElement;
 
           
// Register common events for the data collector
           
// Not all of the events are used in this class
            dataEvents.SessionStart +=
               
new EventHandler<SessionStartEventArgs>(OnSessionStart);
            dataEvents.SessionEnd +=
               
new EventHandler<SessionEndEventArgs
>(OnSessionEnd);
            dataEvents.TestCaseStart +=
               
new EventHandler<TestCaseStartEventArgs
>(OnTestCaseStart);
            dataEvents.TestCaseEnd +=
               
new EventHandler<TestCaseEndEventArgs
>(OnTestCaseEnd);
            dataEvents.DataRequest +=
               
new EventHandler<DataRequestEventArgs
>(OnDataRequest);
        }
 
       
protected override void Dispose(bool
disposing)
        {
           
if
(disposing)
            {
                dataEvents.SessionStart -=
                   
new EventHandler<SessionStartEventArgs
>(OnSessionStart);
                dataEvents.SessionEnd -=
                   
new EventHandler<SessionEndEventArgs
>(OnSessionEnd);
                dataEvents.TestCaseStart -=
                   
new EventHandler<TestCaseStartEventArgs
>(OnTestCaseStart);
                dataEvents.TestCaseEnd -=
                   
new EventHandler<TestCaseEndEventArgs
>(OnTestCaseEnd);
                dataEvents.DataRequest -=
                   
new EventHandler<DataRequestEventArgs
>(OnDataRequest);
            }
        }
 
        #endregion
DataCollector
 
        #region
Event Handlers
 
       
public void OnSessionStart(object sender, SessionStartEventArgs
e)
        {
           
// TODO: Provide implementation

        }
 
       
public void OnSessionEnd(object sender, SessionEndEventArgs e)
        {
           
// TODO: Provide implementation

        }
 
       
public void OnTestCaseStart(object sender, TestCaseEventArgs e)
        {
           
this.PSRLogFile = Path.Combine(Environment.CurrentDirectory, e.TestCaseName + ".psr.zip"
);
 
           
// Kill all running PSR processes

            TryKillProcess(PSRExe);
 
           
// Start PSR
           
try
            {
                InvokeProcess(PSRExe,
String.Format("/start /output \"{0}\" /gui 0 /sc 1 /sketch 1 /maxsc 100", this.PSRLogFile));
            }
           
catch (Exception
exp)
            {
                dataLogger.LogError(e.Context,
string.Format("Unexpected exception while trying to start PSR process : {0}"
, exp));
            }
        }
 
       
public void OnTestCaseEnd(object sender, TestCaseEndEventArgs
e)
        {
           
try

            {
               
// Save the PSR logs
                InvokeProcess(PSRExe,
@"/stop").WaitForExit(60000);
 
               
// Sleep to ensure PSR completes file creation operation

               
Thread.Sleep(SaveDelay);
 
               
if (!File.Exists(this
.PSRLogFile))
                {
                    dataLogger.LogError(e.Context,
"No user actions were recorded by PSR!"
);
                }
               
else if (e.TestOutcome != TestOutcome
.Passed)
                {
                   
string ScreenshotFileName = Path.Combine(Environment.CurrentDirectory, e.TestCaseName + ".screenshot.jpg"
);
                    CaptureScreenshot(ScreenshotFileName);
                    dataSink.SendFileAsync(e.Context, ScreenshotFileName,
false
);
 
                    dataSink.SendFileAsync(e.Context,
this.PSRLogFile, false
);
                }
               
else

                {
                   
File.Delete(this.PSRLogFile);
                }
            }
           
catch (Exception
exp)
            {
                dataLogger.LogError(e.Context,
string.Format("Unexpected exception while trying to stop PSR process : {0}"
, exp));
               
// Kill all PSR processes

                TryKillProcess(PSRExe);
            }
        }
 
       
public void OnDataRequest(object sender, DataRequestEventArgs e)
        {
           
// TODO: Provide implementation

           
// Most likely this occurs because a bug is being filed
        }
 
        #endregion Event Handlers
 
        #region
Helpers
 
       
private Process InvokeProcess(string processName, string
parameters)
        {
           
ProcessStartInfo startInfo = new ProcessStartInfo
(processName, parameters);
            startInfo.WindowStyle =
ProcessWindowStyle
.Hidden;
            startInfo.UseShellExecute =
true
;
            startInfo.ErrorDialog =
false
;
 
           
Process proc = new Process
();
            proc.StartInfo = startInfo;
            proc.Start();
 
           
return
proc;
        }
 
       
private void TryKillProcess(string
processName)
        {
           
Process[] processes = Process
.GetProcessesByName(processName);
           
foreach (Process proc in
processes)
            {
               
try
{ proc.Kill(); }
               
catch (Exception
exp) { }
            }
        }
 
       
private void CaptureScreenshot(string
ScreenshotFileName)
        {
           
Playback
.Initialize();
           
Image img = UITestControl
.Desktop.CaptureImage();
           
Playback
.Cleanup();
            img.Save(ScreenshotFileName,
ImageFormat
.Jpeg);
        }
 
        #endregion
Helpers
    }
}

 

You can download the full project from http://psr4vs.codeplex.com.

 

To install the PSR data collector into Visual Studio, download and build this project. Copy PSRDataCollector.dll to %VSINSTALLDIR%\Common7\IDE\PrivateAssemblies\DataCollectors.

 

To enable PSR data collection, open test settings from Solution Explorer in your CodedUI test and enable ‘Problem Screen Recorder’ under the ‘Data and Diagnostics’ tab.

 

image

 

Run your CodedUI test again and let it fail. Wait for the test run session to complete. Now, you will find the PSR report and final screenshot attached to the test report.

image

 

Debugging automated UI tests just got a whole lot easier!

 

Leave a Comment
  • Please add 1 and 4 and type the answer here:
  • Post
Page 1 of 1 (4 items)