+++
Visual Verification (VV) is the act of verifying that your application or component is displayed correctly on screen. The TestApi library provides a set of VV APIs. This post discusses these APIs.
First and foremost, I want to emphasize that visual verification is a test technique that should be used with caution. It is difficult to do correctly and any extensive use typically results in hard-to-maintain test codebases. Here are a few things to consider before you embark on VV test development:
The WPF test team has gone through several iterations of cleaning up and retiring unnecessary visual verification tests in an attempt to speed up and stabilize our test suite.
The core VV terminology is:
The general VV workflow is:
TestApi provides the following VV technology:
With these prolegomena out of the way, let’s look at some code. The first example below demonstrates master visual verification using a basic color verifier, which ensures that the difference between the master snapshot and the actual snapshot is within a defined tolerance:
// 1. Capture the actual pixels from a given window Snapshot actual = Snapshot.FromRectangle(new Rectangle(0, 0, 100, 100)); // 2. Load the reference/master data from a previously saved file Snapshot expected = Snapshot.FromFile("Expected.png")); // 3. Compare the actual image with the master image // This operation creates a difference image. Any regions which are identical in // the actual and master images appear as black. Areas with significant // differences are shown in other colors. Snapshot difference = actual.CompareTo(expected); // 4. Configure the snapshot verifier - It expects a black image with zero tolerances SnapshotVerifier v = new SnapshotColorVerifier(Color.Black, new ColorDifference()); // 5. Evaluate the difference image if (v.Verify(difference) == VerificationResult.Fail) { // Log failure, and save the diff file for investigation actual.ToFile("Actual.png", ImageFormat.Png); difference.ToFile("Difference.png", ImageFormat.Png); }
This approach works fine if you are evaluating the correctness of an application logo or some other application art. You may find that you will need to increase the tolerance a bit to accommodate differences in GPU rendering, but in general the SnapshotColorVerifier provides all the functionality you need.
A somewhat more sophisticated approach involves using of image histograms (see this link for a good introduction to the subject). An image histogram is a histogram that represents the frequency of pixels with a certain brightness. One can define a histogram that represents his/her expectation of the “proximity of the match” between the actual and expected snapshots.
For example, one can define a histogram semantically equivalent to the following statement:
“When I compare the actual snapshot to the expected snapshot (both of 320 pixels), I expect no more than 30 pixels with color channel difference of 1, no more than 10 pixels with color channel difference of 2, and zero pixels with higher differences.”
This histogram would look as follows:
Figure 1 Image Histogram
Such form of verification is done by using the SnapshotHistogramVerifier and the Histogram classes, as demonstrated in the sample below.
// Take a snapshot, compare to the master image and generate a diff Snapshot actual = Snapshot.FromRectangle(new Rectangle(0, 0, 100, 100)); Snapshot expected = Snapshot.FromFile("Expected.png")); Snapshot difference = actual.CompareTo(expected); // Load the quality histogram from disk and use it to verify the diff SnapshotVerifier v = new SnapshotHistogramVerifier(Histogram.FromFile("ToleranceHistogram.xml")); if (v.Verify(difference) == VerificationResult.Fail) { // Log failure, and save the actual and diff images for investigation actual.ToFile("Actual.png", ImageFormat.Png); difference.ToFile("Difference.png", ImageFormat.Png); }
The histogram file is just a XML file with the following schema:
<histogram> <tolerance> <point x="0" y="0.87500" /> <point x="1" y="0.09375" /> <point x="2" y="0.03125" /> <point x="3" y="0" /> <point x="4" y="0" /> ... <point x="254" y="0" /> <point x="255" y="0" /> </tolerance> </histogram>
This visual verification approach was pioneered in the WPF test organization about 6 years ago by Marc Cauchy and Pierre-Jean Reissman.
However, none of the two approaches above work particularly well for evaluation of a typical application window, containing controls, text, etc. Such windows tend to have regions that need different tolerance settings. For example, consider the application window below:
Figure 2 Sample Application Window
If you try to perform master based visual verification, you will hit 2 issues:
Issue (1) is easy to resolve, using Snapshot.FromWindow(...) and excluding the non-client area from the capture. Issue (2), however, is a bit more involved. Here are the expected, actual and diff snapshots of the client-area of the application.
Figure 3a Expected Client-Area Snapshot
Figure 3b Actual Client-Area Snapshot
Figure 3c Difference Snapshot
Figure 3d Difference Snapshot – Completely Black Pixels Are Replaced With Pink
The differences between the expected snapshot and the actual snapshot are difficult to see on Figure 3c, so on Figure 3d I have replaced purely black pixels with pink.
It is not surprising that most of the variation occurs around the text regions in the application window (ClearType renders differently on different machines). So it may make sense to increase the tolerance (or completely mask away) those regions, providing of course we are not specifically interested in their rendering.
In order to achieve that, we use the SnapshotToleranceMapVerifier class. Here’s an example:
// Take a snapshot, compare to the master image and generate a diff Snapshot actual = Snapshot.FromWindow(hwndOfYourWindow, ); Snapshot expected = Snapshot.FromFile("Expected.png")); Snapshot difference = actual.CompareTo(expected); // Load the tolerance map. Then use it to verify the difference snapshot Snapshot toleranceMap = Snapshot.FromFile("ExpectedImageToleranceMap.png"); SnapshotVerifier v = new SnapshotToleranceMapVerifier(toleranceMap); if (v.Verify(difference) == VerificationResult.Fail) { // Log failure, and save the actual and diff images for investigation actual.ToFile("Actual.png", ImageFormat.Png); difference.ToFile("Difference.png", ImageFormat.Png); }
The tolerance map that we use looks as follows:
Figure 4 Tolerance Map in “ExpectedImageToleranceMap.png”
What appears pure black (0x00FFFFFF) is actually an off-black color (0x000A0A0A) to handle the small variations that appear as black dots on Figure 3d. Then we also have 4 regions with significantly higher tolerance to handle the variability of the text rendering.
The visual verification API in TestApi provides a solid foundation for visual verification tests. As a general best practice, however, avoid visual verification as much as possible.