Ivo Manolov's Blog : TestApi

Introduction to TestApi – Part 5: Managed Code Fault Injection APIs

ivom — Wed, 25 Nov 2009 08:47:00 GMT

Fault Injection is the act of artificially changing the behavior of an existing executable code to simulate various faults. FI is very useful for validation of error handling code paths and for improving code coverage.

There are several types of fault injection. In runtime fault injection, the fault injecting test modifies the execution logic of the application under test (AUT), by injecting faults, triggered by specific runtime conditions. One could for example implement a FI test with the following semantic:

Throw an out-of-memory (OOM) exception, whenever the application calls method CreateWidget of class WidgetManager.

The FI terminology is as follows:

AUT (application under test) – this is the tested application, in which faults are being injected;
Fault Rule – The fault rule is a central construct in an FI test that determines WHEN faults get triggered and WHAT TYPES of faults get triggered. A fault rule consists of:
- a Method Signature, determining the method where the fault will be injected;
- a Fault Condition, determining when the specific fault should be triggered (e.g. every N^th call)
- a Fault -- Determines the type of fault (e.g. throwing an exception, returning a specific value, etc.) that occurs when the fault condition is met.
Fault Session – The fault session is a collection of fault rules that are applied to a given AUT.

TestApi provides a simple, but powerful runtime fault injection API for injecting faults in managed code. The API was originally designed and implemented by Bill Liu et al from the “Essential Business Server” team, and adapted to TestApi by Sam Terilli from our WPF XAML team. The following content provides a quick introduction to the API.

Sample AUT

Following is the code of a trivial AUT that we will use for demonstration purposes:

//
// This is a sample application used for demonstration purposes.
//
using System;

class MyApplication
{
    static void Main(string[] args)
    {
        int a = 2;
        int b = 3;

        for (int i = 0; i < 10; i++)
        {
            Console.WriteLine("{0}) {1} + {2} = {3}", i, a, b, Sum(a, b));
        }
    }

    private static int Sum(int a, int b)
    {
        return a + b;
    }
}

The result of running this application is of course:

> MyApplication.exe 
0) 2 + 3 = 5 
1) 2 + 3 = 5 
2) 2 + 3 = 5 
3) 2 + 3 = 5 
4) 2 + 3 = 5 
5) 2 + 3 = 5 
6) 2 + 3 = 5 
7) 2 + 3 = 5 
8) 2 + 3 = 5 
9) 2 + 3 = 5

A Simple Fault Injection Test

Now, let’s try to inject a fault in the AUT. Let’s assume that we want to modify the return value of Sum. Here’s how we can accomplish that:

// 
// Simple fault injection test
//
using System;
using System.Diagnostics;
using Microsoft.Test.FaultInjection;

public class FaultInjectionTest 
{ 
    public static void Main() 
    { 
        // 
        // Set up a fault rule to return –1000 the second time Sum is called.
        //
        string method = "MyApplication.Sum(int,int)";
        ICondition condition = BuiltInConditions.TriggerOnNthCall(2);
        IFault fault = BuiltInFaults.ReturnValueFault(-1000);
        FaultRule rule = new FaultRule(method, condition, fault);            

        //
        // Establish a session, injecting the faults defined by the fault rule(s)
        //
        FaultSession session = new FaultSession(rule);
        ProcessStartInfo psi = session.GetProcessStartInfo(@".\MyApplication.exe"); 

        //
        // Launch the target process and observe faults
        //
        Process p = Process.Start(psi);        
        p.WaitForExit();
    }
}

Fairly straightforward. Upon running the test (which will itself spawn the AUT) we observe the following output:

> FaultInjectionTest.exe 0) 2 + 3 = 5 1) 2 + 3 = -1000 2) 2 + 3 = 5 3) 2 + 3 = 5 4) 2 + 3 = 5 5) 2 + 3 = 5 6) 2 + 3 = 5 7) 2 + 3 = 5 8) 2 + 3 = 5 9) 2 + 3 = 5

As intended, we injected a runtime fault in MyApplication.exe, which resulted in Sum returning –1000 the second time it got called.

Under The Covers

Under the covers, the managed code Fault Injection API uses the CLR profiling API to modify the prologue of the intercepted method at runtime in order to inject the desired fault. The injected prologue instructions essentially call a method in the library, which then dispatches the call to the specified fault.

Because faults are injected at runtime, the code of the original application is not modified in any way. There is a certain performance degradation, which depends on the number of the injected faults.

The fault injection API provides a variety of built-in conditions and faults (in the BuiltInConditions and BuiltInFaults classes respectively). Users of the API can also create custom conditions and faults (by implementing the ICondition and IFault interfaces respectively). The API also provides a set of classes that expose the ability to fine tune and monitor the injected faults.

In addition, the API provides a facility to set “global faults”, which is useful for server application testing, where one application typically consists of and recycles many different processes.

I have attached the sample above, which should get you up and running with fault injection.

TestApi v0.4 Released!

ivom — Sat, 21 Nov 2009 21:14:00 GMT

I am happy to announce that we have just released version 0.4 of TestApi – the testing API library - on Codeplex.

Version 0.4 contains the following additions and modifications:

An improved Combinatorial Variation Generation API with added support for parameter value weights and tags (for “negative” variations);
A new Memory Leak Detection API allowing capture, comparison and serialization / de-serialization of memory snapshots of a running process;
A new Object Comparison API allowing comparison of arbitrary .NET objects, using arbitrary object comparison strategies;
A new Text String Generation API, allowing generation of random strings, for testing purposes;
Various documentation and sample improvements.

We will be blogging about some of these API in the near future. As always, TestApi is available on http://codeplex.com/testapi.

TestApi Slide-Deck from the Patterns & Practices Summit

ivom — Thu, 15 Oct 2009 05:23:00 GMT

This week is Microsoft’s Patterns & Practices summit in Redmond . Earlier today I gave a 15-minute presentation on TestApi . Here is the slide-deck: Test Api View more presentations from ivom1234 ....(read more)

Introduction to TestApi – Part 4: Combinatorial Variation Generation APIs

ivom — Thu, 27 Aug 2009 03:31:00 GMT

The state and behavior of every software system depends on a number of parameters. These parameters can be both inputs as well as environmental factors.

One can think of software testing as a controlled experiment, where the system under test is observed upon varying of the parameters that affect it, in an attempt to discover unexpected system behavior.

Depending on who you talk to, the act of generation of these variations is called variation generation, matrix expansion, design of experiments (DOE), etc.

+++

The Problem

Modern-day software systems are complex and depend on many parameters. Expanding all possible combinations of all parameter values often results in a phenomenon called matrix explosion – having an overwhelmingly high number of test variations.

Matrix explosion is undesirable because:

It increases the runtime of your tests (which, among other things, makes TDD impractical and TDD is the single most effective way to ensure high product quality);
It increases the support costs of your tests – for a test pool of 100,000 tests, even with 99% pass rate (which in reality is hard to achieve), you still have to investigate 1,000 test failures. Investigating 1,000 test failures is always more expensive than investigating say 10 failures;
It is often impractical or impossible to cover all combinations, so you end up with partial test coverage (very often “vanilla” test coverage).

There are many ways to deal with matrix explosion, which depend on various system and/or experimental constraints. The theory and practice of DOE deals with that.

TestApi’s Solution

TestApi provides a generic API for combinatorial variation generation, using the algorithm presented in Jacek Czerwonka’s “Pairwise Testing in Real World” article. The API uses the following nomenclature:

Parameter – represents a single factor / variable and its values;
Constraint – represents a relationship between parameters, their values, constants, and other constraints;
Model – contains all parameters and constraints for the system, for which we are generating variations;
Variation – represents a tuple with a single value for every Parameter in the Model.

All of these are represented as correspondingly named types. Following are several examples demonstrating the use of the API.

Example 1 : Simple Matrix

For the purposes of a simple artificial example, consider having a system with the following parameters and values:

Parameter	Values
Color	White, Green, Red
Height	Short, Tall
Size	Small, Medium, Large

Assuming, there are no constraints, here is the code you would use to create a model

using System;
using System.Collections.Generic;
using Microsoft.Test.VariationGeneration;

class Example1
{
    static void Main(string[] args)
    {
        //
        // Declare all parameters and construct a model
        //
        Parameter p1 = new Parameter("Color") { "White", "Green", "Red" };
        Parameter p2 = new Parameter("Height") { "Short", "Tall" };
        Parameter p3 = new Parameter("Size") { "Small", "Medium", "Large" };

        Model m = new Model(new List<Parameter>
 { p1, p2, p3 } );

        //
        // Generate and print out all possible variations of the parameters in the model
        //
        foreach (Variation v in m.GenerateVariations(3, 1234))
        {
            System.Console.WriteLine("{0}\t{1}\t{2}", v[p1.Name], v[p2.Name], v[p3.Name]);
        }
    }
}

Executing the code below produces the following 3*2*3=18 variations, representing all possible combinations of the values of the three parameters:

> Example1.exe
White   Short   Small
White   Short   Medium
White   Short   Large
White   Tall    Small
White   Tall    Medium
White   Tall    Large
Green   Short   Small
Green   Short   Medium
Green   Short   Large
Green   Tall    Small
Green   Tall    Medium
Green   Tall    Large
Red     Short   Small
Red     Short   Medium
Red     Short   Large
Red     Tall    Small
Red     Tall    Medium
Red     Tall    Large

The variations are generated by the call to GenerateVariations(3, 1234). The number "1234" is the seed for the random generator, utilized by the algorithm. The number "3" is the order of the generated combinations. The order of generated combinations must be a number between 1 and the number of parameters in the model. The output for orders "1" and "2" are presented below:

Output for order "1":

White   Short   Small
Green   Tall    Medium
Red     Short   Large

Output for order "2":

White   Short   Small
White   Tall    Medium
White   Short   Large
Green   Tall    Small
Green   Short   Medium
Green   Tall    Large
Red     Short   Small
Red     Tall    Medium
Red     Tall    Large

Example 2 : Vacation Planner

This example (created by Nathan Anderson – our engineer who designed and implemented the combinatorial variation generation API) demonstrates the use of parameter constraints.

Parameter destination = new Parameter("Destination") { "Whistler", "Hawaii", "Las Vegas" };
Parameter hotelQuality = new Parameter("Hotel Quality") { 5, 4, 3, 2, 1 };
Parameter activity = new Parameter("Activity") { "gambling", "swimming", "shopping", "skiing" };

List<Parameter>
 parameters = new List<Parameter>
 { destination, hotelQuality, activity };
List<Constraint> constraints = new List<Constraint>
{
    new IfThenConstraint
    {
        If = destination.Equal("Whistler").Or(destination.Equal("Hawaii")),
        Then = activity.NotEqual("gambling")                        
    },
    new IfThenConstraint
    {
        If = destination.Equal("Las Vegas").Or(destination.Equal("Hawaii")),
        Then = activity.NotEqual("skiing")
    },
    new IfThenConstraint
    {
        If = destination.Equal("Whistler"),
        Then = activity.NotEqual("swimming")
    },
};

Model model = new Model(parameters, constraints);

//
// Call the method under test with each generated variation
//
foreach (var variation in model.GenerateVariations())
{
    CallVacationPlanner(
        (string)variation[destination.Name], 
        (int)variation[hotelQuality.Name], 
        (string)variation[activity.Name]);
}

Example 3 : The WPF Platform Matrix

This last example demonstrates how we deal with a real-world problem we face in the WPF team...

WPF must work reliably on all OS configurations, defined by the following matrix:

Parameter	Number of Values	Values
OS	6	XP SP2, Vista SP1, 7, Server 2003 SP2, Server 2008 SP1, Server 2008 R2
Language	25	(using 3-letter language abbreviations) ARA, CHS, CHT, CSY, DAN, DEU, ELL, ENG, ESN, FIN, FRA, HEB, HUN, ITA, JPN, KOR, NLD, NOR, PLK, PSE, PTB, PTG, RUS, SVE, TRK
System Locale	2	Same as OS language, TRK
Flavor	2	Free, Checked
Platform	3	x86, x64, x64 wow
IE version	3	OS default, IE7, IE8
High DPI	2	120 DPI, 96 DPI
Theme	6	Native, Classic, Luna, Royale, Classic High Contrast, Aero
Side-by-side	9	3.5 SP1 + 4 (3.5 tests), 3.5 SP1 + 4 (4 tests), 3.5 SP1 + 4 - 4 (3.5 tests), 4 + Mock 4.5 (4 tests), 4 + Mock 5 (4 tests), 4 + 3.5 SP1 (4 tests), 4 + 3.5 SP1 (3.5 tests), 4 + 3.5 SP1 - 4 (3.5 tests), 4 (4 tests)

Most of the parameters are self-descriptive. “Side-by-side” captures the .NET installation state. For example “3.5 SP1 + 4 - 4 (3.5 tests)” means “install .NET 3.5 SP1, install .NET 4, uninstall .NET 4, run tests built against 3.5 SP1”. This is done to confirm that there are no unexpected side effects as a result of the installation and un-installation of .NET 4.

The trivial full expansion of the matrix results in 583,200 combinations (=6*25*2*2...). Of course, some of these combinations (e.g. XP SP2 OS with a Aero theme) are not valid, but even after removing the invalid combinations, we still end up with a prohibitively large number of platform configurations to test on.

There are several ways to deal with this problem. One is identifying the so called equivalence classes. For example, from the point of view of Side-by-Side, Vista SP1 and Server 2008 SP1 can be regarded as equivalent OS-es and so on. Another popular approach is reducing regular testing to “vanilla configurations” (e.g. mostly ENG (English), 96-DPI configurations), venturing outside of the “vanilla domain” in accordance with a predefined schedule (e.g. during test passes at the end of major milestones). A third approach is using a pair-wise combinatorial variation generator, reducing the number of platform variations to about 230 – still a fairly high number for any real-world test pool, but clearly much better than the original number above.

In the WPF team, we use the third approach, combined with an adaptive random algorithm, which prioritizes testing on platform configurations that have not been tested on recently. The simplified code below demonstrates how to construct a model for platform config variation generation.

using System;
using System.Collections.Generic; 
using Microsoft.Test.CommandLineParsing; 
using Microsoft.Test.VariationGeneration;
using Microsoft.Test.VariationGeneration.Constraints;

public class OsVariationGeneration
{
    public static void Main(string[] args)
    {            
        CommandLineDictionary d = CommandLineDictionary.FromArguments(args);
        int order = Int32.Parse(d["order"]);
        int seed = Int32.Parse(d["seed"]);

        //
        //  Parameters
        //
        Parameter os = new Parameter("OS") { "Windows XP SP3", "Windows Vista SP1", "Windows 7", "Windows Server 2003 SP2", "Windows Server 2008 R2" };
        Parameter language = new Parameter("language") { "ARA", "CHS", "CHT", "CSY", "DAN", "DEU", "ELL", "ENG", "ESN", "FIN", "FRA", "HEB", "HUN", "ITA", "JPN", "KOR", "NLD", "NOR", "PLK", "PSE", "PTB", "PTG", "RUS", "SVE", "TRK" };
        Parameter sysLocale = new Parameter("sysLocale") { "SameAsOsLanguage", "TRK" };
        Parameter flavor  = new Parameter("flavor")     { "fre", "chk" };
        Parameter platform = new Parameter("platform")  { "x86", "x64", "x64wow" };
        Parameter ieVersion = new Parameter("ieVersion") { "osDefault", "ie7", "ie8" };
        Parameter highDpi = new Parameter("hiDpi") { "yes", "no" };
        Parameter theme = new Parameter("theme") { "Classic", "Luna", "Royale", "Classic High Contrast", "Aero Basic", "Aero Glass" };
        Parameter sxs = new Parameter("sxs") 
        {
            "3.5 SP1 + 4 (3.5 tests)",
            "3.5 SP1 + 4 (4 tests)",
            "3.5 SP1 + 4 - 4 (3.5 tests)",
            "4 + Mock 4.5 (4 tests)",
            "4 + Mock 5 (4 tests)",
            "4 + 3.5 SP1 (4 tests)",
            "4 + 3.5 SP1 (3.5 tests)",
            "4 + 3.5 SP1 - 4 (3.5 tests)",
            "4 (4 tests)",
        };
        List<Parameter>
 parameters = new List<Parameter>
 { os, language, sysLocale, flavor, platform, ieVersion, highDpi, theme, sxs };


        //
        //  Constraints
        //
        List<Constraint> constraints = new List<Constraint>
        {
            new IfThenConstraint
            {
                If = os.Equal("Windows XP SP3").Or(os.Equal("Windows Server 2003 SP2")),
                Then = theme.NotEqual("Aero Basic")
            },
            new IfThenConstraint
            {
                If = os.Equal("Windows XP SP3").Or(os.Equal("Windows Server 2003 SP2")),
                Then = theme.NotEqual("Aero Glass")
            },
            new IfThenConstraint
            {
                If = os.Equal("Windows 7").Or(os.Equal("Windows Server 2008 R2")),
                Then = theme.NotEqual("Luna")
            }
        };


        //
        //  Model
        //
        Model m = new Model(parameters, constraints);

        uint i = 0;
        foreach (Variation v in m.GenerateVariations(order, seed))
        {
            Console.WriteLine(
                "{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\t{7}\t{8}\t{9}", 
                i,
                v[os.Name],
                v[language.Name],
                v[sysLocale.Name],
                v[flavor.Name],
                v[platform.Name],
                v[ieVersion.Name],
                v[highDpi.Name],
                v[theme.Name],
                v[sxs.Name]
                );
            i++;
        }
    }
}

Conclusion

Pairwise variation generation is an important tool in your toolbox as a test author. TestApi provides a simple facility for combinatorial variation generation. We will of course be evolving this facility, but do let us know if you have specific scenarios or requirements you'd like to see supported.

TestApi v0.3 Released!

ivom — Fri, 24 Jul 2009 04:52:00 GMT

I am happy to announce that we have just released the third preliminary version of TestApi -- the testing API library. This version introduces some fairly significant changes and new features:

New name spaces (under Microsoft.Test) and new binary names;
New source code organization;
A fairly comprehensive acceptance tests suite (using xUnit);
Fault Injection APIs for managed code (using CLR profiler technology);
Combinatorial Variation Generation API (using the PICT algorithm);
A redesigned Application Control API;
Updates to the Visual Verification API;
Updated samples and documentation; new conceptual documents;

We will be blogging in detail about some of the features in v0.3 in the coming weeks.

TestApi is a policy-free API library, delivering Microsoft testing technology in an easy to use package. It can be used from within any tool (VSTS, NUnit, xUnit, etc.) or environment to automate the testing for desktop and in-browse applications on Windows.

TestApi v0.3 is available on http://testapi.codeplex.com. Thanks for all your feedback. Keep it coming! :)

TestApi v.0.2 Released!

ivom — Fri, 24 Apr 2009 15:53:52 GMT

I am excited to announce that we have just released the second preliminary version of TestApi – the testing API library – on http://codeplex.com/testapi!

The v.0.2 package includes the following additions and modifications:

Improved command-line parsing APIs
Improved visual verification APIs
- A new tolerance map visual verifier in SnapshotToleranceMapVerifier
- New operations on snapshot (And and Or) allowing you to mask
- A new Snapshot.FromWindow(...) constructor with ability to include and exclude the window chrome in snapshots.
- Improved visual verification internals
New application control APIs, enabling in-proc and out-of-proc automation of client applications
Expanded conceptual documentation
CHM API documents (in addition to the HTML documents)
Addition of NUnit and xUnit usage samples (for all of the non-VS crowd out there)

Check out the package and let us know of any feedback.

Introduction to TestApi – Part 3: Visual Verification APIs

ivom — Tue, 21 Apr 2009 01:29:00 GMT

Visual Verification (VV) is the act of verifying that your application or component is displayed correctly on screen. The TestApi library provides a set of VV APIs. This post discusses these APIs.

Avoid It If You Can

First and foremost, I want to emphasize that visual verification is a test technique that should be used with caution. It is difficult to do correctly and any extensive use typically results in hard-to-maintain test codebases. Here are a few things to consider before you embark on VV test development:

The UI of an application tends to change a lot during development. If you use visual verification extensively, you may end up in a situation where you have to update a large number of visual verification tests on a daily basis, which is a waste of effort.
There are small differences in rendering between different video cards on different versions of the OS and of .NET. These differences are particularly pronounced in font rendering.
Before employing any form of UI verification, one should always review the underlying application architecture. Extensive need for UI testing is typically indicative of poorly architected systems, lacking proper view-model separation, so it’s almost always better to invest in proper system architecture than in extensive UI testing.
Whenever possible, attempt to do analytical visual verification i.e. one that does not employ the use of master images.

The WPF test team has gone through several iterations of cleaning up and retiring unnecessary visual verification tests in an attempt to speed up and stabilize our test suite.

General Concepts

The core VV terminology is:

Snapshot: A pixel buffer used for representing and evaluating screen image data.
Verifier: An oracle object which determines whether a snapshot passes against specified inputs.
Actual: The snapshot being evaluated.
Master: The reference data (image) which is used to evaluate the actual snapshot.
Tolerance: The accepted bounding range based on which the actual snapshot will be accepted as valid.

The general VV workflow is:

Capture some screen content.
Generate an expected snapshot (e.g. load a master image from disk, etc.)
Compare the actual snapshot to the expected snapshot and generate the difference (diff) snapshot.
Verify the diff using a verifier.
Report test result.

TestApi Visual Verification Technology

TestApi provides the following VV technology:

Snapshot: this class represents image pixels in a two-dimensional array for use in VV. Every element in the array represents a pixel in a given [row, column] of the image. A Snapshot object can be instantiated from a file (Snapshot.FromFile), or captured from screen (Snapshot.FromWindow and Snapshot.FromRectangle). Snapshot also exposes image cropping (Snapshot.Crop), resizing, diff-ing (Snapshot.CompareTo) masking (Snapshot.And) and merging operations (Snapshot.Or) operations.
Verifiers: the library provides a set of verifiers that can be used to verify a (diff) snapshot. SnapshotColorVerifier reports passing if
Various utilities: example of these are the Histogram class, providing basic functionality for handling image histograms

Examples

SnapshotColorVerifier

With these prolegomena out of the way, let’s look at some code. The first example below demonstrates master visual verification using a basic color verifier, which ensures that the difference between the master snapshot and the actual snapshot is within a defined tolerance:

// 1. Capture the actual pixels from a given window
Snapshot actual = Snapshot.FromRectangle(new Rectangle(0, 0, 100, 100));

// 2. Load the reference/master data from a previously saved file
Snapshot expected = Snapshot.FromFile("Expected.png"));

// 3. Compare the actual image with the master image
//    This operation creates a difference image. Any regions which are identical in 
//    the actual and master images appear as black. Areas with significant 
//    differences are shown in other colors.
Snapshot difference = actual.CompareTo(expected);

// 4. Configure the snapshot verifier - It expects a black image with zero tolerances
SnapshotVerifier v = new SnapshotColorVerifier(Color.Black, new ColorDifference());

// 5. Evaluate the difference image
if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the diff file for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

This approach works fine if you are evaluating the correctness of an application logo or some other application art. You may find that you will need to increase the tolerance a bit to accommodate differences in GPU rendering, but in general the SnapshotColorVerifier provides all the functionality you need.

SnapshotHistogramVerifier

A somewhat more sophisticated approach involves using of image histograms (see this link for a good introduction to the subject). An image histogram is a histogram that represents the frequency of pixels with a certain brightness. One can define a histogram that represents his/her expectation of the “proximity of the match” between the actual and expected snapshots.

For example, one can define a histogram semantically equivalent to the following statement:

“When I compare the actual snapshot to the expected snapshot (both of 320 pixels), I expect no more than 30 pixels with color channel difference of 1, no more than 10 pixels with color channel difference of 2, and zero pixels with higher differences.”

This histogram would look as follows:

Figure 1 Image Histogram

Such form of verification is done by using the SnapshotHistogramVerifier and the Histogram classes, as demonstrated in the sample below.

// Take a snapshot, compare to the master image and generate a diff
Snapshot actual = Snapshot.FromRectangle(new Rectangle(0, 0, 100, 100));
Snapshot expected = Snapshot.FromFile("Expected.png"));
Snapshot difference = actual.CompareTo(expected);

// Load the quality histogram from disk and use it to verify the diff
SnapshotVerifier v = new SnapshotHistogramVerifier(Histogram.FromFile("ToleranceHistogram.xml"));

if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the actual and diff images for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

The histogram file is just a XML file with the following schema:

<histogram>
  <tolerance>
    <point x="0" y="0.87500" />
    <point x="1" y="0.09375" />
    <point x="2" y="0.03125" />
    <point x="3" y="0" />
    <point x="4" y="0" />
    ...
    <point x="254" y="0" />
    <point x="255" y="0" />
  </tolerance>
</histogram>

This visual verification approach was pioneered in the WPF test organization about 6 years ago by Marc Cauchy and Pierre-Jean Reissman.

SnapshotToleranceMapVerifier

However, none of the two approaches above work particularly well for evaluation of a typical application window, containing controls, text, etc. Such windows tend to have regions that need different tolerance settings. For example, consider the application window below:

Figure 2 Sample Application Window

If you try to perform master based visual verification, you will hit 2 issues:

The non-client area of the window (the window frame) will tend to be slightly different between different runs of the application. It will also depend on environment factors such as the desktop wall-paper.
Some regions of the client area of the window will also tend to exhibit significant variance between runs of the application.

Issue (1) is easy to resolve, using Snapshot.FromWindow(...) and excluding the non-client area from the capture. Issue (2), however, is a bit more involved. Here are the expected, actual and diff snapshots of the client-area of the application.

Figure 3a Expected Client-Area Snapshot

Figure 3b Actual Client-Area Snapshot

Figure 3c Difference Snapshot

Figure 3d Difference Snapshot – Completely Black Pixels Are Replaced With Pink

The differences between the expected snapshot and the actual snapshot are difficult to see on Figure 3c, so on Figure 3d I have replaced purely black pixels with pink.

It is not surprising that most of the variation occurs around the text regions in the application window (ClearType renders differently on different machines). So it may make sense to increase the tolerance (or completely mask away) those regions, providing of course we are not specifically interested in their rendering.

In order to achieve that, we use the SnapshotToleranceMapVerifier class. Here’s an example:

// Take a snapshot, compare to the master image and generate a diff
Snapshot actual = Snapshot.FromWindow(hwndOfYourWindow, );
Snapshot expected = Snapshot.FromFile("Expected.png"));
Snapshot difference = actual.CompareTo(expected);

// Load the tolerance map. Then use it to verify the difference snapshot
Snapshot toleranceMap = Snapshot.FromFile("ExpectedImageToleranceMap.png");
SnapshotVerifier v = new SnapshotToleranceMapVerifier(toleranceMap);

if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the actual and diff images for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

The tolerance map that we use looks as follows:

Figure 4 Tolerance Map in “ExpectedImageToleranceMap.png”

What appears pure black (0x00FFFFFF) is actually an off-black color (0x000A0A0A) to handle the small variations that appear as black dots on Figure 3d. Then we also have 4 regions with significantly higher tolerance to handle the variability of the text rendering.

In Conclusion

The visual verification API in TestApi provides a solid foundation for visual verification tests. As a general best practice, however, avoid visual verification as much as possible.

Introduction to TestApi – Part 2: Command-Line Parsing APIs

ivom — Thu, 18 Dec 2008 01:15:00 GMT

Command-line parsers remind me of linked lists in C++: everybody has written several at various points in their careers. While everybody should write each of those at least once, I doubt that many people out there are particularly excited about writing and re-writing fundamental data structures on a regular basis – it gets old very quickly. Not to mention that doing so is error-prone and decreases the maintainability of a world that’s already hard to maintain.

That’s why modern-day frameworks such as .NET provide standard implementations of the common data structures. And that’s why TestApi provides a reusable command-line parsing APIs via the CommandLineDictionary and CommandLineParser classes, the latter being a type-safe layer on top of the former. Obviously, these are not test APIs per se – they are general utility APIs that happen to be more often used when writing tests.

A few quick examples follow.

Simple Command-Line Parsing

As seen from the first example below, extracting command-line parameters that are primitives is easy. Primitive command-line parameters are either boolean (e.g. the “verbose” flag below), or a key-value pair, that one can extract with the indexer of the CommandLineDictionary instance (see the “testId” key below), just as one would expect from a Dictionary.

// 
// EXAMPLE #1: Parsing a command-line such as "RunTests.exe /verbose /testId=123"
// 

using System;
using Microsoft.Test;

public class Program 
{ 
    public static void Main(string[] args) 
    { 
        CommandLineDictionary d = new CommandLineDictionary(args);

        bool verbose = d.ContainsKey("verbose");
        int testId = Int32.Parse(d["testId"]);
                
        // use the parsed command-line parameters
    }
}

By default flags/keys are indicated with the forward slash (“/”) character and values are indicated with the equals character (“=”), but the user can override that upon initialization of of the CommandLineDictionary object:

// 
// EXAMPLE #1b: Parsing a command-line such as "RunTests.exe –verbose –testId:123"
// 

...
        CommandLineDictionary d = new CommandLineDictionary(args, '-', ':');
...

Finally, you one can use the ToString method to get a string representation of the command-line arguments.

Command-Line Argument Structures

Another common pattern when dealing with command-line arguments is populating a structure which contains all parsed arguments. The CommandLineParser class makes this easy:

// EXAMPLE #2:
// Sample for parsing the following command-line:
// Test.exe /verbose /runId=10
// This sample declares a class in which the strongly typed arguments are populated
public class CommandLineArguments
{
   bool? Verbose { get; set; }
   int? RunId { get; set; }
}

CommandLineArguments a = new CommandLineArguments();
CommandLineParser.ParseArguments(a, args);

Type-Safe Commands

A third common approach is forming strongly-typed commands from the command-line parameters. This is common for cases when the command-line looks as follows:

some-exe COMMAND parameters-to-the-command

The parsing in this case is a little bit more involved:

Create one class for every supported command, which derives from the Command abstract base class and implements an expected Execute method.
Pass an expected command along with the command-line arguments to CommandLineParser.ParseCommand – the method will return a strongly-typed Command instance that can be Execute()-d.

// EXAMPLE #3:
// Sample for parsing the following command-line:
// Test.exe run /runId=10 /verbose 
// In this particular case we have an actual command on the command-line (“run”), 
// which we want to effectively de-serialize and execute.
public class RunCommand : Command
{
   bool? Verbose { get; set; }
   int? RunId { get; set; }

   public override void Execute()
   {
      // Implement your "run" execution logic here.
   }
}

Command c = new RunCommand();
CommandLineParser.ParseArguments(c, args); 
c.Execute();

Besides the parsing logic, CommandLineParser provides a few additional helper methods. One of them is CommandLineParser.PrintCommandUsage, which prints the usage for specific commands (or all supported commands) to the console.

In Conclusion

The command-line parsing APIs released with TestApi provide a simple and “layered” access to the command-line. Strictly speaking these APIs are not test APIs, but have nevertheless been included in TestApi as tests often have a need of parsing parameters on the command-line.

Introduction to TestApi – Part 1: Input Injection APIs

ivom — Tue, 16 Dec 2008 06:49:00 GMT

I am starting a series of posts introducing some of the facilities available in TestApi 0.1, a test and utility API library, which we recently released on CodePlex. Most of this content is already available in the documentation provided with the library.

The first post is on input injection – a fairly common activity in UI testing.

General Notes

Input injection is the act of simulating user input. In general, there are several ways to simulate user input, in the following progressively increasing levels of realism:

Direct method invocation: A test programmatically triggers events by directly calling methods on the target UI element. For example, a test can call the Button.IsPressed method to simulate pressing a WPF button.
Invocation using an accessibility interface (UIA, MSAA, etc.): A test programmatically triggers events by calling methods on an AutomationElement instance that represents the target UI element.
Simulation using low-level input: A test simulates input by using low-level input facilities provided by the host operating system. Examples of such facilities on Windows are the SendInput Win32 API and the Raw Input Win32 API, which inject input directly into the OS input stream.
Simulation using a device driver: A test uses a device driver to simulate input at the device-driver level.
Simulation using a robot: A test controls a robot to simulate direct human interaction with an input device (for example, pressing keys on a keyboard).

Technique A is framework-specific; what works for WPF does not work for Windows Forms and vice versa. Technique B is less framework-specific than A, but still has limitations, because some frameworks differ in their implementations of the required accessibility interfaces. Techniques C and D are OS-specific. Technique D is significantly more difficult to implement and deploy than C, without a corresponding increase in its level of realism. Technique E is universal, albeit much slower and much more expensive than the other options.

The TestApi library provides facilities both for B (through the AutomationUtilities class) and for C (through the Mouse and Keyboard classes), which are the most generally useful techniques of input simulation.

Examples

The AutomationUtilities class provides wrappers for common UIA operations, such as discovery of UI elements. The first example below demonstrates how to discover and click a WPF Button in a WPF Window, by using the AutomationUtilities class and the Mouse class.

//
// EXAMPLE #1
// This code below discovers and clicks the Close button in an About dialog box, thus
// dismissing the About dialog box.
//

string aboutDialogName = “About”;
string closeButtonName = “Close”;

AutomationElementCollection aboutDialogs = AutomationUtilities.FindElementsByName(
    AutomationElement.RootElement,
    aboutDialogName);

AutomationElementCollection closeButtons = AutomationUtilities.FindElementsByName(
    aboutDialogs[0],
    closeButtonName);

//
// You can either invoke the discovered control, through its invoke pattern...
//

InvokePattern p = 
    closeButtons[0].GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
p.Invoke();

//
// ... or you can handle the Mouse directly and click on the control.
//

Mouse.MoveTo(closeButton.GetClickablePoint());
Mouse.Click();

The second example below demonstrates how to discover a TextBox instance and type in it, using the Mouse and Keyboard classes as wrappers of common mouse and keyboard operations:

//
// EXAMPLE #2
// Discover the location of a TextBox with a given name.
//

string textboxName = “ssnInputField”;

AutomationElement textBox = AutomationUtilities.FindElementsByName(
    AutomationElement.RootElement,
    textboxName)[0];

Point textboxLocation = textbox.GetClickablePoint();

//
// Move the mouse to the textbox, click, then type something
//

Mouse.MoveTo(textboxLocation);
Mouse.Click();

Keyboard.Type(“Hello world. ”);
Keyboard.Press(Key.Shift);
Keyboard.Type(“hello, capitalized world.”);
Keyboard.Release(Key.Shift);

In Conclusion

The Mouse and Keyboard classes in the TestApi library can be used for automating of any application, running on Windows. The classes are completely policy- and context-free – their usage is not dependent on a specific test framework or on a specific test workflow. TestApi provides full source code and XML documentation for these classes, so you can either integrate them in your own projects or reference the pre-built DLLs.

Note that, even though Mouse, Keyboard and AutomationUtilities make the life of the test automation developer quite a bit easier, UI testing is tricky and should be avoided whenever possible. It’s always preferable to design your application as a multi-tier application, with a “thin” UI layer, so that you can bypass the UI in most of your tests.

TestApi v.0.1 Released

ivom — Sat, 06 Dec 2008 21:30:00 GMT

On Friday, we released the first preliminary version of an experimental library with test APIs, named ... TestApi :). With this library, we are sharing internal Microsoft testing technology with the developer and tester community.

We wanted to keep the first release small and contained, so that it can be easily grasped and critiqued, but complete enough to provide actual user value and to clearly demonstrate the direction of this effort.

In addition to the binaries and the source code, we have provided full documentation (both MSDN-style API documents and conceptual documents) as well as a couple of VSTS samples to get you started.

Check out the release on http://codeplex.com/TestApi and let us know what you think.