Ivo Manolov's Blog

TestApi Slide-Deck from the Patterns & Practices Summit

This week is Microsoft’s Patterns & Practices summit in Redmond. Earlier today I gave a 15-minute presentation on TestApi. Here is the slide-deck:

Posted Wednesday, October 14, 2009 10:23 PM by ivom | 0 Comments

The Gradual Development Paradigm

Software project management is tricky. The industry proves this over and over again: delivering less than expected, slipping features, projects, milestones, products…

The gradual development idea presented below is not something new – it's more or less common-sense. Often, however, not all positive implications of it are fully understood or internalized, so I thought I’d share this article, which I sent to my team several years ago.

Admittedly, some of this is old news. After all we live in the age of Agile, rapid iteration and integration, test-driven design and development. It still surprises me how many teams tend to slip in old behaviors on a regular basis, so I have personally found these practices worth reviewing and repeating.

Case Study

Imagine that you have 5 developers and that you are 1 day before your deadline. If they have not checked in anything, you are scared (understandably): they may deliver anything between 500% (100% each) and 0%. That's a lot of stress for a team lead.

Now imagine that you have 5 developers, who have delivered gradually in smaller pieces. Understandably, they have delivered less (because gradual development is more expensive), but at day-1, you know that you are at say 450% so your uncertainty is only 50% (rather than 500%) or even less (since as a result of the gradual development you have lost time, so you know you can only accomplish 20% more summing up to say 470%).

Which situation would you choose to be in: 500% of risk with a potential heart attack or 20% of risk with a known under-delivery and 8-hour work days at the end of the project?

The Theory of Gradual Development

The diagram below presents the principle of gradual development. Person A has adopted the "I will code it on my computer and worry about code and design review, deployment, logistics at the end when I am done" approach. Person B has adopted the "gradual development" approach.

But wait a minute, I hear you say, person B is actually slipping while person A may deliver on time after all.

What is missing from the diagram is that it is never a 1-person team. We usually talk about teams of 5, 10, 20 people. Let's assume that there is a 5% probability that Person A will not deliver on time i.e. there is a 95% probability that Person A will deliver as expected (that's a good healthy number – everyone who has shipped a product will tell you that is a pretty high confidence.

Then for a team of 20 people, the combined probability to actually deliver on time is 95%²⁰ = 36%. That's encouraging![1] Here are the expanded results:

Or, as a chart:

Note that for a team of 200 people, the probability to deliver on time is close to 0%!

How does gradual development help? Well, on "ship day - 1", you know that your team has already shipped 90% of the product, so the probability to ship on time is 90% + (10% * 95%²⁰) = 94%. Way better than 0%, isn't it?

The Practice of Gradual Development

Here are some practical directions:

1. Defy the urge to produce big chunks

All of us want to impress our bosses or peers by creating something big and delivering it as a whole. That's one sure way to produce the "wow effect". In reality however, you rarely (if ever) get to impress anybody that much. However, you can, truly, build a reputation of a rock-solid and predictable achiever. People who have this reputation are awarded and valued everywhere, because the mere ability to ship and sell is what defines a business.

2. Constantly ask yourself "how can I deploy this today?"

Gradual development is great in theory, but in practice it's just too easy to slip back to the regular "I am going to show them how fast and complete I am" or "I'll have enough time to deploy after I am done" routines.

Gradual development does not come naturally to any of us. It (much like OOP) is an acquired taste, so we have to constantly remind ourselves that we have to constantly ship. You are not ready with the whole implementation? Not a problem! Stub it (e.g. make the unimplemented functions to return E_NOTIMPL) and ship it.

“Deploying” or “Shipping” may mean different things to different people:

• To a developer, "shipping" means adding the feature to the build and to the installation of the product.
• To a tester, "shipping" means deploying a test in the lab (as "lab ready").
• To a PM, "shipping" means publishing a spec (although the spec may not be fully ready).

3. Maintain a rapid create-test-deploy cycle

People often forget that deployment is part of the job. In fact it is a very significant and time consuming part of the job. We all came into the software development business, because we wanted to program. It turns out, however, that software development is not only about programming. Strangely enough pure programming does not really occupy most of our time: deployment and maintenance does. Embrace this reality and make it a goal not to program, but to deploy.

4. If you are a manager, reward gradual development

You want predictability from your business? Then reward predictability (e.g. predictable achievers, rather than unpredictable Einsteins[2].)

5. If you are an individual contributor, keep in mind the perspective of your manager

Remember that while you can happily live with a confidence of 95%, your manager is a manager of 5, 10 20 people, so he / she has to live with a significantly reduced confidence.

+++

[1] For simplicity, it is assumed that the probability of successful delivery from a given team member is independent from other team member's deliveries i.e. for a team of two people, the combined probability P_combined = P(contributor 1 delivers on time) * P(contributor 2 delivers on time). If this assumption in not valid, the resulting combined probability is even lower!

[2] This of course is taken to extreme. You absolutely need the Einsteins. In fact, it is “manageable Einsteins” that often make the difference between a moderately successful and a hugely successful product.

Posted Wednesday, October 14, 2009 5:03 AM by ivom | 1 Comments

Introduction to TestApi – Part 4: Combinatorial Variation Generation APIs

The state and behavior of every software system depends on a number of parameters. These parameters can be both inputs as well as environmental factors.

One can think of software testing as a controlled experiment, where the system under test is observed upon varying of the parameters that affect it, in an attempt to discover unexpected system behavior.

Depending on who you talk to, the act of generation of these variations is called variation generation, matrix expansion, design of experiments (DOE), etc.

+++

The Problem

Modern-day software systems are complex and depend on many parameters. Expanding all possible combinations of all parameter values often results in a phenomenon called matrix explosion – having an overwhelmingly high number of test variations.

Matrix explosion is undesirable because:

1. It increases the runtime of your tests (which, among other things, makes TDD impractical and TDD is the single most effective way to ensure high product quality);
2. It increases the support costs of your tests – for a test pool of 100,000 tests, even with 99% pass rate (which in reality is hard to achieve), you still have to investigate 1,000 test failures. Investigating 1,000 test failures is always more expensive than investigating say 10 failures;
3. It is often impractical or impossible to cover all combinations, so you end up with partial test coverage (very often “vanilla” test coverage).

There are many ways to deal with matrix explosion, which depend on various system and/or experimental constraints. The theory and practice of DOE deals with that. 

TestApi’s Solution

TestApi provides a generic API for combinatorial variation generation, using the algorithm presented in Jacek Czerwonka’s “Pairwise Testing in Real World” article. The API uses the following nomenclature:

• Parameter – represents a single factor / variable and its values;
• Constraint – represents a relationship between parameters, their values, constants, and other constraints;
• Model – contains all parameters and constraints for the system, for which we are generating variations;
• Variation – represents a tuple with a single value for every Parameter in the Model.

All of these are represented as correspondingly named types. Following are several examples demonstrating the use of the API.

Example 1 : Simple Matrix

For the purposes of a simple artificial example, consider having a system with the following parameters and values:

Assuming, there are no constraints, here is the code you would use to create a model

using System;
using System.Collections.Generic;
using Microsoft.Test.VariationGeneration;

class Example1
{
    static void Main(string[] args)
    {
        //
        // Declare all parameters and construct a model
        //
        Parameter p1 = new Parameter("Color") { "White", "Green", "Red" };
        Parameter p2 = new Parameter("Height") { "Short", "Tall" };
        Parameter p3 = new Parameter("Size") { "Small", "Medium", "Large" };

        Model m = new Model(new List<Parameter>
 { p1, p2, p3 } );

        //
        // Generate and print out all possible variations of the parameters in the model
        //
        foreach (Variation v in m.GenerateVariations(3, 1234))
        {
            System.Console.WriteLine("{0}\t{1}\t{2}", v[p1.Name], v[p2.Name], v[p3.Name]);
        }
    }
}

Executing the code below produces the following 3*2*3=18 variations, representing all possible combinations of the values of the three parameters:

> Example1.exe
White   Short   Small
White   Short   Medium
White   Short   Large
White   Tall    Small
White   Tall    Medium
White   Tall    Large
Green   Short   Small
Green   Short   Medium
Green   Short   Large
Green   Tall    Small
Green   Tall    Medium
Green   Tall    Large
Red     Short   Small
Red     Short   Medium
Red     Short   Large
Red     Tall    Small
Red     Tall    Medium
Red     Tall    Large

The variations are generated by the call to GenerateVariations(3, 1234). The number "1234" is the seed for the random generator, utilized by the algorithm. The number "3" is the order of the generated combinations. The order of generated combinations must be a number between 1 and the number of parameters in the model. The output for orders "1" and "2" are presented below:

White   Short   Small
Green   Tall    Medium
Red     Short   Large

White   Short   Small
White   Tall    Medium
White   Short   Large
Green   Tall    Small
Green   Short   Medium
Green   Tall    Large
Red     Short   Small
Red     Tall    Medium
Red     Tall    Large

Example 2 : Vacation Planner

This example (created by Nathan Anderson – our engineer who designed and implemented the combinatorial variation generation API) demonstrates the use of parameter constraints.

Parameter destination = new Parameter("Destination") { "Whistler", "Hawaii", "Las Vegas" };
Parameter hotelQuality = new Parameter("Hotel Quality") { 5, 4, 3, 2, 1 };
Parameter activity = new Parameter("Activity") { "gambling", "swimming", "shopping", "skiing" };

List<Parameter>
 parameters = new List<Parameter>
 { destination, hotelQuality, activity };
List<Constraint> constraints = new List<Constraint>
{
    new IfThenConstraint
    {
        If = destination.Equal("Whistler").Or(destination.Equal("Hawaii")),
        Then = activity.NotEqual("gambling")                        
    },
    new IfThenConstraint
    {
        If = destination.Equal("Las Vegas").Or(destination.Equal("Hawaii")),
        Then = activity.NotEqual("skiing")
    },
    new IfThenConstraint
    {
        If = destination.Equal("Whistler"),
        Then = activity.NotEqual("swimming")
    },
};

Model model = new Model(parameters, constraints);

//
// Call the method under test with each generated variation
//
foreach (var variation in model.GenerateVariations())
{
    CallVacationPlanner(
        (string)variation[destination.Name], 
        (int)variation[hotelQuality.Name], 
        (string)variation[activity.Name]);
}

Example 3 : The WPF Platform Matrix

This last example demonstrates how we deal with a real-world problem we face in the WPF team...

WPF must work reliably on all OS configurations, defined by the following matrix:

Most of the parameters are self-descriptive. “Side-by-side” captures the .NET installation state. For example “3.5 SP1 + 4 - 4 (3.5 tests)” means “install .NET 3.5 SP1, install .NET 4, uninstall .NET 4, run tests built against 3.5 SP1”. This is done to confirm that there are no unexpected side effects as a result of the installation and un-installation of .NET 4. 

The trivial full expansion of the matrix results in 583,200 combinations (=6*25*2*2...). Of course, some of these combinations (e.g. XP SP2 OS with a Aero theme) are not valid, but even after removing the invalid combinations, we still end up with a prohibitively large number of platform configurations to test on.

There are several ways to deal with this problem. One is identifying the so called equivalence classes. For example, from the point of view of Side-by-Side, Vista SP1 and Server 2008 SP1 can be regarded as equivalent OS-es and so on. Another popular approach is reducing regular testing to “vanilla configurations” (e.g. mostly ENG (English), 96-DPI configurations), venturing outside of the “vanilla domain” in accordance with a predefined schedule (e.g. during test passes at the end of major milestones).  A third approach is using a pair-wise combinatorial variation generator, reducing the number of platform variations to about 230 – still a fairly high number for any real-world test pool, but clearly much better than the original number above.

In the WPF team, we use the third approach, combined with an adaptive random algorithm, which prioritizes testing on platform configurations that have not been tested on recently. The simplified code below demonstrates how to construct a model for platform config variation generation.

using System;
using System.Collections.Generic; 
using Microsoft.Test.CommandLineParsing; 
using Microsoft.Test.VariationGeneration;
using Microsoft.Test.VariationGeneration.Constraints;

public class OsVariationGeneration
{
    public static void Main(string[] args)
    {            
        CommandLineDictionary d = CommandLineDictionary.FromArguments(args);
        int order = Int32.Parse(d["order"]);
        int seed = Int32.Parse(d["seed"]);

        //
        //  Parameters
        //
        Parameter os = new Parameter("OS") { "Windows XP SP3", "Windows Vista SP1", "Windows 7", "Windows Server 2003 SP2", "Windows Server 2008 R2" };
        Parameter language = new Parameter("language") { "ARA", "CHS", "CHT", "CSY", "DAN", "DEU", "ELL", "ENG", "ESN", "FIN", "FRA", "HEB", "HUN", "ITA", "JPN", "KOR", "NLD", "NOR", "PLK", "PSE", "PTB", "PTG", "RUS", "SVE", "TRK" };
        Parameter sysLocale = new Parameter("sysLocale") { "SameAsOsLanguage", "TRK" };
        Parameter flavor  = new Parameter("flavor")     { "fre", "chk" };
        Parameter platform = new Parameter("platform")  { "x86", "x64", "x64wow" };
        Parameter ieVersion = new Parameter("ieVersion") { "osDefault", "ie7", "ie8" };
        Parameter highDpi = new Parameter("hiDpi") { "yes", "no" };
        Parameter theme = new Parameter("theme") { "Classic", "Luna", "Royale", "Classic High Contrast", "Aero Basic", "Aero Glass" };
        Parameter sxs = new Parameter("sxs") 
        {
            "3.5 SP1 + 4 (3.5 tests)",
            "3.5 SP1 + 4 (4 tests)",
            "3.5 SP1 + 4 - 4 (3.5 tests)",
            "4 + Mock 4.5 (4 tests)",
            "4 + Mock 5 (4 tests)",
            "4 + 3.5 SP1 (4 tests)",
            "4 + 3.5 SP1 (3.5 tests)",
            "4 + 3.5 SP1 - 4 (3.5 tests)",
            "4 (4 tests)",
        };
        List<Parameter>
 parameters = new List<Parameter>
 { os, language, sysLocale, flavor, platform, ieVersion, highDpi, theme, sxs };


        //
        //  Constraints
        //
        List<Constraint> constraints = new List<Constraint>
        {
            new IfThenConstraint
            {
                If = os.Equal("Windows XP SP3").Or(os.Equal("Windows Server 2003 SP2")),
                Then = theme.NotEqual("Aero Basic")
            },
            new IfThenConstraint
            {
                If = os.Equal("Windows XP SP3").Or(os.Equal("Windows Server 2003 SP2")),
                Then = theme.NotEqual("Aero Glass")
            },
            new IfThenConstraint
            {
                If = os.Equal("Windows 7").Or(os.Equal("Windows Server 2008 R2")),
                Then = theme.NotEqual("Luna")
            }
        };


        //
        //  Model
        //
        Model m = new Model(parameters, constraints);

        uint i = 0;
        foreach (Variation v in m.GenerateVariations(order, seed))
        {
            Console.WriteLine(
                "{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\t{7}\t{8}\t{9}", 
                i,
                v[os.Name],
                v[language.Name],
                v[sysLocale.Name],
                v[flavor.Name],
                v[platform.Name],
                v[ieVersion.Name],
                v[highDpi.Name],
                v[theme.Name],
                v[sxs.Name]
                );
            i++;
        }
    }
}

Conclusion

Pairwise variation generation is an important tool in your toolbox as a test author. TestApi provides a simple facility for combinatorial variation generation. We will of course be evolving this facility, but do let us know if you have specific scenarios or requirements you'd like to see supported.

Posted Wednesday, August 26, 2009 8:31 PM by ivom | 0 Comments

WPF Control Verifier v0.1 Released!

Our strategy to improve WPF and .NET application quality has 3 major pillars:

1. Provide guidance on how to create high-quality applications;
2. Provide reusable API libraries for automated application testing and monitoring;
3. Provide tools to help create and debug applications and components.

To address (1), we have released several versions of the WPF Application Quality Guide, the initial version of the WPF Model-View-ViewModel Toolkit (available for download here), the WPF Performance Guide documents on MSDN, and others. To address (2), we are investing in TestApi – the test API library. For (3), we have released the WPF Performance Suite on MSDN.

I am excited to announce that we have another addition to the third group : the WPF Control Verifier v0.1 (available for download from here).

The WPF Control Verifier is a tool that verifies the correctness of WPF controls. This first version of the tool includes a set of default style tests that you can run on your WPF custom controls. The verifier was designed, implemented and tested by Vincent Sibal and Alexis Roosa – engineers on our Controls team. The tools has already proven extremely valuable in our internal testing of the standard WPF controls uncovering several important bugs that have been fixed, so do check it out. And of course send us your feedback, feature requests, etc.

Posted Wednesday, August 19, 2009 9:59 AM by ivom | 0 Comments

TestApi v0.3 Released!

I am happy to announce that we have just released the third preliminary version of TestApi -- the testing API library. This version introduces some fairly significant changes and new features:

• New name spaces (under Microsoft.Test) and new binary names;
  
• New source code organization;
  
• A fairly comprehensive acceptance tests suite (using xUnit);
• Fault Injection APIs for managed code (using CLR profiler technology);
• Combinatorial Variation Generation API (using the PICT algorithm);
• A redesigned Application Control API;
• Updates to the Visual Verification API;
  
• Updated samples and documentation; new conceptual documents;
  

TestApi is a policy-free API library, delivering Microsoft testing technology in an easy to use package. It can be used from within any tool (VSTS, NUnit, xUnit, etc.) or environment to automate the testing for desktop and in-browse applications on Windows.

TestApi v0.3 is available on http://testapi.codeplex.com. Thanks for all your feedback. Keep it coming! :)



 

Posted Thursday, July 23, 2009 6:52 PM by ivom | 1 Comments

WPF MVVM Toolkit 0.1 Released!

Model-View separation is not a novel idea in the software industry. It has been around for at least 30 years. Recently, however, MV separation is seeing a lot of renewed interest, powered both by the growing complexity of software and by the need to provide different UI experiences, reusing the same underlying business logic.

WPF provides a lot of native functionality (commanding, databinding, etc.), enabling developers to build applications with proper model-view separation. So far, however, we have not provided simple guidance on how to utilize these features to build properly factored applications with model-view separation.

***

I am very happy to announce that on Friday we released the first preliminary version of the WPF Model-View-ViewModel Toolkit (MVVM is WPF’s equivalent of the classic MVC design pattern). The toolkit is available for download on the “WPF Futures”  codeplex site.

The toolkit consists of:

• A Visual Studio template, allowing you to quickly create WPF Model-View-ViewModel Applications
• Documentation
  • A general introduction to the MVVM design pattern
  • A detailed walk-through, demonstrating how to create a simple MVVM application, starting from the skeleton generated by the template
• A complete MVVM application example – a WPF messenger application.

Big kudos to Patrick Danino – an engineer on our team – who worked on the toolkit from inception to its successful first preliminary release.

***

This first preliminary version of the toolkit is fairly simple and clearly not aimed at MVVM experts. We have adopted an evolutionary approach of delivery, so we invite everybody out there – from folks looking to pick up MVVM for their next project to MVVM experts – to collaborate with us on the toolkit, by sharing both real-world experiences as well as first-time-exposure feedback.

We will use this feedback to drive improvements in the toolkit, in the WPF platform and in the relevant authoring tools (Visual Studio, Expression Blend, etc).

Posted Sunday, May 03, 2009 1:22 PM by ivom | 6 Comments

WPF Application Quality Guide v0.5 Released!

I am happy to announce that we have released CTP 5 of the “WPF Application Quality Guide” – our single-stop document for WPF application and component developers and testers.

CTP 5 comes with the following new and updated content:

• New content:
  • “Considerations for WPF Browser Applications”
  • “Integration and Scenario Testing”
  • “XAML Editing Tools and Visual Studio Add-ins”
  • “Building a WPF Application Test Suite by Using VSTS, NUnit, or xUnit”
    
• Updated content:
  • Reading roadmap
  • “Performance and Scalability Testing”
  • “TestApi”
  • “Debugging Tools”
  • “Performance Profiling Tools”
  • “WPF Application Design and Development Tools”

The Guide is available here and here as a DOC file. Big kudos to Anne Gao on our team, who has been PM-ing all releases of the guide to date.

***

Those of you who have been following the Guide have seen it evolve from its first CTP (link) of 20 or so pages to its present size of more than 80 pages.

Any 80-page document is tricky to grok, so we have been thinking of reorganizing the content and potentially splitting the content into two documents – one on client application development and testing fundamentals and one on WPF specifics. We prefer to do any edits based on user feedback, so please let us know of the current organization and content of the document and share your ideas on how we could improve both.

Posted Tuesday, April 28, 2009 8:58 PM by ivom | 5 Comments

TestApi v.0.2 Released!

I am excited to announce that we have just released the second preliminary version of TestApi – the testing API library – on http://codeplex.com/testapi!

The v.0.2 package includes the following additions and modifications:

• Improved command-line parsing APIs
• Improved visual verification APIs
  • A new tolerance map visual verifier in SnapshotToleranceMapVerifier
  • New operations on snapshot (And and Or) allowing you to mask
  • A new Snapshot.FromWindow(...) constructor with ability to include and exclude the window chrome in snapshots.
  • Improved visual verification internals
• New application control APIs, enabling in-proc and out-of-proc automation of client applications
• Expanded conceptual documentation
• CHM API documents (in addition to the HTML documents)
• Addition of NUnit and xUnit usage samples (for all of the non-VS crowd out there)

Check out the package and let us know of any feedback.

Posted Friday, April 24, 2009 5:53 AM by ivom | 9 Comments

Introduction to TestApi – Part 3: Visual Verification APIs

Visual Verification (VV) is the act of verifying that your application or component is displayed correctly on screen. The TestApi library provides a set of VV APIs. This post discusses these APIs.

Avoid It If You Can

First and foremost, I want to emphasize that visual verification is a test technique that should be used with caution. It is difficult to do correctly and any extensive use typically results in hard-to-maintain test codebases. Here are a few things to consider before you embark on VV test development:

1. The UI of an application tends to change a lot during development. If you use visual verification extensively, you may end up in a situation where you have to update a large number of visual verification tests on a daily basis, which is a waste of effort.
2. There are small differences in rendering between different video cards on different versions of the OS and of .NET. These differences are particularly pronounced in font rendering.
3. Before employing any form of UI verification, one should always review the underlying application architecture. Extensive need for UI testing is typically indicative of poorly architected systems, lacking proper view-model separation, so it’s almost always better to invest in proper system architecture than in extensive UI testing.
4. Whenever possible, attempt to do analytical visual verification i.e. one that does not employ the use of master images.

The WPF test team has gone through several iterations of cleaning up and retiring unnecessary visual verification tests in an attempt to speed up and stabilize our test suite.

General Concepts

The core VV terminology is:

• Snapshot: A pixel buffer used for representing and evaluating screen image data.
• Verifier: An oracle object which determines whether a snapshot passes against specified inputs.
• Actual: The snapshot being evaluated.
• Master: The reference data (image) which is used to evaluate the actual snapshot.
• Tolerance: The accepted bounding range based on which the actual snapshot will be accepted as valid.

The general VV workflow is:

1. Capture some screen content.
2. Generate an expected snapshot (e.g. load a master image from disk, etc.)
3. Compare the actual snapshot to the expected snapshot and generate the difference (diff) snapshot.
4. Verify the diff using a verifier.
5. Report test result.

TestApi Visual Verification Technology

TestApi provides the following VV technology:

• Snapshot: this class represents image pixels in a two-dimensional array for use in VV. Every element in the array represents a pixel in a given [row, column] of the image. A Snapshot object can be instantiated from a file (Snapshot.FromFile), or captured from screen (Snapshot.FromWindow and Snapshot.FromRectangle). Snapshot also exposes image cropping (Snapshot.Crop), resizing, diff-ing (Snapshot.CompareTo) masking (Snapshot.And) and merging operations (Snapshot.Or) operations.
  
• Verifiers: the library provides a set of verifiers that can be used to verify a (diff) snapshot. SnapshotColorVerifier reports passing if
  
• Various utilities: example of these are the Histogram class, providing basic functionality for handling image histograms

Examples

SnapshotColorVerifier

With these prolegomena out of the way, let’s look at some code. The first example below demonstrates master visual verification using a basic color verifier, which ensures that the difference between the master snapshot and the actual snapshot is within a defined tolerance:

// 1. Capture the actual pixels from a given window
Snapshot actual = Snapshot.FromRectangle(new Rectangle(0, 0, 100, 100));

// 2. Load the reference/master data from a previously saved file
Snapshot expected = Snapshot.FromFile("Expected.png"));

// 3. Compare the actual image with the master image
//    This operation creates a difference image. Any regions which are identical in 
//    the actual and master images appear as black. Areas with significant 
//    differences are shown in other colors.
Snapshot difference = actual.CompareTo(expected);

// 4. Configure the snapshot verifier - It expects a black image with zero tolerances
SnapshotVerifier v = new SnapshotColorVerifier(Color.Black, new ColorDifference());

// 5. Evaluate the difference image
if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the diff file for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

This approach works fine if you are evaluating  the correctness of an application logo or some other application art. You may find that you will need to increase the tolerance a bit to accommodate differences in GPU rendering, but in general the SnapshotColorVerifier provides all the functionality you need.

SnapshotHistogramVerifier

A somewhat more sophisticated approach involves using of image histograms (see this link for a good introduction to the subject). An image histogram is a histogram that represents the frequency of pixels with a certain brightness. One can define a histogram that represents his/her expectation of the “proximity of the match” between the actual and expected snapshots.

For example, one can define a histogram semantically equivalent to the following statement:

“When I compare the actual snapshot to the expected snapshot (both of 320 pixels), I expect no more than 30 pixels with color channel difference of 1, no more than 10 pixels with color channel difference of 2, and zero pixels with higher differences.”

This histogram would look as follows:

Figure 1  Image Histogram

Such form of verification is done by using the SnapshotHistogramVerifier and the Histogram classes, as demonstrated in the sample below.

// Take a snapshot, compare to the master image and generate a diff
Snapshot actual = Snapshot.FromRectangle(new Rectangle(0, 0, 100, 100));
Snapshot expected = Snapshot.FromFile("Expected.png"));
Snapshot difference = actual.CompareTo(expected);

// Load the quality histogram from disk and use it to verify the diff
SnapshotVerifier v = new SnapshotHistogramVerifier(Histogram.FromFile("ToleranceHistogram.xml"));

if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the actual and diff images for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

The histogram file is just a XML file with the following schema:

<histogram>
  <tolerance>
    <point x="0" y="0.87500" />
    <point x="1" y="0.09375" />
    <point x="2" y="0.03125" />
    <point x="3" y="0" />
    <point x="4" y="0" />
    ...
    <point x="254" y="0" />
    <point x="255" y="0" />
  </tolerance>
</histogram>

This visual verification approach was pioneered in the WPF test organization about 6 years ago by Marc Cauchy and Pierre-Jean Reissman.

SnapshotToleranceMapVerifier

However, none of the two approaches above work particularly well for evaluation of a typical application window, containing controls, text, etc. Such windows tend to have regions that need different tolerance settings. For example, consider the application window below:

Figure 2  Sample Application Window

If you try to perform master based visual verification, you will hit 2 issues:

1. The non-client area of the window (the window frame) will tend to be slightly different between different runs of the application. It will also depend on environment factors such as the desktop wall-paper.
   
2. Some regions of the client area of the window will also tend to exhibit significant variance between runs of the application.

Issue (1) is easy to resolve, using Snapshot.FromWindow(...) and excluding the non-client area from the capture. Issue (2), however, is a bit more involved. Here are the expected, actual and diff snapshots of the client-area of the application.

 Figure 3a  Expected Client-Area Snapshot

Figure 3b  Actual Client-Area Snapshot

  Figure 3c  Difference Snapshot

Figure 3d  Difference Snapshot – Completely Black Pixels Are Replaced With Pink

The differences between the expected snapshot and the actual snapshot are difficult to see on Figure 3c, so on Figure 3d I have replaced purely black pixels with pink.

It is not surprising that most of the variation occurs around the text regions in the application window (ClearType renders differently on different machines).  So it may make sense to increase the tolerance (or completely mask away) those regions, providing of course we are not specifically interested in their rendering.

In order to achieve that, we use the SnapshotToleranceMapVerifier class. Here’s an example:

// Take a snapshot, compare to the master image and generate a diff
Snapshot actual = Snapshot.FromWindow(hwndOfYourWindow, );
Snapshot expected = Snapshot.FromFile("Expected.png"));
Snapshot difference = actual.CompareTo(expected);

// Load the tolerance map. Then use it to verify the difference snapshot
Snapshot toleranceMap = Snapshot.FromFile("ExpectedImageToleranceMap.png");
SnapshotVerifier v = new SnapshotToleranceMapVerifier(toleranceMap);

if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the actual and diff images for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

The tolerance map that we use looks as follows:

Figure 4  Tolerance Map in “ExpectedImageToleranceMap.png”

What appears pure black (0x00FFFFFF) is actually an off-black color (0x000A0A0A) to handle the small variations that appear as black dots on Figure 3d. Then we also have 4 regions with significantly higher tolerance to handle the variability of the text rendering.

In Conclusion

The visual verification API in TestApi provides a solid foundation for visual verification tests. As a general best practice, however, avoid visual verification as much as possible.

Posted Monday, April 20, 2009 3:29 PM by ivom | 14 Comments

PhotoSuru – A Great WPF Sample Application

I am pleased to announce that our team just released PhotoSuru - a complete sample application demonstrating how to use the Syndicated Client Experiences Starter Kit for photo-viewing scenarios.

Big kudos to Kevin Gjerstad (our GPM) and the many other folks on the team who worked on this fantastic WPF application.

Check out photoSuru here and here. It will blow your mind :)

Posted Friday, January 09, 2009 12:44 PM by ivom | 4 Comments

WPF Application Quality Guide v.0.4 Released!

I am happy to announce that we have just released the fourth preliminary version of the “WPF Application Quality Guide” with the following new sections:

• a new Data-Driven Testing section,
• an updated article on Globalization and Localization Testing,
• a new article on Stability and Stress Testing,
• a new article on Accessibility Testing,
• and a brief introduction on TestApi – the test API library we released a couple of weeks ago.

From here, we will be heading to releasing v1 in the first quarter of 2009. As always, your feedback is greatly appreciated.

The guide is available here. You can also download it as a Word document.

Posted Thursday, December 18, 2008 7:31 PM by ivom | 3 Comments

Introduction to TestApi – Part 2: Command-Line Parsing APIs

Command-line parsers remind me of linked lists in C++: everybody has written several at various points in their careers. While everybody should write each of those at least once, I doubt that many people out there are particularly excited about writing and re-writing fundamental data structures on a regular basis – it gets old very quickly. Not to mention that doing so is error-prone and decreases the maintainability of a world that’s already hard to maintain.

That’s why modern-day frameworks such as .NET provide standard implementations of the common data structures. And that’s why TestApi provides a reusable command-line parsing APIs via the CommandLineDictionary and CommandLineParser classes, the latter being a type-safe layer on top of the former. Obviously, these are not test APIs per se – they are general utility APIs that happen to be more often used when writing tests.

A few quick examples follow.

Simple Command-Line Parsing

As seen from the first example below, extracting command-line parameters that are primitives is easy. Primitive command-line parameters are either boolean (e.g. the “verbose” flag below), or a key-value pair, that one can extract with the indexer of the CommandLineDictionary instance (see the “testId” key below), just as one would expect from a Dictionary.

// 
// EXAMPLE #1: Parsing a command-line such as "RunTests.exe /verbose /testId=123"
// 

using System;
using Microsoft.Test;

public class Program 
{ 
    public static void Main(string[] args) 
    { 
        CommandLineDictionary d = new CommandLineDictionary(args);

        bool verbose = d.ContainsKey("verbose");
        int testId = Int32.Parse(d["testId"]);
                
        // use the parsed command-line parameters
    }
}

By default flags/keys are indicated with the forward slash (“/”) character and values are indicated with the equals character (“=”), but the user can override that upon initialization of of the CommandLineDictionary object:

// 
// EXAMPLE #1b: Parsing a command-line such as "RunTests.exe –verbose –testId:123"
// 

...
        CommandLineDictionary d = new CommandLineDictionary(args, '-', ':');
...

Finally, you one can use the ToString method to get a string representation of the command-line arguments.

Command-Line Argument Structures

Another common pattern when dealing with command-line arguments is populating a structure which contains all parsed arguments. The CommandLineParser class makes this easy:

// EXAMPLE #2:
// Sample for parsing the following command-line:
// Test.exe /verbose /runId=10
// This sample declares a class in which the strongly typed arguments are populated
public class CommandLineArguments
{
   bool? Verbose { get; set; }
   int? RunId { get; set; }
}

CommandLineArguments a = new CommandLineArguments();
CommandLineParser.ParseArguments(a, args);

Type-Safe Commands

A third common approach is forming strongly-typed commands from the command-line parameters. This is common for cases when the command-line looks as follows:

some-exe  COMMAND  parameters-to-the-command

The parsing in this case is a little bit more involved:

1. Create one class for every supported command, which derives from the Command abstract base class and implements an expected Execute method.

2. Pass an expected command along with the command-line arguments to CommandLineParser.ParseCommand – the method will return a  strongly-typed Command instance that can be Execute()-d.

// EXAMPLE #3:
// Sample for parsing the following command-line:
// Test.exe run /runId=10 /verbose 
// In this particular case we have an actual command on the command-line (“run”), 
// which we want to effectively de-serialize and execute.
public class RunCommand : Command
{
   bool? Verbose { get; set; }
   int? RunId { get; set; }

   public override void Execute()
   {
      // Implement your "run" execution logic here.
   }
}

Command c = new RunCommand();
CommandLineParser.ParseArguments(c, args); 
c.Execute();

Besides the parsing logic, CommandLineParser provides a few additional helper methods. One of them is CommandLineParser.PrintCommandUsage, which prints the usage for specific commands (or all supported commands) to the console.

In Conclusion

The command-line parsing APIs released with TestApi provide a simple and “layered” access to the command-line. Strictly speaking these APIs are not test APIs, but have nevertheless been included in TestApi as tests often have a need of parsing parameters on the command-line.

Posted Wednesday, December 17, 2008 2:15 PM by ivom | 6 Comments

Introduction to TestApi – Part 1: Input Injection APIs

I am starting a series of posts introducing some of the facilities available in TestApi 0.1, a test and utility API library, which we recently released on CodePlex. Most of this content is already available in the documentation provided with the library.

The first post is on input injection – a fairly common activity in UI testing.

General Notes

Input injection is the act of simulating user input. In general, there are several ways to simulate user input, in the following progressively increasing levels of realism:

1. Direct method invocation: A test programmatically triggers events by directly calling methods on the target UI element. For example, a test can call the Button.IsPressed method to simulate pressing a WPF button.
2. Invocation using an accessibility interface (UIA, MSAA, etc.): A test programmatically triggers events by calling methods on an AutomationElement instance that represents the target UI element.
3. Simulation using low-level input: A test simulates input by using low-level input facilities provided by the host operating system. Examples of such facilities on Windows are the SendInput Win32 API and the Raw Input Win32 API, which inject input directly into the OS input stream.
4. Simulation using a device driver: A test uses a device driver to simulate input at the device-driver level.
5. Simulation using a robot: A test controls a robot to simulate direct human interaction with an input device (for example, pressing keys on a keyboard).

Technique A is framework-specific; what works for WPF does not work for Windows Forms and vice versa. Technique B is less framework-specific than A, but still has limitations, because some frameworks differ in their implementations of the required accessibility interfaces. Techniques C and D are OS-specific. Technique D is significantly more difficult to implement and deploy than C, without a corresponding increase in its level of realism. Technique E is universal, albeit much slower and much more expensive than the other options.

The TestApi library provides facilities both for B (through the AutomationUtilities class) and for C (through the Mouse and Keyboard classes), which are the most generally useful techniques of input simulation.

Examples

The AutomationUtilities class provides wrappers for common UIA operations, such as discovery of UI elements. The first example below demonstrates how to discover and click a WPF Button in a WPF Window, by using the AutomationUtilities class and the Mouse class.

//
// EXAMPLE #1
// This code below discovers and clicks the Close button in an About dialog box, thus
// dismissing the About dialog box.
//

string aboutDialogName = “About”;
string closeButtonName = “Close”;

AutomationElementCollection aboutDialogs = AutomationUtilities.FindElementsByName(
    AutomationElement.RootElement,
    aboutDialogName);

AutomationElementCollection closeButtons = AutomationUtilities.FindElementsByName(
    aboutDialogs[0],
    closeButtonName);

//
// You can either invoke the discovered control, through its invoke pattern...
//

InvokePattern p = 
    closeButtons[0].GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
p.Invoke();

//
// ... or you can handle the Mouse directly and click on the control.
//

Mouse.MoveTo(closeButton.GetClickablePoint());
Mouse.Click();

The second example below demonstrates how to discover a TextBox instance and type in it, using the Mouse and Keyboard classes as wrappers of common mouse and keyboard operations:

//
// EXAMPLE #2
// Discover the location of a TextBox with a given name.
//

string textboxName = “ssnInputField”;

AutomationElement textBox = AutomationUtilities.FindElementsByName(
    AutomationElement.RootElement,
    textboxName)[0];

Point textboxLocation = textbox.GetClickablePoint();

//
// Move the mouse to the textbox, click, then type something
//

Mouse.MoveTo(textboxLocation);
Mouse.Click();

Keyboard.Type(“Hello world. ”);
Keyboard.Press(Key.Shift);
Keyboard.Type(“hello, capitalized world.”);
Keyboard.Release(Key.Shift);

In Conclusion

The Mouse and Keyboard classes in the TestApi library can be used for automating of any application, running on Windows. The classes are completely policy- and context-free – their usage is not dependent on a specific test framework or on a specific test workflow. TestApi provides full source code and XML documentation for these classes, so you can either integrate them in your own projects or reference the pre-built DLLs.

Note that, even though Mouse, Keyboard and AutomationUtilities make the life of the test automation developer quite a bit easier, UI testing is tricky and should be avoided whenever possible. It’s always preferable to design your application as a multi-tier application, with a “thin” UI layer, so that you can bypass the UI in most of your tests.

Posted Monday, December 15, 2008 7:49 PM by ivom | 12 Comments

TestApi v.0.1 Released

On Friday, we released the first preliminary version of an experimental library with test APIs, named ... TestApi :). With this library, we are sharing internal Microsoft testing technology with the developer and tester community.

We wanted to keep the first release small and contained, so that it can be easily grasped and critiqued, but complete enough to provide actual user value and to clearly demonstrate the direction of this effort.

In addition to the binaries and the source code, we have provided full documentation (both MSDN-style API documents and conceptual documents) as well as a couple of VSTS samples to get you started.

Check out the release on http://codeplex.com/TestApi and let us know what you think.

Posted Saturday, December 06, 2008 10:30 AM by ivom | 1 Comments

WPF Application Quality Guide v.0.3 Released!

On Friday, we released the third preliminary version of the "WPF Application Quality Guide", adding a section 3 new sections: a suggested reading roadmap, a section on globalization and localization testing and a section on manual testing and record and replay.

The guide is available here. You can also download it as a Word document.

As always, please send us suggestions both on the content and on the organization of the document. You can post feedback directly here or send it to wpftbest@microsoft.com.

Posted Sunday, September 28, 2008 8:52 PM by ivom | 9 Comments