Ivo Manolov's Blog

Introduction to TestApi – Part 5: Managed Code Fault Injection APIs

Fault Injection is the act of artificially changing the behavior of an existing executable code to simulate various faults. FI is very useful for validation of error handling code paths and for improving code coverage.

There are several types of fault injection. In runtime fault injection, the fault injecting test modifies the execution logic of the application under test (AUT), by injecting faults, triggered by specific runtime conditions. One could for example implement a FI test with the following semantic:

Throw an out-of-memory (OOM) exception, whenever the application calls method CreateWidget of class WidgetManager.

The FI terminology is as follows:

• AUT (application under test) – this is the tested application, in which faults are being injected;
• Fault Rule – The fault rule is a central construct in an FI test that determines WHEN faults get triggered and WHAT TYPES of faults get triggered. A fault rule consists of:
  • a Method Signature, determining the method where the fault will be injected;
  • a Fault Condition, determining when the specific fault should be triggered (e.g. every N^th call)
  • a Fault -- Determines the type of fault (e.g. throwing an exception, returning a specific value, etc.) that occurs when the fault condition is met.
• Fault Session – The fault session is a collection of fault rules that are applied to a given AUT.

TestApi provides a simple, but powerful runtime fault injection API for injecting faults in managed code. The API was originally designed and implemented by Bill Liu et al from the “Essential Business Server” team, and adapted to TestApi by Sam Terilli from our WPF XAML team. The following content provides a quick introduction to the API.

Sample AUT

Following is the code of a trivial AUT that we will use for demonstration purposes:

//
// This is a sample application used for demonstration purposes.
//
using System;

class MyApplication
{
    static void Main(string[] args)
    {
        int a = 2;
        int b = 3;

        for (int i = 0; i < 10; i++)
        {
            Console.WriteLine("{0}) {1} + {2} = {3}", i, a, b, Sum(a, b));
        }
    }

    private static int Sum(int a, int b)
    {
        return a + b;
    }
}

The result of running this application is of course:

A Simple Fault Injection Test

Now, let’s try to inject a fault in the AUT. Let’s assume that we want to modify the return value of Sum. Here’s how we can accomplish that:

// 
// Simple fault injection test
//
using System;
using System.Diagnostics;
using Microsoft.Test.FaultInjection;

public class FaultInjectionTest 
{ 
    public static void Main() 
    { 
        // 
        // Set up a fault rule to return –1000 the second time Sum is called.
        //
        string method = "MyApplication.Sum(int,int)";
        ICondition condition = BuiltInConditions.TriggerOnNthCall(2);
        IFault fault = BuiltInFaults.ReturnValueFault(-1000);
        FaultRule rule = new FaultRule(method, condition, fault);            

        //
        // Establish a session, injecting the faults defined by the fault rule(s)
        //
        FaultSession session = new FaultSession(rule);
        ProcessStartInfo psi = session.GetProcessStartInfo(@".\MyApplication.exe"); 

        //
        // Launch the target process and observe faults
        //
        Process p = Process.Start(psi);        
        p.WaitForExit();
    }
}

Fairly straightforward. Upon running the test (which will itself spawn the AUT) we observe the following output:

> FaultInjectionTest.exe
0) 2 + 3 = 5
1) 2 + 3 = -1000
2) 2 + 3 = 5
3) 2 + 3 = 5
4) 2 + 3 = 5
5) 2 + 3 = 5
6) 2 + 3 = 5
7) 2 + 3 = 5
8) 2 + 3 = 5
9) 2 + 3 = 5

As intended, we injected a runtime fault in MyApplication.exe, which resulted in Sum returning –1000 the second time it got called.

Under The Covers

Under the covers, the managed code Fault Injection API uses the CLR profiling API to modify the prologue of the intercepted method at runtime in order to inject the desired fault. The injected prologue instructions essentially call a method in the library, which then dispatches the call to the specified fault.

Because faults are injected at runtime, the code of the original application is not modified in any way. There is a certain performance degradation, which depends on the number of the injected faults.

The fault injection API provides a variety of built-in conditions and faults (in the BuiltInConditions and BuiltInFaults classes respectively). Users of the API can also create custom conditions and faults (by implementing the ICondition and IFault interfaces respectively). The API also provides a set of classes that expose the ability to fine tune and monitor the injected faults.

In addition, the API provides a facility to set “global faults”, which is useful for server application testing, where one application typically consists of and recycles many different processes.

I have attached the sample above, which should get you up and running with fault injection.

Application Accessibility Testing

A lot of our customers and partners have asked us to provide guidance on how to make their WPF and Silverlight applications accessible, so I decided to publish a post folks can refer to. Note most of the content below is directly applicable to any other Windows application.

Introduction

In general “Accessibility” means two things:

1. Nominal accessibility: An application or an application programming stack exposes the necessary information so that it makes its UI elements accessible;
   
2. Accessibility with an accessibility tool: An accessibility tool (e.g. a screen reader) can access the UI of a given application and expose the UI to its users.

For an application to be “accessible”, ideally it would satisfy both (1) and (2) i.e. be both nominally accessible and also accessible with popular accessibility tools. Satisfying (1) guarantees that the level of accessibility of an application does not depend significantly on the accessibility tool. Satisfying (2) guarantees that a disabled user can interact with an application, when using a particular accessibility tool. Satisfying (2) is clearly more important, and as a matter of fact the various accessibility tools go to extraordinary lengths to make applications that are not nominally accessible be accessible through the tool.

+++

Accessibility tools use several techniques to build an accurate in-memory representation of what is presented on the screen:

1. Public accessibility API
2. Public platform API
3. Kernel interfaces
4. In-process code injection

Microsoft provides two accessibility API – MSAA and UIA. MSAA is a legacy API, introduced circa 1996, which is not extensible and is no longer sufficient for modern UIs. UIA is the new much richer and fully extensible Accessibility API from Microsoft, originally introduced in WPF / Windows Vista in 2007.

It’s important to note that many existing Win32 applications are not nominally accessible. I.e. they do not expose the right accessibility information, using public accessibility API. They are only made accessible through the use (by the accessibility tools) of techniques such as (ii), (iii) and (iv) above.

+++

Accessibility API typically expose:

1. Provider API : these are the API used by an application stack to expose its UI elements;
2. Client API: these are the API used by an accessibility tool (e.g. a screen reader) to query the UI elements of an application.

MSAA and UIA both conform to that client-provider model. There is also a bridge between MSAA and UIA, which allows a client using the MSAA client API to access an application exposed through UIA.

General Accessibility Testing Workflow

Microsoft provides a number of accessibility testing tools that can help with identifying of nominal accessibility problems. Combining these with the target accessibility tool is the winning recipe for creating of fully accessible applications.

Below is a suggested accessibility testing work-flow:

For contrast, DPI, color and magnification verification (step 4 above), use the following tools and steps:

WPF Accessibility

WPF is a fully accessible stack. WPF uses UIA (in fact the original UIA team used to be part of the WPF team) to provide access to the various UI elements. All standard WPF elements (packaged controls, etc.) implement the necessary client-side UIA interfaces to expose themselves to accessibility readers.

Development and testing of nominally accessible WPF applications is fairly straightforward. For more information, refer to the “Accessibility Testing” section in the “WPF Application Quality Guide”.

While following these guidelines will result in nominally accessible applications, it is important to once again stress that all applications should also be tested with the target accessibility tool to confirm that they are truly accessible from the point of view of a disabled user.

TestApi v0.4 Released!

I am happy to announce that we have just released version 0.4 of TestApi – the testing API library - on Codeplex.

Version 0.4 contains the following additions and modifications:

• An improved Combinatorial Variation Generation API with added support for parameter value weights and tags (for “negative” variations);
• A new Memory Leak Detection API allowing capture, comparison and serialization / de-serialization of memory snapshots of a running process;
• A new Object Comparison API allowing comparison of arbitrary .NET objects, using arbitrary object comparison strategies;
• A new Text String Generation API, allowing generation of random strings, for testing purposes;
• Various documentation and sample improvements.

We will be blogging about some of these API in the near future. As always, TestApi is available on http://codeplex.com/testapi.

TestApi Slide-Deck from the Patterns & Practices Summit

This week is Microsoft’s Patterns & Practices summit in Redmond. Earlier today I gave a 15-minute presentation on TestApi. Here is the slide-deck:

The Gradual Development Paradigm

Software project management is tricky. The industry proves this over and over again: delivering less than expected, slipping features, projects, milestones, products…

The gradual development idea presented below is not something new – it's more or less common-sense. Often, however, not all positive implications of it are fully understood or internalized, so I thought I’d share this article, which I sent to my team several years ago.

Admittedly, some of this is old news. After all we live in the age of Agile, rapid iteration and integration, test-driven design and development. It still surprises me how many teams tend to slip in old behaviors on a regular basis, so I have personally found these practices worth reviewing and repeating.

Case Study

Imagine that you have 5 developers and that you are 1 day before your deadline. If they have not checked in anything, you are scared (understandably): they may deliver anything between 500% (100% each) and 0%. That's a lot of stress for a team lead.

Now imagine that you have 5 developers, who have delivered gradually in smaller pieces. Understandably, they have delivered less (because gradual development is more expensive), but at day-1, you know that you are at say 450% so your uncertainty is only 50% (rather than 500%) or even less (since as a result of the gradual development you have lost time, so you know you can only accomplish 20% more summing up to say 470%).

Which situation would you choose to be in: 500% of risk with a potential heart attack or 20% of risk with a known under-delivery and 8-hour work days at the end of the project?

The Theory of Gradual Development

The diagram below presents the principle of gradual development. Person A has adopted the "I will code it on my computer and worry about code and design review, deployment, logistics at the end when I am done" approach. Person B has adopted the "gradual development" approach.

But wait a minute, I hear you say, person B is actually slipping while person A may deliver on time after all.

What is missing from the diagram is that it is never a 1-person team. We usually talk about teams of 5, 10, 20 people. Let's assume that there is a 5% probability that Person A will not deliver on time i.e. there is a 95% probability that Person A will deliver as expected (that's a good healthy number – everyone who has shipped a product will tell you that is a pretty high confidence.

Then for a team of 20 people, the combined probability to actually deliver on time is 95%²⁰ = 36%. That's encouraging![1] Here are the expanded results:

Or, as a chart:

Note that for a team of 200 people, the probability to deliver on time is close to 0%!

How does gradual development help? Well, on "ship day - 1", you know that your team has already shipped 90% of the product, so the probability to ship on time is 90% + (10% * 95%²⁰) = 94%. Way better than 0%, isn't it?

The Practice of Gradual Development

Here are some practical directions:

1. Defy the urge to produce big chunks

All of us want to impress our bosses or peers by creating something big and delivering it as a whole. That's one sure way to produce the "wow effect". In reality however, you rarely (if ever) get to impress anybody that much. However, you can, truly, build a reputation of a rock-solid and predictable achiever. People who have this reputation are awarded and valued everywhere, because the mere ability to ship and sell is what defines a business.

2. Constantly ask yourself "how can I deploy this today?"

Gradual development is great in theory, but in practice it's just too easy to slip back to the regular "I am going to show them how fast and complete I am" or "I'll have enough time to deploy after I am done" routines.

Gradual development does not come naturally to any of us. It (much like OOP) is an acquired taste, so we have to constantly remind ourselves that we have to constantly ship. You are not ready with the whole implementation? Not a problem! Stub it (e.g. make the unimplemented functions to return E_NOTIMPL) and ship it.

“Deploying” or “Shipping” may mean different things to different people:

• To a developer, "shipping" means adding the feature to the build and to the installation of the product.
• To a tester, "shipping" means deploying a test in the lab (as "lab ready").
• To a PM, "shipping" means publishing a spec (although the spec may not be fully ready).

3. Maintain a rapid create-test-deploy cycle

People often forget that deployment is part of the job. In fact it is a very significant and time consuming part of the job. We all came into the software development business, because we wanted to program. It turns out, however, that software development is not only about programming. Strangely enough pure programming does not really occupy most of our time: deployment and maintenance does. Embrace this reality and make it a goal not to program, but to deploy.

4. If you are a manager, reward gradual development

You want predictability from your business? Then reward predictability (e.g. predictable achievers, rather than unpredictable Einsteins[2].)

5. If you are an individual contributor, keep in mind the perspective of your manager

Remember that while you can happily live with a confidence of 95%, your manager is a manager of 5, 10 20 people, so he / she has to live with a significantly reduced confidence.

+++

[1] For simplicity, it is assumed that the probability of successful delivery from a given team member is independent from other team member's deliveries i.e. for a team of two people, the combined probability P_combined = P(contributor 1 delivers on time) * P(contributor 2 delivers on time). If this assumption in not valid, the resulting combined probability is even lower!

[2] This of course is taken to extreme. You absolutely need the Einsteins. In fact, it is “manageable Einsteins” that often make the difference between a moderately successful and a hugely successful product.

Introduction to TestApi – Part 4: Combinatorial Variation Generation APIs

The state and behavior of every software system depends on a number of parameters. These parameters can be both inputs as well as environmental factors.

One can think of software testing as a controlled experiment, where the system under test is observed upon varying of the parameters that affect it, in an attempt to discover unexpected system behavior.

Depending on who you talk to, the act of generation of these variations is called variation generation, matrix expansion, design of experiments (DOE), etc.

+++

The Problem

Modern-day software systems are complex and depend on many parameters. Expanding all possible combinations of all parameter values often results in a phenomenon called matrix explosion – having an overwhelmingly high number of test variations.

Matrix explosion is undesirable because:

1. It increases the runtime of your tests (which, among other things, makes TDD impractical and TDD is the single most effective way to ensure high product quality);
2. It increases the support costs of your tests – for a test pool of 100,000 tests, even with 99% pass rate (which in reality is hard to achieve), you still have to investigate 1,000 test failures. Investigating 1,000 test failures is always more expensive than investigating say 10 failures;
3. It is often impractical or impossible to cover all combinations, so you end up with partial test coverage (very often “vanilla” test coverage).

There are many ways to deal with matrix explosion, which depend on various system and/or experimental constraints. The theory and practice of DOE deals with that. 

TestApi’s Solution

TestApi provides a generic API for combinatorial variation generation, using the algorithm presented in Jacek Czerwonka’s “Pairwise Testing in Real World” article. The API uses the following nomenclature:

• Parameter – represents a single factor / variable and its values;
• Constraint – represents a relationship between parameters, their values, constants, and other constraints;
• Model – contains all parameters and constraints for the system, for which we are generating variations;
• Variation – represents a tuple with a single value for every Parameter in the Model.

All of these are represented as correspondingly named types. Following are several examples demonstrating the use of the API.

Example 1 : Simple Matrix

For the purposes of a simple artificial example, consider having a system with the following parameters and values:

Assuming, there are no constraints, here is the code you would use to create a model

using System;
using System.Collections.Generic;
using Microsoft.Test.VariationGeneration;

class Example1
{
    static void Main(string[] args)
    {
        //
        // Declare all parameters and construct a model
        //
        Parameter p1 = new Parameter("Color") { "White", "Green", "Red" };
        Parameter p2 = new Parameter("Height") { "Short", "Tall" };
        Parameter p3 = new Parameter("Size") { "Small", "Medium", "Large" };

        Model m = new Model(new List<Parameter>
 { p1, p2, p3 } );

        //
        // Generate and print out all possible variations of the parameters in the model
        //
        foreach (Variation v in m.GenerateVariations(3, 1234))
        {
            System.Console.WriteLine("{0}\t{1}\t{2}", v[p1.Name], v[p2.Name], v[p3.Name]);
        }
    }
}

Executing the code below produces the following 3*2*3=18 variations, representing all possible combinations of the values of the three parameters:

> Example1.exe
White   Short   Small
White   Short   Medium
White   Short   Large
White   Tall    Small
White   Tall    Medium
White   Tall    Large
Green   Short   Small
Green   Short   Medium
Green   Short   Large
Green   Tall    Small
Green   Tall    Medium
Green   Tall    Large
Red     Short   Small
Red     Short   Medium
Red     Short   Large
Red     Tall    Small
Red     Tall    Medium
Red     Tall    Large

The variations are generated by the call to GenerateVariations(3, 1234). The number "1234" is the seed for the random generator, utilized by the algorithm. The number "3" is the order of the generated combinations. The order of generated combinations must be a number between 1 and the number of parameters in the model. The output for orders "1" and "2" are presented below:

White   Short   Small
Green   Tall    Medium
Red     Short   Large

White   Short   Small
White   Tall    Medium
White   Short   Large
Green   Tall    Small
Green   Short   Medium
Green   Tall    Large
Red     Short   Small
Red     Tall    Medium
Red     Tall    Large

Example 2 : Vacation Planner

This example (created by Nathan Anderson – our engineer who designed and implemented the combinatorial variation generation API) demonstrates the use of parameter constraints.

Parameter destination = new Parameter("Destination") { "Whistler", "Hawaii", "Las Vegas" };
Parameter hotelQuality = new Parameter("Hotel Quality") { 5, 4, 3, 2, 1 };
Parameter activity = new Parameter("Activity") { "gambling", "swimming", "shopping", "skiing" };

List<Parameter>
 parameters = new List<Parameter>
 { destination, hotelQuality, activity };
List<Constraint> constraints = new List<Constraint>
{
    new IfThenConstraint
    {
        If = destination.Equal("Whistler").Or(destination.Equal("Hawaii")),
        Then = activity.NotEqual("gambling")                        
    },
    new IfThenConstraint
    {
        If = destination.Equal("Las Vegas").Or(destination.Equal("Hawaii")),
        Then = activity.NotEqual("skiing")
    },
    new IfThenConstraint
    {
        If = destination.Equal("Whistler"),
        Then = activity.NotEqual("swimming")
    },
};

Model model = new Model(parameters, constraints);

//
// Call the method under test with each generated variation
//
foreach (var variation in model.GenerateVariations())
{
    CallVacationPlanner(
        (string)variation[destination.Name], 
        (int)variation[hotelQuality.Name], 
        (string)variation[activity.Name]);
}

Example 3 : The WPF Platform Matrix

This last example demonstrates how we deal with a real-world problem we face in the WPF team...

WPF must work reliably on all OS configurations, defined by the following matrix:

Most of the parameters are self-descriptive. “Side-by-side” captures the .NET installation state. For example “3.5 SP1 + 4 - 4 (3.5 tests)” means “install .NET 3.5 SP1, install .NET 4, uninstall .NET 4, run tests built against 3.5 SP1”. This is done to confirm that there are no unexpected side effects as a result of the installation and un-installation of .NET 4. 

The trivial full expansion of the matrix results in 583,200 combinations (=6*25*2*2...). Of course, some of these combinations (e.g. XP SP2 OS with a Aero theme) are not valid, but even after removing the invalid combinations, we still end up with a prohibitively large number of platform configurations to test on.

There are several ways to deal with this problem. One is identifying the so called equivalence classes. For example, from the point of view of Side-by-Side, Vista SP1 and Server 2008 SP1 can be regarded as equivalent OS-es and so on. Another popular approach is reducing regular testing to “vanilla configurations” (e.g. mostly ENG (English), 96-DPI configurations), venturing outside of the “vanilla domain” in accordance with a predefined schedule (e.g. during test passes at the end of major milestones).  A third approach is using a pair-wise combinatorial variation generator, reducing the number of platform variations to about 230 – still a fairly high number for any real-world test pool, but clearly much better than the original number above.

In the WPF team, we use the third approach, combined with an adaptive random algorithm, which prioritizes testing on platform configurations that have not been tested on recently. The simplified code below demonstrates how to construct a model for platform config variation generation.

using System;
using System.Collections.Generic; 
using Microsoft.Test.CommandLineParsing; 
using Microsoft.Test.VariationGeneration;
using Microsoft.Test.VariationGeneration.Constraints;

public class OsVariationGeneration
{
    public static void Main(string[] args)
    {            
        CommandLineDictionary d = CommandLineDictionary.FromArguments(args);
        int order = Int32.Parse(d["order"]);
        int seed = Int32.Parse(d["seed"]);

        //
        //  Parameters
        //
        Parameter os = new Parameter("OS") { "Windows XP SP3", "Windows Vista SP1", "Windows 7", "Windows Server 2003 SP2", "Windows Server 2008 R2" };
        Parameter language = new Parameter("language") { "ARA", "CHS", "CHT", "CSY", "DAN", "DEU", "ELL", "ENG", "ESN", "FIN", "FRA", "HEB", "HUN", "ITA", "JPN", "KOR", "NLD", "NOR", "PLK", "PSE", "PTB", "PTG", "RUS", "SVE", "TRK" };
        Parameter sysLocale = new Parameter("sysLocale") { "SameAsOsLanguage", "TRK" };
        Parameter flavor  = new Parameter("flavor")     { "fre", "chk" };
        Parameter platform = new Parameter("platform")  { "x86", "x64", "x64wow" };
        Parameter ieVersion = new Parameter("ieVersion") { "osDefault", "ie7", "ie8" };
        Parameter highDpi = new Parameter("hiDpi") { "yes", "no" };
        Parameter theme = new Parameter("theme") { "Classic", "Luna", "Royale", "Classic High Contrast", "Aero Basic", "Aero Glass" };
        Parameter sxs = new Parameter("sxs") 
        {
            "3.5 SP1 + 4 (3.5 tests)",
            "3.5 SP1 + 4 (4 tests)",
            "3.5 SP1 + 4 - 4 (3.5 tests)",
            "4 + Mock 4.5 (4 tests)",
            "4 + Mock 5 (4 tests)",
            "4 + 3.5 SP1 (4 tests)",
            "4 + 3.5 SP1 (3.5 tests)",
            "4 + 3.5 SP1 - 4 (3.5 tests)",
            "4 (4 tests)",
        };
        List<Parameter>
 parameters = new List<Parameter>
 { os, language, sysLocale, flavor, platform, ieVersion, highDpi, theme, sxs };


        //
        //  Constraints
        //
        List<Constraint> constraints = new List<Constraint>
        {
            new IfThenConstraint
            {
                If = os.Equal("Windows XP SP3").Or(os.Equal("Windows Server 2003 SP2")),
                Then = theme.NotEqual("Aero Basic")
            },
            new IfThenConstraint
            {
                If = os.Equal("Windows XP SP3").Or(os.Equal("Windows Server 2003 SP2")),
                Then = theme.NotEqual("Aero Glass")
            },
            new IfThenConstraint
            {
                If = os.Equal("Windows 7").Or(os.Equal("Windows Server 2008 R2")),
                Then = theme.NotEqual("Luna")
            }
        };


        //
        //  Model
        //
        Model m = new Model(parameters, constraints);

        uint i = 0;
        foreach (Variation v in m.GenerateVariations(order, seed))
        {
            Console.WriteLine(
                "{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\t{7}\t{8}\t{9}", 
                i,
                v[os.Name],
                v[language.Name],
                v[sysLocale.Name],
                v[flavor.Name],
                v[platform.Name],
                v[ieVersion.Name],
                v[highDpi.Name],
                v[theme.Name],
                v[sxs.Name]
                );
            i++;
        }
    }
}

Conclusion

Pairwise variation generation is an important tool in your toolbox as a test author. TestApi provides a simple facility for combinatorial variation generation. We will of course be evolving this facility, but do let us know if you have specific scenarios or requirements you'd like to see supported.

WPF Control Verifier v0.1 Released!

Our strategy to improve WPF and .NET application quality has 3 major pillars:

1. Provide guidance on how to create high-quality applications;
2. Provide reusable API libraries for automated application testing and monitoring;
3. Provide tools to help create and debug applications and components.

To address (1), we have released several versions of the WPF Application Quality Guide, the initial version of the WPF Model-View-ViewModel Toolkit (available for download here), the WPF Performance Guide documents on MSDN, and others. To address (2), we are investing in TestApi – the test API library. For (3), we have released the WPF Performance Suite on MSDN.

I am excited to announce that we have another addition to the third group : the WPF Control Verifier v0.1 (available for download from here).

The WPF Control Verifier is a tool that verifies the correctness of WPF controls. This first version of the tool includes a set of default style tests that you can run on your WPF custom controls. The verifier was designed, implemented and tested by Vincent Sibal and Alexis Roosa – engineers on our Controls team. The tools has already proven extremely valuable in our internal testing of the standard WPF controls uncovering several important bugs that have been fixed, so do check it out. And of course send us your feedback, feature requests, etc.

TestApi v0.3 Released!

I am happy to announce that we have just released the third preliminary version of TestApi -- the testing API library. This version introduces some fairly significant changes and new features:

• New name spaces (under Microsoft.Test) and new binary names;
  
• New source code organization;
  
• A fairly comprehensive acceptance tests suite (using xUnit);
• Fault Injection APIs for managed code (using CLR profiler technology);
• Combinatorial Variation Generation API (using the PICT algorithm);
• A redesigned Application Control API;
• Updates to the Visual Verification API;
  
• Updated samples and documentation; new conceptual documents;
  

TestApi is a policy-free API library, delivering Microsoft testing technology in an easy to use package. It can be used from within any tool (VSTS, NUnit, xUnit, etc.) or environment to automate the testing for desktop and in-browse applications on Windows.

TestApi v0.3 is available on http://testapi.codeplex.com. Thanks for all your feedback. Keep it coming! :)



 

WPF MVVM Toolkit 0.1 Released!

Model-View separation is not a novel idea in the software industry. It has been around for at least 30 years. Recently, however, MV separation is seeing a lot of renewed interest, powered both by the growing complexity of software and by the need to provide different UI experiences, reusing the same underlying business logic.

WPF provides a lot of native functionality (commanding, databinding, etc.), enabling developers to build applications with proper model-view separation. So far, however, we have not provided simple guidance on how to utilize these features to build properly factored applications with model-view separation.

***

I am very happy to announce that on Friday we released the first preliminary version of the WPF Model-View-ViewModel Toolkit (MVVM is WPF’s equivalent of the classic MVC design pattern). The toolkit is available for download on the “WPF Futures”  codeplex site.

The toolkit consists of:

• A Visual Studio template, allowing you to quickly create WPF Model-View-ViewModel Applications
• Documentation
  • A general introduction to the MVVM design pattern
  • A detailed walk-through, demonstrating how to create a simple MVVM application, starting from the skeleton generated by the template
• A complete MVVM application example – a WPF messenger application.

Big kudos to Patrick Danino – an engineer on our team – who worked on the toolkit from inception to its successful first preliminary release.

***

This first preliminary version of the toolkit is fairly simple and clearly not aimed at MVVM experts. We have adopted an evolutionary approach of delivery, so we invite everybody out there – from folks looking to pick up MVVM for their next project to MVVM experts – to collaborate with us on the toolkit, by sharing both real-world experiences as well as first-time-exposure feedback.

We will use this feedback to drive improvements in the toolkit, in the WPF platform and in the relevant authoring tools (Visual Studio, Expression Blend, etc).

WPF Application Quality Guide v0.5 Released!

I am happy to announce that we have released CTP 5 of the “WPF Application Quality Guide” – our single-stop document for WPF application and component developers and testers.

CTP 5 comes with the following new and updated content:

• New content:
  • “Considerations for WPF Browser Applications”
  • “Integration and Scenario Testing”
  • “XAML Editing Tools and Visual Studio Add-ins”
  • “Building a WPF Application Test Suite by Using VSTS, NUnit, or xUnit”
    
• Updated content:
  • Reading roadmap
  • “Performance and Scalability Testing”
  • “TestApi”
  • “Debugging Tools”
  • “Performance Profiling Tools”
  • “WPF Application Design and Development Tools”

The Guide is available here and here as a DOC file. Big kudos to Anne Gao on our team, who has been PM-ing all releases of the guide to date.

***

Those of you who have been following the Guide have seen it evolve from its first CTP (link) of 20 or so pages to its present size of more than 80 pages.

Any 80-page document is tricky to grok, so we have been thinking of reorganizing the content and potentially splitting the content into two documents – one on client application development and testing fundamentals and one on WPF specifics. We prefer to do any edits based on user feedback, so please let us know of the current organization and content of the document and share your ideas on how we could improve both.

TestApi v.0.2 Released!

I am excited to announce that we have just released the second preliminary version of TestApi – the testing API library – on http://codeplex.com/testapi!

The v.0.2 package includes the following additions and modifications:

• Improved command-line parsing APIs
• Improved visual verification APIs
  • A new tolerance map visual verifier in SnapshotToleranceMapVerifier
  • New operations on snapshot (And and Or) allowing you to mask
  • A new Snapshot.FromWindow(...) constructor with ability to include and exclude the window chrome in snapshots.
  • Improved visual verification internals
• New application control APIs, enabling in-proc and out-of-proc automation of client applications
• Expanded conceptual documentation
• CHM API documents (in addition to the HTML documents)
• Addition of NUnit and xUnit usage samples (for all of the non-VS crowd out there)

Check out the package and let us know of any feedback.

Introduction to TestApi – Part 3: Visual Verification APIs

Visual Verification (VV) is the act of verifying that your application or component is displayed correctly on screen. The TestApi library provides a set of VV APIs. This post discusses these APIs.

Avoid It If You Can

First and foremost, I want to emphasize that visual verification is a test technique that should be used with caution. It is difficult to do correctly and any extensive use typically results in hard-to-maintain test codebases. Here are a few things to consider before you embark on VV test development:

1. The UI of an application tends to change a lot during development. If you use visual verification extensively, you may end up in a situation where you have to update a large number of visual verification tests on a daily basis, which is a waste of effort.
2. There are small differences in rendering between different video cards on different versions of the OS and of .NET. These differences are particularly pronounced in font rendering.
3. Before employing any form of UI verification, one should always review the underlying application architecture. Extensive need for UI testing is typically indicative of poorly architected systems, lacking proper view-model separation, so it’s almost always better to invest in proper system architecture than in extensive UI testing.
4. Whenever possible, attempt to do analytical visual verification i.e. one that does not employ the use of master images.

The WPF test team has gone through several iterations of cleaning up and retiring unnecessary visual verification tests in an attempt to speed up and stabilize our test suite.

General Concepts

The core VV terminology is:

• Snapshot: A pixel buffer used for representing and evaluating screen image data.
• Verifier: An oracle object which determines whether a snapshot passes against specified inputs.
• Actual: The snapshot being evaluated.
• Master: The reference data (image) which is used to evaluate the actual snapshot.
• Tolerance: The accepted bounding range based on which the actual snapshot will be accepted as valid.

The general VV workflow is:

1. Capture some screen content.
2. Generate an expected snapshot (e.g. load a master image from disk, etc.)
3. Compare the actual snapshot to the expected snapshot and generate the difference (diff) snapshot.
4. Verify the diff using a verifier.
5. Report test result.

TestApi Visual Verification Technology

TestApi provides the following VV technology:

• Snapshot: this class represents image pixels in a two-dimensional array for use in VV. Every element in the array represents a pixel in a given [row, column] of the image. A Snapshot object can be instantiated from a file (Snapshot.FromFile), or captured from screen (Snapshot.FromWindow and Snapshot.FromRectangle). Snapshot also exposes image cropping (Snapshot.Crop), resizing, diff-ing (Snapshot.CompareTo) masking (Snapshot.And) and merging operations (Snapshot.Or) operations.
  
• Verifiers: the library provides a set of verifiers that can be used to verify a (diff) snapshot. SnapshotColorVerifier reports passing if
  
• Various utilities: example of these are the Histogram class, providing basic functionality for handling image histograms

Examples

SnapshotColorVerifier

With these prolegomena out of the way, let’s look at some code. The first example below demonstrates master visual verification using a basic color verifier, which ensures that the difference between the master snapshot and the actual snapshot is within a defined tolerance:

// 1. Capture the actual pixels from a given window
Snapshot actual = Snapshot.FromRectangle(new Rectangle(0, 0, 100, 100));

// 2. Load the reference/master data from a previously saved file
Snapshot expected = Snapshot.FromFile("Expected.png"));

// 3. Compare the actual image with the master image
//    This operation creates a difference image. Any regions which are identical in 
//    the actual and master images appear as black. Areas with significant 
//    differences are shown in other colors.
Snapshot difference = actual.CompareTo(expected);

// 4. Configure the snapshot verifier - It expects a black image with zero tolerances
SnapshotVerifier v = new SnapshotColorVerifier(Color.Black, new ColorDifference());

// 5. Evaluate the difference image
if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the diff file for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

This approach works fine if you are evaluating  the correctness of an application logo or some other application art. You may find that you will need to increase the tolerance a bit to accommodate differences in GPU rendering, but in general the SnapshotColorVerifier provides all the functionality you need.

SnapshotHistogramVerifier

A somewhat more sophisticated approach involves using of image histograms (see this link for a good introduction to the subject). An image histogram is a histogram that represents the frequency of pixels with a certain brightness. One can define a histogram that represents his/her expectation of the “proximity of the match” between the actual and expected snapshots.

For example, one can define a histogram semantically equivalent to the following statement:

“When I compare the actual snapshot to the expected snapshot (both of 320 pixels), I expect no more than 30 pixels with color channel difference of 1, no more than 10 pixels with color channel difference of 2, and zero pixels with higher differences.”

This histogram would look as follows:

Figure 1  Image Histogram

Such form of verification is done by using the SnapshotHistogramVerifier and the Histogram classes, as demonstrated in the sample below.

// Take a snapshot, compare to the master image and generate a diff
Snapshot actual = Snapshot.FromRectangle(new Rectangle(0, 0, 100, 100));
Snapshot expected = Snapshot.FromFile("Expected.png"));
Snapshot difference = actual.CompareTo(expected);

// Load the quality histogram from disk and use it to verify the diff
SnapshotVerifier v = new SnapshotHistogramVerifier(Histogram.FromFile("ToleranceHistogram.xml"));

if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the actual and diff images for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

The histogram file is just a XML file with the following schema:

<histogram>
  <tolerance>
    <point x="0" y="0.87500" />
    <point x="1" y="0.09375" />
    <point x="2" y="0.03125" />
    <point x="3" y="0" />
    <point x="4" y="0" />
    ...
    <point x="254" y="0" />
    <point x="255" y="0" />
  </tolerance>
</histogram>

This visual verification approach was pioneered in the WPF test organization about 6 years ago by Marc Cauchy and Pierre-Jean Reissman.

SnapshotToleranceMapVerifier

However, none of the two approaches above work particularly well for evaluation of a typical application window, containing controls, text, etc. Such windows tend to have regions that need different tolerance settings. For example, consider the application window below:

Figure 2  Sample Application Window

If you try to perform master based visual verification, you will hit 2 issues:

1. The non-client area of the window (the window frame) will tend to be slightly different between different runs of the application. It will also depend on environment factors such as the desktop wall-paper.
   
2. Some regions of the client area of the window will also tend to exhibit significant variance between runs of the application.

Issue (1) is easy to resolve, using Snapshot.FromWindow(...) and excluding the non-client area from the capture. Issue (2), however, is a bit more involved. Here are the expected, actual and diff snapshots of the client-area of the application.

 Figure 3a  Expected Client-Area Snapshot

Figure 3b  Actual Client-Area Snapshot

  Figure 3c  Difference Snapshot

Figure 3d  Difference Snapshot – Completely Black Pixels Are Replaced With Pink

The differences between the expected snapshot and the actual snapshot are difficult to see on Figure 3c, so on Figure 3d I have replaced purely black pixels with pink.

It is not surprising that most of the variation occurs around the text regions in the application window (ClearType renders differently on different machines).  So it may make sense to increase the tolerance (or completely mask away) those regions, providing of course we are not specifically interested in their rendering.

In order to achieve that, we use the SnapshotToleranceMapVerifier class. Here’s an example:

// Take a snapshot, compare to the master image and generate a diff
Snapshot actual = Snapshot.FromWindow(hwndOfYourWindow, );
Snapshot expected = Snapshot.FromFile("Expected.png"));
Snapshot difference = actual.CompareTo(expected);

// Load the tolerance map. Then use it to verify the difference snapshot
Snapshot toleranceMap = Snapshot.FromFile("ExpectedImageToleranceMap.png");
SnapshotVerifier v = new SnapshotToleranceMapVerifier(toleranceMap);

if (v.Verify(difference) == VerificationResult.Fail)
{
    // Log failure, and save the actual and diff images for investigation
    actual.ToFile("Actual.png", ImageFormat.Png);
    difference.ToFile("Difference.png", ImageFormat.Png);
}

The tolerance map that we use looks as follows:

Figure 4  Tolerance Map in “ExpectedImageToleranceMap.png”

What appears pure black (0x00FFFFFF) is actually an off-black color (0x000A0A0A) to handle the small variations that appear as black dots on Figure 3d. Then we also have 4 regions with significantly higher tolerance to handle the variability of the text rendering.

In Conclusion

The visual verification API in TestApi provides a solid foundation for visual verification tests. As a general best practice, however, avoid visual verification as much as possible.

PhotoSuru – A Great WPF Sample Application

I am pleased to announce that our team just released PhotoSuru - a complete sample application demonstrating how to use the Syndicated Client Experiences Starter Kit for photo-viewing scenarios.

Big kudos to Kevin Gjerstad (our GPM) and the many other folks on the team who worked on this fantastic WPF application.

Check out photoSuru here and here. It will blow your mind :)

WPF Application Quality Guide v.0.4 Released!

I am happy to announce that we have just released the fourth preliminary version of the “WPF Application Quality Guide” with the following new sections:

• a new Data-Driven Testing section,
• an updated article on Globalization and Localization Testing,
• a new article on Stability and Stress Testing,
• a new article on Accessibility Testing,
• and a brief introduction on TestApi – the test API library we released a couple of weeks ago.

From here, we will be heading to releasing v1 in the first quarter of 2009. As always, your feedback is greatly appreciated.

The guide is available here. You can also download it as a Word document.

Introduction to TestApi – Part 2: Command-Line Parsing APIs

Command-line parsers remind me of linked lists in C++: everybody has written several at various points in their careers. While everybody should write each of those at least once, I doubt that many people out there are particularly excited about writing and re-writing fundamental data structures on a regular basis – it gets old very quickly. Not to mention that doing so is error-prone and decreases the maintainability of a world that’s already hard to maintain.

That’s why modern-day frameworks such as .NET provide standard implementations of the common data structures. And that’s why TestApi provides a reusable command-line parsing APIs via the CommandLineDictionary and CommandLineParser classes, the latter being a type-safe layer on top of the former. Obviously, these are not test APIs per se – they are general utility APIs that happen to be more often used when writing tests.

A few quick examples follow.

Simple Command-Line Parsing

As seen from the first example below, extracting command-line parameters that are primitives is easy. Primitive command-line parameters are either boolean (e.g. the “verbose” flag below), or a key-value pair, that one can extract with the indexer of the CommandLineDictionary instance (see the “testId” key below), just as one would expect from a Dictionary.

// 
// EXAMPLE #1: Parsing a command-line such as "RunTests.exe /verbose /testId=123"
// 

using System;
using Microsoft.Test;

public class Program 
{ 
    public static void Main(string[] args) 
    { 
        CommandLineDictionary d = new CommandLineDictionary(args);

        bool verbose = d.ContainsKey("verbose");
        int testId = Int32.Parse(d["testId"]);
                
        // use the parsed command-line parameters
    }
}

By default flags/keys are indicated with the forward slash (“/”) character and values are indicated with the equals character (“=”), but the user can override that upon initialization of of the CommandLineDictionary object:

// 
// EXAMPLE #1b: Parsing a command-line such as "RunTests.exe –verbose –testId:123"
// 

...
        CommandLineDictionary d = new CommandLineDictionary(args, '-', ':');
...

Finally, you one can use the ToString method to get a string representation of the command-line arguments.

Command-Line Argument Structures

Another common pattern when dealing with command-line arguments is populating a structure which contains all parsed arguments. The CommandLineParser class makes this easy:

// EXAMPLE #2:
// Sample for parsing the following command-line:
// Test.exe /verbose /runId=10
// This sample declares a class in which the strongly typed arguments are populated
public class CommandLineArguments
{
   bool? Verbose { get; set; }
   int? RunId { get; set; }
}

CommandLineArguments a = new CommandLineArguments();
CommandLineParser.ParseArguments(a, args);

Type-Safe Commands

A third common approach is forming strongly-typed commands from the command-line parameters. This is common for cases when the command-line looks as follows:

some-exe  COMMAND  parameters-to-the-command

The parsing in this case is a little bit more involved:

1. Create one class for every supported command, which derives from the Command abstract base class and implements an expected Execute method.

2. Pass an expected command along with the command-line arguments to CommandLineParser.ParseCommand – the method will return a  strongly-typed Command instance that can be Execute()-d.

// EXAMPLE #3:
// Sample for parsing the following command-line:
// Test.exe run /runId=10 /verbose 
// In this particular case we have an actual command on the command-line (“run”), 
// which we want to effectively de-serialize and execute.
public class RunCommand : Command
{
   bool? Verbose { get; set; }
   int? RunId { get; set; }

   public override void Execute()
   {
      // Implement your "run" execution logic here.
   }
}

Command c = new RunCommand();
CommandLineParser.ParseArguments(c, args); 
c.Execute();

Besides the parsing logic, CommandLineParser provides a few additional helper methods. One of them is CommandLineParser.PrintCommandUsage, which prints the usage for specific commands (or all supported commands) to the console.

In Conclusion

The command-line parsing APIs released with TestApi provide a simple and “layered” access to the command-line. Strictly speaking these APIs are not test APIs, but have nevertheless been included in TestApi as tests often have a need of parsing parameters on the command-line.