Ivo Manolov's Blog : Software Test Automation

Introduction to TestApi – Part 5: Managed Code Fault Injection APIs

ivom — Wed, 25 Nov 2009 08:47:00 GMT

Fault Injection is the act of artificially changing the behavior of an existing executable code to simulate various faults. FI is very useful for validation of error handling code paths and for improving code coverage.

There are several types of fault injection. In runtime fault injection, the fault injecting test modifies the execution logic of the application under test (AUT), by injecting faults, triggered by specific runtime conditions. One could for example implement a FI test with the following semantic:

Throw an out-of-memory (OOM) exception, whenever the application calls method CreateWidget of class WidgetManager.

The FI terminology is as follows:

AUT (application under test) – this is the tested application, in which faults are being injected;
Fault Rule – The fault rule is a central construct in an FI test that determines WHEN faults get triggered and WHAT TYPES of faults get triggered. A fault rule consists of:
- a Method Signature, determining the method where the fault will be injected;
- a Fault Condition, determining when the specific fault should be triggered (e.g. every N^th call)
- a Fault -- Determines the type of fault (e.g. throwing an exception, returning a specific value, etc.) that occurs when the fault condition is met.
Fault Session – The fault session is a collection of fault rules that are applied to a given AUT.

TestApi provides a simple, but powerful runtime fault injection API for injecting faults in managed code. The API was originally designed and implemented by Bill Liu et al from the “Essential Business Server” team, and adapted to TestApi by Sam Terilli from our WPF XAML team. The following content provides a quick introduction to the API.

Sample AUT

Following is the code of a trivial AUT that we will use for demonstration purposes:

//
// This is a sample application used for demonstration purposes.
//
using System;

class MyApplication
{
    static void Main(string[] args)
    {
        int a = 2;
        int b = 3;

        for (int i = 0; i < 10; i++)
        {
            Console.WriteLine("{0}) {1} + {2} = {3}", i, a, b, Sum(a, b));
        }
    }

    private static int Sum(int a, int b)
    {
        return a + b;
    }
}

The result of running this application is of course:

> MyApplication.exe 
0) 2 + 3 = 5 
1) 2 + 3 = 5 
2) 2 + 3 = 5 
3) 2 + 3 = 5 
4) 2 + 3 = 5 
5) 2 + 3 = 5 
6) 2 + 3 = 5 
7) 2 + 3 = 5 
8) 2 + 3 = 5 
9) 2 + 3 = 5

A Simple Fault Injection Test

Now, let’s try to inject a fault in the AUT. Let’s assume that we want to modify the return value of Sum. Here’s how we can accomplish that:

// 
// Simple fault injection test
//
using System;
using System.Diagnostics;
using Microsoft.Test.FaultInjection;

public class FaultInjectionTest 
{ 
    public static void Main() 
    { 
        // 
        // Set up a fault rule to return –1000 the second time Sum is called.
        //
        string method = "MyApplication.Sum(int,int)";
        ICondition condition = BuiltInConditions.TriggerOnNthCall(2);
        IFault fault = BuiltInFaults.ReturnValueFault(-1000);
        FaultRule rule = new FaultRule(method, condition, fault);            

        //
        // Establish a session, injecting the faults defined by the fault rule(s)
        //
        FaultSession session = new FaultSession(rule);
        ProcessStartInfo psi = session.GetProcessStartInfo(@".\MyApplication.exe"); 

        //
        // Launch the target process and observe faults
        //
        Process p = Process.Start(psi);        
        p.WaitForExit();
    }
}

Fairly straightforward. Upon running the test (which will itself spawn the AUT) we observe the following output:

> FaultInjectionTest.exe 0) 2 + 3 = 5 1) 2 + 3 = -1000 2) 2 + 3 = 5 3) 2 + 3 = 5 4) 2 + 3 = 5 5) 2 + 3 = 5 6) 2 + 3 = 5 7) 2 + 3 = 5 8) 2 + 3 = 5 9) 2 + 3 = 5

As intended, we injected a runtime fault in MyApplication.exe, which resulted in Sum returning –1000 the second time it got called.

Under The Covers

Under the covers, the managed code Fault Injection API uses the CLR profiling API to modify the prologue of the intercepted method at runtime in order to inject the desired fault. The injected prologue instructions essentially call a method in the library, which then dispatches the call to the specified fault.

Because faults are injected at runtime, the code of the original application is not modified in any way. There is a certain performance degradation, which depends on the number of the injected faults.

The fault injection API provides a variety of built-in conditions and faults (in the BuiltInConditions and BuiltInFaults classes respectively). Users of the API can also create custom conditions and faults (by implementing the ICondition and IFault interfaces respectively). The API also provides a set of classes that expose the ability to fine tune and monitor the injected faults.

In addition, the API provides a facility to set “global faults”, which is useful for server application testing, where one application typically consists of and recycles many different processes.

I have attached the sample above, which should get you up and running with fault injection.

TestApi v0.4 Released!

ivom — Sat, 21 Nov 2009 21:14:00 GMT

I am happy to announce that we have just released version 0.4 of TestApi – the testing API library - on Codeplex.

Version 0.4 contains the following additions and modifications:

An improved Combinatorial Variation Generation API with added support for parameter value weights and tags (for “negative” variations);
A new Memory Leak Detection API allowing capture, comparison and serialization / de-serialization of memory snapshots of a running process;
A new Object Comparison API allowing comparison of arbitrary .NET objects, using arbitrary object comparison strategies;
A new Text String Generation API, allowing generation of random strings, for testing purposes;
Various documentation and sample improvements.

We will be blogging about some of these API in the near future. As always, TestApi is available on http://codeplex.com/testapi.

TestApi Slide-Deck from the Patterns & Practices Summit

ivom — Thu, 15 Oct 2009 05:23:00 GMT

This week is Microsoft’s Patterns & Practices summit in Redmond . Earlier today I gave a 15-minute presentation on TestApi . Here is the slide-deck: Test Api View more presentations from ivom1234 ....(read more)

Introduction to TestApi – Part 4: Combinatorial Variation Generation APIs

ivom — Thu, 27 Aug 2009 03:31:00 GMT

The state and behavior of every software system depends on a number of parameters. These parameters can be both inputs as well as environmental factors.

One can think of software testing as a controlled experiment, where the system under test is observed upon varying of the parameters that affect it, in an attempt to discover unexpected system behavior.

Depending on who you talk to, the act of generation of these variations is called variation generation, matrix expansion, design of experiments (DOE), etc.

+++

The Problem

Modern-day software systems are complex and depend on many parameters. Expanding all possible combinations of all parameter values often results in a phenomenon called matrix explosion – having an overwhelmingly high number of test variations.

Matrix explosion is undesirable because:

It increases the runtime of your tests (which, among other things, makes TDD impractical and TDD is the single most effective way to ensure high product quality);
It increases the support costs of your tests – for a test pool of 100,000 tests, even with 99% pass rate (which in reality is hard to achieve), you still have to investigate 1,000 test failures. Investigating 1,000 test failures is always more expensive than investigating say 10 failures;
It is often impractical or impossible to cover all combinations, so you end up with partial test coverage (very often “vanilla” test coverage).

There are many ways to deal with matrix explosion, which depend on various system and/or experimental constraints. The theory and practice of DOE deals with that.

TestApi’s Solution

TestApi provides a generic API for combinatorial variation generation, using the algorithm presented in Jacek Czerwonka’s “Pairwise Testing in Real World” article. The API uses the following nomenclature:

Parameter – represents a single factor / variable and its values;
Constraint – represents a relationship between parameters, their values, constants, and other constraints;
Model – contains all parameters and constraints for the system, for which we are generating variations;
Variation – represents a tuple with a single value for every Parameter in the Model.

All of these are represented as correspondingly named types. Following are several examples demonstrating the use of the API.

Example 1 : Simple Matrix

For the purposes of a simple artificial example, consider having a system with the following parameters and values:

Parameter	Values
Color	White, Green, Red
Height	Short, Tall
Size	Small, Medium, Large

Assuming, there are no constraints, here is the code you would use to create a model

using System;
using System.Collections.Generic;
using Microsoft.Test.VariationGeneration;

class Example1
{
    static void Main(string[] args)
    {
        //
        // Declare all parameters and construct a model
        //
        Parameter p1 = new Parameter("Color") { "White", "Green", "Red" };
        Parameter p2 = new Parameter("Height") { "Short", "Tall" };
        Parameter p3 = new Parameter("Size") { "Small", "Medium", "Large" };

        Model m = new Model(new List<Parameter>
 { p1, p2, p3 } );

        //
        // Generate and print out all possible variations of the parameters in the model
        //
        foreach (Variation v in m.GenerateVariations(3, 1234))
        {
            System.Console.WriteLine("{0}\t{1}\t{2}", v[p1.Name], v[p2.Name], v[p3.Name]);
        }
    }
}

Executing the code below produces the following 3*2*3=18 variations, representing all possible combinations of the values of the three parameters:

> Example1.exe
White   Short   Small
White   Short   Medium
White   Short   Large
White   Tall    Small
White   Tall    Medium
White   Tall    Large
Green   Short   Small
Green   Short   Medium
Green   Short   Large
Green   Tall    Small
Green   Tall    Medium
Green   Tall    Large
Red     Short   Small
Red     Short   Medium
Red     Short   Large
Red     Tall    Small
Red     Tall    Medium
Red     Tall    Large

The variations are generated by the call to GenerateVariations(3, 1234). The number "1234" is the seed for the random generator, utilized by the algorithm. The number "3" is the order of the generated combinations. The order of generated combinations must be a number between 1 and the number of parameters in the model. The output for orders "1" and "2" are presented below:

Output for order "1":

White   Short   Small
Green   Tall    Medium
Red     Short   Large

Output for order "2":

White   Short   Small
White   Tall    Medium
White   Short   Large
Green   Tall    Small
Green   Short   Medium
Green   Tall    Large
Red     Short   Small
Red     Tall    Medium
Red     Tall    Large

Example 2 : Vacation Planner

This example (created by Nathan Anderson – our engineer who designed and implemented the combinatorial variation generation API) demonstrates the use of parameter constraints.

Parameter destination = new Parameter("Destination") { "Whistler", "Hawaii", "Las Vegas" };
Parameter hotelQuality = new Parameter("Hotel Quality") { 5, 4, 3, 2, 1 };
Parameter activity = new Parameter("Activity") { "gambling", "swimming", "shopping", "skiing" };

List<Parameter>
 parameters = new List<Parameter>
 { destination, hotelQuality, activity };
List<Constraint> constraints = new List<Constraint>
{
    new IfThenConstraint
    {
        If = destination.Equal("Whistler").Or(destination.Equal("Hawaii")),
        Then = activity.NotEqual("gambling")                        
    },
    new IfThenConstraint
    {
        If = destination.Equal("Las Vegas").Or(destination.Equal("Hawaii")),
        Then = activity.NotEqual("skiing")
    },
    new IfThenConstraint
    {
        If = destination.Equal("Whistler"),
        Then = activity.NotEqual("swimming")
    },
};

Model model = new Model(parameters, constraints);

//
// Call the method under test with each generated variation
//
foreach (var variation in model.GenerateVariations())
{
    CallVacationPlanner(
        (string)variation[destination.Name], 
        (int)variation[hotelQuality.Name], 
        (string)variation[activity.Name]);
}

Example 3 : The WPF Platform Matrix

This last example demonstrates how we deal with a real-world problem we face in the WPF team...

WPF must work reliably on all OS configurations, defined by the following matrix:

Parameter	Number of Values	Values
OS	6	XP SP2, Vista SP1, 7, Server 2003 SP2, Server 2008 SP1, Server 2008 R2
Language	25	(using 3-letter language abbreviations) ARA, CHS, CHT, CSY, DAN, DEU, ELL, ENG, ESN, FIN, FRA, HEB, HUN, ITA, JPN, KOR, NLD, NOR, PLK, PSE, PTB, PTG, RUS, SVE, TRK
System Locale	2	Same as OS language, TRK
Flavor	2	Free, Checked
Platform	3	x86, x64, x64 wow
IE version	3	OS default, IE7, IE8
High DPI	2	120 DPI, 96 DPI
Theme	6	Native, Classic, Luna, Royale, Classic High Contrast, Aero
Side-by-side	9	3.5 SP1 + 4 (3.5 tests), 3.5 SP1 + 4 (4 tests), 3.5 SP1 + 4 - 4 (3.5 tests), 4 + Mock 4.5 (4 tests), 4 + Mock 5 (4 tests), 4 + 3.5 SP1 (4 tests), 4 + 3.5 SP1 (3.5 tests), 4 + 3.5 SP1 - 4 (3.5 tests), 4 (4 tests)

Most of the parameters are self-descriptive. “Side-by-side” captures the .NET installation state. For example “3.5 SP1 + 4 - 4 (3.5 tests)” means “install .NET 3.5 SP1, install .NET 4, uninstall .NET 4, run tests built against 3.5 SP1”. This is done to confirm that there are no unexpected side effects as a result of the installation and un-installation of .NET 4.

The trivial full expansion of the matrix results in 583,200 combinations (=6*25*2*2...). Of course, some of these combinations (e.g. XP SP2 OS with a Aero theme) are not valid, but even after removing the invalid combinations, we still end up with a prohibitively large number of platform configurations to test on.

There are several ways to deal with this problem. One is identifying the so called equivalence classes. For example, from the point of view of Side-by-Side, Vista SP1 and Server 2008 SP1 can be regarded as equivalent OS-es and so on. Another popular approach is reducing regular testing to “vanilla configurations” (e.g. mostly ENG (English), 96-DPI configurations), venturing outside of the “vanilla domain” in accordance with a predefined schedule (e.g. during test passes at the end of major milestones). A third approach is using a pair-wise combinatorial variation generator, reducing the number of platform variations to about 230 – still a fairly high number for any real-world test pool, but clearly much better than the original number above.

In the WPF team, we use the third approach, combined with an adaptive random algorithm, which prioritizes testing on platform configurations that have not been tested on recently. The simplified code below demonstrates how to construct a model for platform config variation generation.

using System;
using System.Collections.Generic; 
using Microsoft.Test.CommandLineParsing; 
using Microsoft.Test.VariationGeneration;
using Microsoft.Test.VariationGeneration.Constraints;

public class OsVariationGeneration
{
    public static void Main(string[] args)
    {            
        CommandLineDictionary d = CommandLineDictionary.FromArguments(args);
        int order = Int32.Parse(d["order"]);
        int seed = Int32.Parse(d["seed"]);

        //
        //  Parameters
        //
        Parameter os = new Parameter("OS") { "Windows XP SP3", "Windows Vista SP1", "Windows 7", "Windows Server 2003 SP2", "Windows Server 2008 R2" };
        Parameter language = new Parameter("language") { "ARA", "CHS", "CHT", "CSY", "DAN", "DEU", "ELL", "ENG", "ESN", "FIN", "FRA", "HEB", "HUN", "ITA", "JPN", "KOR", "NLD", "NOR", "PLK", "PSE", "PTB", "PTG", "RUS", "SVE", "TRK" };
        Parameter sysLocale = new Parameter("sysLocale") { "SameAsOsLanguage", "TRK" };
        Parameter flavor  = new Parameter("flavor")     { "fre", "chk" };
        Parameter platform = new Parameter("platform")  { "x86", "x64", "x64wow" };
        Parameter ieVersion = new Parameter("ieVersion") { "osDefault", "ie7", "ie8" };
        Parameter highDpi = new Parameter("hiDpi") { "yes", "no" };
        Parameter theme = new Parameter("theme") { "Classic", "Luna", "Royale", "Classic High Contrast", "Aero Basic", "Aero Glass" };
        Parameter sxs = new Parameter("sxs") 
        {
            "3.5 SP1 + 4 (3.5 tests)",
            "3.5 SP1 + 4 (4 tests)",
            "3.5 SP1 + 4 - 4 (3.5 tests)",
            "4 + Mock 4.5 (4 tests)",
            "4 + Mock 5 (4 tests)",
            "4 + 3.5 SP1 (4 tests)",
            "4 + 3.5 SP1 (3.5 tests)",
            "4 + 3.5 SP1 - 4 (3.5 tests)",
            "4 (4 tests)",
        };
        List<Parameter>
 parameters = new List<Parameter>
 { os, language, sysLocale, flavor, platform, ieVersion, highDpi, theme, sxs };


        //
        //  Constraints
        //
        List<Constraint> constraints = new List<Constraint>
        {
            new IfThenConstraint
            {
                If = os.Equal("Windows XP SP3").Or(os.Equal("Windows Server 2003 SP2")),
                Then = theme.NotEqual("Aero Basic")
            },
            new IfThenConstraint
            {
                If = os.Equal("Windows XP SP3").Or(os.Equal("Windows Server 2003 SP2")),
                Then = theme.NotEqual("Aero Glass")
            },
            new IfThenConstraint
            {
                If = os.Equal("Windows 7").Or(os.Equal("Windows Server 2008 R2")),
                Then = theme.NotEqual("Luna")
            }
        };


        //
        //  Model
        //
        Model m = new Model(parameters, constraints);

        uint i = 0;
        foreach (Variation v in m.GenerateVariations(order, seed))
        {
            Console.WriteLine(
                "{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\t{7}\t{8}\t{9}", 
                i,
                v[os.Name],
                v[language.Name],
                v[sysLocale.Name],
                v[flavor.Name],
                v[platform.Name],
                v[ieVersion.Name],
                v[highDpi.Name],
                v[theme.Name],
                v[sxs.Name]
                );
            i++;
        }
    }
}

Conclusion

Pairwise variation generation is an important tool in your toolbox as a test author. TestApi provides a simple facility for combinatorial variation generation. We will of course be evolving this facility, but do let us know if you have specific scenarios or requirements you'd like to see supported.

TestApi v0.3 Released!

ivom — Fri, 24 Jul 2009 04:52:00 GMT

I am happy to announce that we have just released the third preliminary version of TestApi -- the testing API library. This version introduces some fairly significant changes and new features:

New name spaces (under Microsoft.Test) and new binary names;
New source code organization;
A fairly comprehensive acceptance tests suite (using xUnit);
Fault Injection APIs for managed code (using CLR profiler technology);
Combinatorial Variation Generation API (using the PICT algorithm);
A redesigned Application Control API;
Updates to the Visual Verification API;
Updated samples and documentation; new conceptual documents;

We will be blogging in detail about some of the features in v0.3 in the coming weeks.

TestApi is a policy-free API library, delivering Microsoft testing technology in an easy to use package. It can be used from within any tool (VSTS, NUnit, xUnit, etc.) or environment to automate the testing for desktop and in-browse applications on Windows.

TestApi v0.3 is available on http://testapi.codeplex.com. Thanks for all your feedback. Keep it coming! :)

WPF Application Quality Guide v0.5 Released!

ivom — Wed, 29 Apr 2009 06:58:00 GMT

I am happy to announce that we have released CTP 5 of the “WPF Application Quality Guide” – our single-stop document for WPF application and component developers and testers.

CTP 5 comes with the following new and updated content:

New content:
- “Considerations for WPF Browser Applications”
- “Integration and Scenario Testing”
- “XAML Editing Tools and Visual Studio Add-ins”
- “Building a WPF Application Test Suite by Using VSTS, NUnit, or xUnit”
Updated content:
- Reading roadmap
- “Performance and Scalability Testing”
- “TestApi”
- “Debugging Tools”
- “Performance Profiling Tools”
- “WPF Application Design and Development Tools”

The Guide is available here and here as a DOC file. Big kudos to Anne Gao on our team, who has been PM-ing all releases of the guide to date.

***

Those of you who have been following the Guide have seen it evolve from its first CTP (link) of 20 or so pages to its present size of more than 80 pages.

Any 80-page document is tricky to grok, so we have been thinking of reorganizing the content and potentially splitting the content into two documents – one on client application development and testing fundamentals and one on WPF specifics. We prefer to do any edits based on user feedback, so please let us know of the current organization and content of the document and share your ideas on how we could improve both.

TestApi v.0.2 Released!

ivom — Fri, 24 Apr 2009 15:53:52 GMT

I am excited to announce that we have just released the second preliminary version of TestApi – the testing API library – on http://codeplex.com/testapi!

The v.0.2 package includes the following additions and modifications:

Improved command-line parsing APIs
Improved visual verification APIs
- A new tolerance map visual verifier in SnapshotToleranceMapVerifier
- New operations on snapshot (And and Or) allowing you to mask
- A new Snapshot.FromWindow(...) constructor with ability to include and exclude the window chrome in snapshots.
- Improved visual verification internals
New application control APIs, enabling in-proc and out-of-proc automation of client applications
Expanded conceptual documentation
CHM API documents (in addition to the HTML documents)
Addition of NUnit and xUnit usage samples (for all of the non-VS crowd out there)

Check out the package and let us know of any feedback.

Introduction to TestApi – Part 2: Command-Line Parsing APIs

ivom — Thu, 18 Dec 2008 01:15:00 GMT

Command-line parsers remind me of linked lists in C++: everybody has written several at various points in their careers. While everybody should write each of those at least once, I doubt that many people out there are particularly excited about writing and re-writing fundamental data structures on a regular basis – it gets old very quickly. Not to mention that doing so is error-prone and decreases the maintainability of a world that’s already hard to maintain.

That’s why modern-day frameworks such as .NET provide standard implementations of the common data structures. And that’s why TestApi provides a reusable command-line parsing APIs via the CommandLineDictionary and CommandLineParser classes, the latter being a type-safe layer on top of the former. Obviously, these are not test APIs per se – they are general utility APIs that happen to be more often used when writing tests.

A few quick examples follow.

Simple Command-Line Parsing

As seen from the first example below, extracting command-line parameters that are primitives is easy. Primitive command-line parameters are either boolean (e.g. the “verbose” flag below), or a key-value pair, that one can extract with the indexer of the CommandLineDictionary instance (see the “testId” key below), just as one would expect from a Dictionary.

// 
// EXAMPLE #1: Parsing a command-line such as "RunTests.exe /verbose /testId=123"
// 

using System;
using Microsoft.Test;

public class Program 
{ 
    public static void Main(string[] args) 
    { 
        CommandLineDictionary d = new CommandLineDictionary(args);

        bool verbose = d.ContainsKey("verbose");
        int testId = Int32.Parse(d["testId"]);
                
        // use the parsed command-line parameters
    }
}

By default flags/keys are indicated with the forward slash (“/”) character and values are indicated with the equals character (“=”), but the user can override that upon initialization of of the CommandLineDictionary object:

// 
// EXAMPLE #1b: Parsing a command-line such as "RunTests.exe –verbose –testId:123"
// 

...
        CommandLineDictionary d = new CommandLineDictionary(args, '-', ':');
...

Finally, you one can use the ToString method to get a string representation of the command-line arguments.

Command-Line Argument Structures

Another common pattern when dealing with command-line arguments is populating a structure which contains all parsed arguments. The CommandLineParser class makes this easy:

// EXAMPLE #2:
// Sample for parsing the following command-line:
// Test.exe /verbose /runId=10
// This sample declares a class in which the strongly typed arguments are populated
public class CommandLineArguments
{
   bool? Verbose { get; set; }
   int? RunId { get; set; }
}

CommandLineArguments a = new CommandLineArguments();
CommandLineParser.ParseArguments(a, args);

Type-Safe Commands

A third common approach is forming strongly-typed commands from the command-line parameters. This is common for cases when the command-line looks as follows:

some-exe COMMAND parameters-to-the-command

The parsing in this case is a little bit more involved:

Create one class for every supported command, which derives from the Command abstract base class and implements an expected Execute method.
Pass an expected command along with the command-line arguments to CommandLineParser.ParseCommand – the method will return a strongly-typed Command instance that can be Execute()-d.

// EXAMPLE #3:
// Sample for parsing the following command-line:
// Test.exe run /runId=10 /verbose 
// In this particular case we have an actual command on the command-line (“run”), 
// which we want to effectively de-serialize and execute.
public class RunCommand : Command
{
   bool? Verbose { get; set; }
   int? RunId { get; set; }

   public override void Execute()
   {
      // Implement your "run" execution logic here.
   }
}

Command c = new RunCommand();
CommandLineParser.ParseArguments(c, args); 
c.Execute();

Besides the parsing logic, CommandLineParser provides a few additional helper methods. One of them is CommandLineParser.PrintCommandUsage, which prints the usage for specific commands (or all supported commands) to the console.

In Conclusion

The command-line parsing APIs released with TestApi provide a simple and “layered” access to the command-line. Strictly speaking these APIs are not test APIs, but have nevertheless been included in TestApi as tests often have a need of parsing parameters on the command-line.

Introduction to TestApi – Part 1: Input Injection APIs

ivom — Tue, 16 Dec 2008 06:49:00 GMT

I am starting a series of posts introducing some of the facilities available in TestApi 0.1, a test and utility API library, which we recently released on CodePlex. Most of this content is already available in the documentation provided with the library.

The first post is on input injection – a fairly common activity in UI testing.

General Notes

Input injection is the act of simulating user input. In general, there are several ways to simulate user input, in the following progressively increasing levels of realism:

Direct method invocation: A test programmatically triggers events by directly calling methods on the target UI element. For example, a test can call the Button.IsPressed method to simulate pressing a WPF button.
Invocation using an accessibility interface (UIA, MSAA, etc.): A test programmatically triggers events by calling methods on an AutomationElement instance that represents the target UI element.
Simulation using low-level input: A test simulates input by using low-level input facilities provided by the host operating system. Examples of such facilities on Windows are the SendInput Win32 API and the Raw Input Win32 API, which inject input directly into the OS input stream.
Simulation using a device driver: A test uses a device driver to simulate input at the device-driver level.
Simulation using a robot: A test controls a robot to simulate direct human interaction with an input device (for example, pressing keys on a keyboard).

Technique A is framework-specific; what works for WPF does not work for Windows Forms and vice versa. Technique B is less framework-specific than A, but still has limitations, because some frameworks differ in their implementations of the required accessibility interfaces. Techniques C and D are OS-specific. Technique D is significantly more difficult to implement and deploy than C, without a corresponding increase in its level of realism. Technique E is universal, albeit much slower and much more expensive than the other options.

The TestApi library provides facilities both for B (through the AutomationUtilities class) and for C (through the Mouse and Keyboard classes), which are the most generally useful techniques of input simulation.

Examples

The AutomationUtilities class provides wrappers for common UIA operations, such as discovery of UI elements. The first example below demonstrates how to discover and click a WPF Button in a WPF Window, by using the AutomationUtilities class and the Mouse class.

//
// EXAMPLE #1
// This code below discovers and clicks the Close button in an About dialog box, thus
// dismissing the About dialog box.
//

string aboutDialogName = “About”;
string closeButtonName = “Close”;

AutomationElementCollection aboutDialogs = AutomationUtilities.FindElementsByName(
    AutomationElement.RootElement,
    aboutDialogName);

AutomationElementCollection closeButtons = AutomationUtilities.FindElementsByName(
    aboutDialogs[0],
    closeButtonName);

//
// You can either invoke the discovered control, through its invoke pattern...
//

InvokePattern p = 
    closeButtons[0].GetCurrentPattern(InvokePattern.Pattern) as InvokePattern;
p.Invoke();

//
// ... or you can handle the Mouse directly and click on the control.
//

Mouse.MoveTo(closeButton.GetClickablePoint());
Mouse.Click();

The second example below demonstrates how to discover a TextBox instance and type in it, using the Mouse and Keyboard classes as wrappers of common mouse and keyboard operations:

//
// EXAMPLE #2
// Discover the location of a TextBox with a given name.
//

string textboxName = “ssnInputField”;

AutomationElement textBox = AutomationUtilities.FindElementsByName(
    AutomationElement.RootElement,
    textboxName)[0];

Point textboxLocation = textbox.GetClickablePoint();

//
// Move the mouse to the textbox, click, then type something
//

Mouse.MoveTo(textboxLocation);
Mouse.Click();

Keyboard.Type(“Hello world. ”);
Keyboard.Press(Key.Shift);
Keyboard.Type(“hello, capitalized world.”);
Keyboard.Release(Key.Shift);

In Conclusion

The Mouse and Keyboard classes in the TestApi library can be used for automating of any application, running on Windows. The classes are completely policy- and context-free – their usage is not dependent on a specific test framework or on a specific test workflow. TestApi provides full source code and XML documentation for these classes, so you can either integrate them in your own projects or reference the pre-built DLLs.

Note that, even though Mouse, Keyboard and AutomationUtilities make the life of the test automation developer quite a bit easier, UI testing is tricky and should be avoided whenever possible. It’s always preferable to design your application as a multi-tier application, with a “thin” UI layer, so that you can bypass the UI in most of your tests.