Test Guide : Automation Stack

Automate This!

micahel — Wed, 11 Apr 2007 19:30:00 GMT

How much of your testing do you automate? How do you know whether you have automated enough - or too much?

My current team is taking the Automate Everything approach. This means we automate every last test case. "110% automation", as one of our executives liked to say. While certain tests may remain unautomated due to time constraints or extreme technical difficulties, we are working hard to keep that number lower than humanly possible.

Automating Everything has considerable payoffs: every test can be run on every build, ensuring our application does not regress in any way we have encountered thus far. Supporting additional operating systems (oh look - now there's a Windows XP Service Pack 42!) and languages (flash from Marketing: now we are localizing into Lower Elbonian!) requires nothing more than adding another row to the configuration matrix and kicking off another test run. Hot fixes can be tested without subjecting the test team to a fire drill. Pretty much nirvana, right?

Automating Everything has considerable downsides as well: Automated tests are by nature scripted, not exploratory. Even with an automation stack which injects all sorts of variability, the tests wear grooves in those areas of the product they cover and they ignore everything else. When something unexpected happens they are likely to die, and even if they are able to recover they are not able to stop what they were doing and investigate that unexpected happening. And don't forget the maintenance required to keep those tests running - which efforts are not helping you find defects in your application. Say, have you had time to actually use your application yet?

On the other extreme is the Automate Nothing approach. Here every test case is executed manually by a person physically using their mouse and keyboard. This has considerable payoffs: every test can be exploratory. The entire surface of the product will likely be covered. When something unexpected happens it is easily followed up. No maintenance is required to keep the test cases up to date with changes in the application. Everybody is always using the application. Pretty much nirvana, right?

Automating Nothing has considerable downsides as well: It is unlikely that every test will be run on every build (unless you only get builds every two weeks - in which case you have my sympathies!), so regressions may not be found until long after they are introduced, if they are found at all. Supporting an additional configuration means either running another full test pass or scoping down your testing and hoping you do not miss anything important - no economy of scale benefits here! Every hot fix requires yet another full test pass. Not to mention that it can be difficult for people to stay Brain Engaged when running a test for the tenth or twentieth or two hundredth time.

I struggle. The benefits of automating are clear to me. So are the downsides. Some tests - or parts of tests - are eminently automatable. Other tests are tedious or boring to do manually. Automated tests lend themselves to spitting out data in pretty graphs, which management generally likes. Session-Based Test Management seems an effective way to leverage testers' exploratory and critical thinking skills - to keep them Brain Engaged - while also giving management the data they require. I wonder however whether it scales to my context.

It is clear to me that Automating Everything is taking things too far. So is Automating Nothing. I have not yet found a balance I like. How about you?

Tester Mentality

micahel — Wed, 18 Oct 2006 23:17:00 GMT

I tech screened a tester this morning for a position on my team. I always ask the candidate to describe the tester mentality. I loved today's interviewee's answer:

More than the product working perfectly without any crash assert, if it is not intuitive and user friendly it is useless. What will the customer use it for? Does it provide value for the scenarios the customer cares about? Think of the variety of customers - not everybody will have a top-of-the-line machine, or the same technical expertise, or the same expectation of the product. More than just breaking software, the tester should make sure it has the capabilities the customer needs.

Right on! This is what our automation stack is all about - keeping that customer focus.

If you aren't thinking about your customer, keeping them in mind as you test, then you aren't testng.

Fantasy Island?

micahel — Wed, 06 Sep 2006 19:30:00 GMT

Recently one of my teammates was showing off a WPF application he had built. Since it's easy to do, she built a Logical Functional Model and used it to script her demo.

One of the cool things her app supported was reskinning (including rearranging and relocating the various widgets), and at various points during the demo she switched to yet another skin. Of course the LFM didn't care one whit about these UI changes and so the demo continued without pause.

Most of the audience didn't catch the significance of this. I did however see one PM nudge his neighbor excitedly, remarking "The UI just completely changed! And the test didn't even notice!"

This is the power of our automation stack. When you write tests in terms of user actions rather than UI, you can change that UI in radical ways, and as long as the application semantics don't change it all just works - you don't need to touch a single test case.

This may seem like a fantasy world but my team is living in it today. Other teams within Microsoft are starting to live there too, as are companies around the world. What's stopping you?

Dr. Dobb's + Me

micahel — Tue, 11 Oct 2005 20:50:00 GMT

Dr. Dobb's published my article "Achieve More Comprehensive Verification With Less Work" online - check it out!

The LFM Applied

micahel — Wed, 27 Jul 2005 18:15:00 GMT

Jerrad is writing a Logical Functional Model for a website. He is approaching the problem exactly right: figure out what the user actions are, and then organize them into a user-sane order. His example is all about searching, so the LFM could be very small, just a few different FindXxx methods. He has so many different ways to search, though, that his approach of separating the different types of searches makes a lot of sense as well. Deciding how to partition your LFM is a core issue that can mean the difference between the LFM being understandable or a jumbled mess. My philosophy here mirrors what I do everywhere else: start simple and expand into more detail as needed. For example, in a drawing program like Microsoft Visio there are a lot of actions that can be done with shapes on the active page. I would initially put all of these into Logical.Page.Shapes, but when it eventually grows so large that understandability and discoverability start to suffer I would split it out into e.g., Logical.Page.Shapes.Creation and Logical.Page.Shapes.Formatting and Logical.Page.Shapes.Editing.

Jerrad asks how to write an LFM for an application that has multiple user roles. For each of his features, a user can take a variety of actions and an administrator can configure a variety of things. Both the user and the administrator have to be authenticated before they can do anything. Different multi-role applications work in very different ways (e.g., in some everyone starts out a user and then authenticate up to admin but in others people log in as one or the other (and presumably have different UIs); in some when a user authenticates up to admin the administrator features tack on to the user features but in others the administrator role effectively replaces the user role), but the user actions in each case are very similar:

Authenticate as a user.
Authenticate as an administrator.
Execute user actions.
Execute admin actions.

Jerrad wonders whether partitioning actions based on role is the right thing to do:

Application

Login(UserRole, ...)
Logout

User

DoSomething
DoSomethingElse

Administrator

ConfigureSomething
ConfigureSomethingElse

This could work, especially if there a huge number of actions in both the user and administrator branches. In my mind the most important thing is what is being done, not who is doing it, though, so I would probably start with a strictly functional view though:

Application

Login(UserRole, ...)
Logout

FunctionA

ActionA
ActionB

FunctionB

ActionA
ActionB

Configuration

ConfigureSomething
ConfigureSomethingElse

Application

Login(UserRole, ...)
Logout

FunctionA

Execution

ActionA
ActionB

Configuration

ConfigureSomething
ConfigureSomethingElse

FunctionB

Execution

ActionA
ActionB

Configuration

ConfigureSomething
ConfigureSomethingElse

The most important thing here is not whether you have the "right" model but rather that you spend time discussing it with your team and develop something that makes sense to everyone. Not very prescriptive advice, I know! <g/> But staying focused on actions, and keeping away from a scripting/VBA object model, is the main point. One practice I have found helpful is to compare the test cases you could write with your LFM to the steps listed in your help documentation. Test cases that sound like help steps are a Very Good Thing!

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: michhu at microsoft dot com. Great coding skills required.

Verily, 'Tis Truth

micahel — Wed, 20 Jul 2005 20:30:00 GMT

Your Logical Functional Model lets you write test cases from your user's point of view, test cases that focus on what is being done rather than how it is being done. Your Physical Object Model lets your test cases ignore the details behind interacting with your user interface. Execution Behaviors let you push into your test automation stack decisions about which of the plethora of ways each user action can be executed. Data Manager does the same for the values your test cases use. Together these let your test cases be straightforward and clear and simple to write. Just one thing is lacking: verification.

Loosely Coupled Comprehensive Verification lets you verify everything you care about all the time. The first step is to notify Verification Manager every time an action starts or finishes:

public static void Logical.Question.SelectAnswer(string answerId)
{
Verification.ActionStarting();

Logical.MainWindow.CurrentQuestion.GetAnswer(answerId).Invoke();

Verification.ActionEnding();
}

That's it as far as the LFM is concerned! You see why we call it Loosely Coupled: the only point at which verification is coupled to anything else is here, where the LFM tells the verification system when things happen.

Things are rather more interesting on the verification side of things.

public static void VerificationManager.ActionStarting()
{
    string executingActionsName = this.GetNameOfCaller();
    object[] executingActionsArguments = this.GetArgumentsForCaller();
    this.ExpectedState = this.SnapshotCurrentState();
    this.NotifyExpectedStateGenerators(executingActionsName, executingActionsArguments, this.ExpectedState);
}

When Verification Manager is notified that an action is starting, Verification Manager determines the name of and arguments for the action being called. (In .Net, the name is simple to determine via Reflection. Due to security reasons, however, the arguments are impossible to determine, so they must be retrieved through some other mechanism.) Next it baselines its state to the current application state. Finally it passes this information on to its Expected State Generators.

private static void VerificationManager.NotifyExpectedStateGenerators(
string executingActionsName, object[] executingActionsArguments, State expectedState)
{
// Magic happens here!
}

Notifying the Expected State Generators can be done any number of different ways. Your options include:

Switch on the action name and then notify a hard-coded set of Expected State Generators. You will have to make modifications any time actions or Generators are added, but if both sets are mostly stable the simple implementation may be worth the occasional edit.
Pass every event on to every Generator; each Generator then has to determine whether to respond to or ignore each event. This keeps knowledge of which events a particular Generator handles isolated within the Generator, but notifying every Generator every action will have a runtime cost.
Have each Generator register for the events it is interested in. This gives the Generators the most responsibility for managing themselves while ensuring you notify the minimum set of events, but you do have to manage all those registrations.

When an Expected State Generator is told that an action is starting, it does whatever it does to decide how the application state should change and then it updates the expected state to record that information:

internal void OnSelectAnswer(object[] executingActionsArguments, State expectedState)
{
    foreach (State.Answer answer in expectedState.Questions[expectedState.CurrentQuestion].Answers)
    {
        answer.IsSelected = false;
    }

expectedState.Questions[expectedState.CurrentQuestion].Answers[executingActionsArguments[0]].IsSelected = true;
}

In the case of selecting an answer, I mark every answer as not selected and then go back and mark the selected answer as selected.

After every Expected State Generator has done its thing, control returns back to the LFM which then goes about executing the action. When execution is complete Verification Manager goes back into action:

public static void VerificationManager.ActionEnding()
{
State actualState = this.SnapshotCurrentState();
this.ExpectedState.CompareTo(actualState);
}

This is the easy part: compare the expected state to current state and log any differences as failures.

A common question about this scheme is performance. "Don't all those snapshots slow things down?" "Won't all those calculations take too long?" Our experience has been that they don't. However, while the snapshots must of course occur in realtime, both generating expected state and comparing that to actual state can occur at some time after the test has completed. This change minimizes the runtime cost while still allowing you to verify everything (or as much of everything as you care about) all the time.

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: michhu at microsoft dot com. Great coding skills required.

No Guts, But Lots Of Glory

micahel — Wed, 13 Jul 2005 19:30:00 GMT

My previous post hinted that the Physical Object Model takes advantage of some underlying magic that makes different types of controls look similar. This "smudging" of controls into similarity comes courtesy of the controls abstraction layer.

While each different type of control is unique (else we wouldn't need so many of them) in the main they support a much smaller set of actions. Although you select a particular radio button but check or uncheck a text box but type text into a text box, you could also say that each case simply sets the control's value. (UIAutomation formalizes many of these patterns; our patterns do not exactly match up to UIAutomation's patterns however.) Thus users of the POM do not have to know the exact type of a control but rather simply that it can be invoked, or its value set, or whatever. This prevents many user interface changes from rippling out beyond the POM - while the specific makeup of a UI is likely to change multiple times, its semantics tend to be much more stable.

The controls abstraction layer also hides the method used to interact with a particular control. Consider a button: it can be clicked with the mouse, or the keyboard can be used to give it focus and then invoke it, or it can be virtually invoked via accessibility APIs. The controls abstraction layer wraps these details into control-specific providers that can be swapped in and out to change how a specific action is executed. Automatically varying execution method is of course exactly what Execution Behaviors are all about, so we Execution Behavior this swapping and take advantage of all the goodness that brings to the table.

Here's what all that theory looks like in code:

public class UICheckBox : UIControl<CheckBox>, IValueSettable<bool>
{
internal UICheckBox(UIObject knownUIObject) : base(new Avalon.CheckBox(knownUIObject)) { }

    public void SetValue(bool value)
    {
        Providers.ChooseNewProvider();
        if (value)
        {
            this.Check();
        }
        else
        {
            this.Uncheck();
        }
    }

public bool IsChecked { get { return (this.CheckState != ToggleState.Off); } }

    public ToggleState CheckState
    {
        get { return Providers.CheckBoxProvider.CheckState; }
        set
        {
            Providers.ChooseNewProvider();
            Providers.CheckBoxProvider.CheckState = value;
            DelayUtility.WaitForIdle();
        }
    }

    public void Check()
    {
        Providers.ChooseNewProvider();
        Providers.CheckBoxProvider.Check();
        DelayUtility.WaitForIdle();
    }

    public void Uncheck()
    {
        Providers.ChooseNewProvider();
        Providers.CheckBoxProvider.Uncheck();
        DelayUtility.WaitForIdle();
    }

    public void Toggle()
    {
        Providers.ChooseNewProvider();
        Providers.CheckBoxProvider.Toggle();
        DelayUtility.WaitForIdle();
    }
}

A check box supports the generic Value Settable pattern, so UICheckBox implements SetValue. It can occasionally be useful to interact with a control outside the bounds of the generic patterns, so UICheckBox provides check box-specific methods like Check and Uncheck as well. The implementation of each these methods simply delegates most of the work to whatever Check Box Provider the global Providers registry happens to have on hand at the time.

The providers are Execution Behvaiored by calling Providers.ChooseNewProvider at the start of every method. ChooseNewProvider is actually a Composite Execution Behavior that chooses and activates one of the sets of providers (i.e., mouse, keyboard, shortcut keys, accessibility APIs). Thus each successive POM call may execute using any of these options.

The controls abstraction layer is the last piece of the core automation stack. By combining it with the Physical Object Model and Logical Functional Model, and integrating Execution Behvaiors throughout, scripted test cases - which typically only ever do exactly the same thing - can be more than just regression tests. Rather than explicitly stating exactly what to do and how to do it they can instead give general directions. Each time a test case runs it is likely to take a different path than it has previously - and so it is likely to find new bugs. This lets scripted test cases start earning their keep!

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: michhu at microsoft dot com. Great coding skills required.

A Peek Behind The Curtains

micahel — Wed, 06 Jul 2005 20:32:00 GMT

With the LFM defined and hooked up to Execution Behaviors it makes sense to move on to defining the Physical Object Model.

The POM is often the simplest part because it's just a straightforward API around the user interface. Given Surveyor's UI:

a likely POM would be:

public class UIApplicationWindow : UIWindow
{
internal UIApplicationWindow(UIObject applicationWindow)
: base(new Window(applicationWindow)) { }

    public UIQuestionSetButton SwitchToNotYetAnsweredQuestions { get { } }
    public UIQuestionSetButton SwitchToSkippedQuestions { get { } }
    public UIQuestionSetButton SwitchToAnsweredQuestions { get { } }

public UIQuestion CurrentQuestion { get { } }

    public UIButton MoveToPreviousQuestion { get { } }
    public UIButton MoveToNextQuestion { get { } }
    public UIButton FinishSurvey { get { } }
}
Although the POM is a very thin wrapper around the UI, omitting UI elements used only for layout can vastly simplify the POM. For example, Surveyor's UI uses stack panels and assorted other layout mechanisms to format its user interface, but none of these are important for execution or verification and so the POM does not mention them.

Much of the POM uses predefined classes that provide standard behavior (such as UIWindow, which has Close, Move, and Resize methods), but many applications require some custom controls as well. Custom controls that merely add functionality to a standard control are straightforward to handle: simply derive from that standard control and add the necessary behaviors. Surveyor's question set navigation buttons (the ones that allow the user to switch between those questions they haven't yet answered, those they've skipped, and those they've answered) are nothing more than styled radio buttons, so UIQuestionSetButton class could be defined like so:
public class UIQuestionSetButton : UIRadioButton
{
internal UIQuestionSetButton(UIObject theQuestionSetButton)
: base(theQuestionSetButton) { }

public int QuestionCount { get { } }
}
where UIRadioButton provides the expected ability to select itself and to report whether it is selected.

Other cases that involve a truly custom control are handled by deriving from (wait for it...) UICustomControl. An example of this in Surveyor is the current question:
public class UIQuestion : UICustomControl
{
internal UIQuestion(UIObject theQuestion)
: base(theQuestion) { }

public string Question { get { } }

public UIRadioButton GetAnswer(string name) { get { } }

public UIButton ClearAnswer { get { } }
public UIButton SkipQuestion { get { } }
}
Although the UI does not clearly separate it as such, a question has multiple parts: the text of the question, the set of possible answers, the button to clear the selected answer, and the button to skip the question. Grouping those related concepts into a single object helps make the POM easier to work with and understand (for me, at least).

In the main, implementing the individual properties is trivial. UIQuestion.Question's getter is just:
public string Question
{
    get
    {
        UICustomControl value = this.FindDescendant<UICustomControl>("Question");
        return value.Name;
    }
}
Its ClearAnswer getter is even simpler:
public UIButton ClearAnswer
{
    get
    {
        return this.FindDescendant<UIRadioButton>("ClearAnswer");
    }
}
FindDescendant wraps the tree walk services that Avalon and UIAutomation provide. Generics (new in version two of the .Net Framework) takes care of casting the generic object those services return to the specific type of control we're looking for. (And yes, I normally pull those inlined constants out to named constants!)

To use the POM, simply dot your way down the tree:
// Logical.Application.MainWindow returns a UIApplicationWindow.
UIRadioButton answer = Logical.Application.MainWindow.CurrentQuestion.GetAnswer(answerId);
answer.Invoke();
Logical.VerificationLog.VerifyTrue(answer.IsSelected, "Answer " + answer.Label + " is selected");

As I said, straightforward stuff!

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: michhu at microsoft dot com. Great coding skills required.

Who Ya Gonna Call?

micahel — Wed, 29 Jun 2005 20:17:00 GMT

Once you create your Logical Functional Model the other pieces can follow in any order immediately as chunks of your feature and LFM are defined. I'll follow the most typical pattern here, which brings us to Execution Behaviors.

When you first define your LFM you focus on what the user is doing not how they are doing it. The next step - defining Execution Behaviors - starts to drill down into the how. For each user action you defined in your LFM, determine every possible way the user can invoke that action. For example, every action in Surveyor can be invoked by both the mouse and the keyboard; to indicate they are done with the survey (to nest examples), the user can:

click the Done button with their mouse
press the Tab key until the Done button is highlighted and then press the spacebar to invoke it

These two possibilities are pretty much it in Surveyor, but the more complicated your application the more execution variants you are likely to have. (To wit: how many different ways can you create a new document in your word processor?)

After you have identified the various Execution Behaviors for an action you organize them in two ways:

Into semantically equivalent groups. It is vital that each Execution Behavior in a group has exactly the same end result as each other Execution Behavior in the group. Remember that the whole point of Execution Behaviors is that the test case doesn't care which one will be selected; if two Execution Behaviors have different results which one is selected obviously will matter.
Into an execution method graph. This graph is tiny for Surveyor - simply two sibling leaf nodes:
- using the keyboard
- using the mouse
The example of creating a new document is slightly more complex:
- using the menu
  - navigating the menu using the keyboard
  - navigating the menu using the mouse
- using the toolbar
  - using the mouse
- using the object model

Each of these groups translates directly to a Composite Execution Behavior / child Execution Behavior relationship:

Each semantically equivalent group requires a separate top-level Composite Execution Behavior.
Each branch in the execution method graph defines a Composite Execution Behavior; each child of a branch defines its child Execution Behaviors.

Now you're ready to turn your LFM stubs into Composite Execution Behaviors and stub out the corresponding child Execution Behaviors. Surveyor's Logical.Navigation.IndicateAmDone method

public static void IndicateAmDone() { }

becomes

[CompositeExecutionBehavior("IndicateAmDoneUsingMouse", "IndicateAmDoneUsingKeyboard")]
public static void IndicateAmDone()
{
Delegate chosenChild = ExecutionBehaviorManager.ChooseAny(Logical.Navigation.IndicateAmDone);
chosenChild();
}

public static void IndicateAmDoneUsingMouse() { }
public static void IndicateAmDoneUsingKeyboard() { }

Execution Behaviors for creating a new document might code up thusly:
[CompositeExecutionBehavior("CreateNewDocumentUsingMenu", "CreateNewDocumentUsingToolbar", "CreateNewDocumentUsingObjectModel")]
public static void CreateNewDocument()
{
Delegate chosenChild = ExecutionBehaviorManager.ChooseAny(Logical.Documents.CreateNewDocument);
chosenChild();
}

[CompositeExecutionBehavior("CreateNewDocumentUsingMenuUsingKeyboard", "CreateNewDocumentUsingMenuUsingMouse")]
public static void CreateNewDocumentUsingMenu()
{
Delegate chosenChild = ExecutionBehaviorManager.ChooseAny(Logical.Documents.CreateNewDocumentUsingMenu);
chosenChild();
}
public static void CreateNewDocumentUsingMenuUsingKeyboard() { }
public static void CreateNewDocumentUsingMenuUsingMouse() { }

[CompositeExecutionBehavior("CreateNewDocumentUsingToolbarUsingMouse")]
public static void CreateNewDocumentUsingToolbar()
{
Delegate chosenChild = ExecutionBehaviorManager.ChooseAny(Logical.Documents.CreateNewDocumentUsingToolbar);
chosenChild();
}
public static void CreateNewDocumentUsingToolbarUsingMouse() { }

public static void CreateNewDocumentUsingObjectModel() { }

Test cases now have a variety of options when they need to create a new document:

If they don't at all care how the action is invoked, they simply call the top level Composite Execution Behavior (i.e., CreateNewDocument).
If they partially care how the action is invoked, they call one of the intermediate Composite Execution Behaviors (e.g., CreateNewDocumentUsingMenu).
If the completely care how the action is invoked, they call the corresponding child Execution Behavior (e.g., CreateNewDocumentUsingMenuUsingMouse).

This is why I made the "using toolbar" variant a Composite even though it only has a single child - doing so allows test cases to state whether they care how the toolbar button is invoked.

With Execution Behaviors integrated into your LFM you get automatic variation over the full set of execution methods, which lets you write fewer test cases (since you no longer need an explicit test case for each different implementation). Individual test cases however have the flexibility to use a specific implementation, so regression tests and such can still be written against the LFM. This is one case where you really can have it both ways!

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: michhu at microsoft dot com. Great coding skills required.

Use Your Users' Viewpoint

micahel — Wed, 22 Jun 2005 20:50:00 GMT

As soon as you have even a vague idea about what your feature is all about you can start defining your Logical Functional Model. Ideally your entire feature team will take part in this exercise, but you can certainly do it on your own as well. Ask yourself "What are the user features here?" Don't think about UI or implementation but rather try to see your app through the user's eyes.

For example, I see the user actions in Surveyor as:

Start and stop the app
Select an answer for the current question
Clear the selected answer for the current question
Move to the next or previous question
Indicate that the user is done with the survey
Skip the current question
Switch between the various question navigation sets

Additional user actions may become available after the user completes the survey, but as Surveyor itself is not concerned with that its LFM won't cover any of the myriad possibilities.

Next organize these actions however makes sense to you and your feature team. This organization is mostly just an aid to navigating the LFM, but if you group things similarly to how your users might your LFM will be more intuitive. For example:

Application
- Start and stop the app
Navigation
- Switch between the various question navigation sets
- Move to the next or previous question
- Indicate that the user is done with the survey
Question
- Select an answer for the current question
- Clear the selected answer for the current question
- Skip the current question

Next convert this to code:

public static class Logical
{
  public static class Application
  {
    public static void Close() { }
  }

  public static class Navigation
  {
    public static void SwitchToNotYetAnsweredQuestions() { }
    public static void SwitchToSkippedQuestions() { }
    public static void SwitchToAnsweredQuestions() { }
    public static void MoveToNextQuestion() { }
    public static void MoveToPreviousQuestion() { }
    public static void IndicateAmDone() { }
  }

  public static class Question
  {
    public static void SelectAnswer() { }
    public static void ClearAnswer() { }
    public static void SkipQuestion() { }
  }
}
I used nested classes as a navigation aid, and static classes because you only act on the "active" item (e.g., the current question or the current application), but these are minor implementation details. More important is:

The active voice used to name the methods, which directly describe the actions your user will take.
Each method's implied target of the active item.

Although these are somewhat orthogonal as well - the main point is to embed a user action-focused point of view into your LFM.

This is all you need to write test cases!
[TestMethod("TestMethodSetup", "TestMethodTeardown")]
public void CanSwitchBetweenAllQuestionSetsAfterAnsweringAllQuestions()
{
this.AnswerAllQuestions();
this.SwitchBetweenAllQuestionSets();
}

private void AnswerAllQuestions()
{
  this.MoveToFirstQuestion();
  do
  {
    Logical.Question.SelectAnswer(this.SurveyNavigator.CurrentQuestion.PossibleAnswers[1].Label);
    Logical.Navigation.MoveToNextQuestion();
  } while (this.SurveyNavigator.ThereIsANextQuestion());
}
private void SwitchBetweenAllQuestionSets()
{
  Logical.Navigation.SwitchToAnsweredQuestions();
  Logical.Navigation.SwitchToSkippedQuestions();
  Logical.Navigation.SwitchToNotYetAnsweredQuestions();
}
You can't actually run the test cases of course (well, not and have them actually do anything anyway), but they will compile. They don't contain any verification or UI or other implementation details, so you won't have to edit your test cases as those details change. The intent of each test is immediately clear - you don't have to hack through thickets of UI manipulation, application internal data structure accesses, and verification code to figure out what their points are. It is quite likely that once your LFM methods are filled out and the application under test implements the corresponding functionality your test cases will just "light up" and start working. The only reason you should need to modify a test case, in fact, is if the user feature it is testing radically changes! If the feature is cut or given completely different semantics than you will need to revisit test cases, but simply changing how the UI works or adding a new way to execute the action generally won't have any repercussions on your existing test cases.

*** Comments, questions, feedback? Let's talk: michhu at microsoft dot com.

Nuts And Bolts

micahel — Wed, 15 Jun 2005 20:45:00 GMT

All that theory is great, but without a practical application none of it matters. Thus Surveyor.

Surveyor is a simple application for creating online surveys. The UI will look something like this:

You have three sets of questions: those you have yet to answer, those you have skipped, and those you have answered. Results are displayed when you complete the survey. That's pretty much it from the user's point of view.

The implementation isn't much more complicated. This first view is an Avalon control that you databind to an XML survey definition. When the user completes the survey you are provided the results and can then do whatever you like with them. Most of the UI is just what it looks like; the question set navigation controls [top-left corner of the window] are styled radio buttons.

As I said, pretty simple. There is however enough meat to explain in detail how to turn all that theory into an automation stack. I won't post detailed code, but I will post C# snippets and a lot of pseudocode.

At a minimum I'll have individual posts covering the Logical Functional Model, Physical Object Model, Execution Behaviors, and Comprehensive Verification. I rather expect at least the latter two will require multiple posts, but the level of detail to which I go will in large part be based on what y'all want to see and hear, so let me know what that is!

*** Comments, questions, feedback? Want a fun job on a great team? I need a tester! Interested? Let's talk: michhu at microsoft dot com. Great coding skills required.

All For One And One For All: Our Complete Automation Stack

micahel — Mon, 06 Jun 2005 19:30:00 GMT

Factoring these details out of the test case into intuitively organized libraries allows the test case to focus on the essence of what it is testing rather than incidental details. This helps us write more comprehensive tests faster but spend less time maintaining them.

It also enables us to approach testing in a very different fashion. We can write test cases before the feature has been coded. We can write test cases before the UI has been built. We can write test cases as the feature is being designed. Being able to write test cases immediately upon entering the milestone allows us to be front loaded rather than eternally catching up to Dev. This is how we do it:

It all starts with user features. Along with deciding what the feature should do, how the feature should work, what the workflow should be, and what the UI should be, the feature team designs the Logical Functional Model for the feature. The LFM is really part of the feature, so the LFM is defined inline to the feature specification. This makes clear the similarities between the LFM and the feature it is modeling and highlights any mismatches between them. Delivering the framework necessary to test the feature is part of the cost of building the feature, so implementation of the LFM is costed alongside implementation of the feature. We have found that stubbing out the LFM and starting to think about its implementation as it and the feature are designed can surface issues that would otherwise have been missed.
Next, we can start writing test cases. Build verification and exit criteria tests can be available for developers to run before they check in the corresponding code, with basic and extended functionality test cases following throughout the milestone. Similarly, developers can use the LFM to write their integration tests prior to implementing their features. Having all this code checked in but never executed and thus never tested is a bit unsettling at first. It definitely puts the onus on the test case writer to understand what each test is supposed to do and how the application is supposed to respond. Writing test cases that simply “light up” as soon as the corresponding application, LFM, and POM code is checked in is a bit of a challenge but is by no means impossible. We have had some great successes using this technique.
Stub out the Physical Object Model as soon as the UI is defined. This can happen quite late in the development cycle since the test cases are written against the LFM and thus do not need to be changed when the UI changes.
Implement the LFM, POM, Application Internals, and Expected State Generators as application code comes online. The feature team has more latitude scheduling this work since each can be implemented mostly independently from the others as well as from the test cases.
Design and implement the verification model throughout. As with the LFM this is designed through the joint efforts of the feature team and recorded inline to the feature specification. Defining the verification model forces the feature team to truly understand the feature’s inner workings. The resulting discussions are incredibly effective at finding areas where the specification is less than forthcoming or less than precise. Because Loosely Coupled Comprehensive Verification mostly decouples verifying actions from executing them, verification can start small and ramp up over time. As the feature becomes better defined and application code comes online verification can be brought online as well. Regardless of how much or how little verification is enabled at any one point in time, test cases continue to run and continue to find problems.
Define and implement Execution Behaviors and Data Providers as the need arises. Execution Behaviors are likely to be identified while the feature team is designing the LFM; the need for specialized Data Providers may become evident during feature team discussions of the test cases. As with the verification model, feature team discussions regarding the various ways a feature can be executed and the types and ranges of data a feature must handle help the team to identify areas of the feature that require further thought.

Teamwork combined with these innovative testing techniques has already proven dividends. This is however just a fragment of the benefits we will see from testing smarter. You can too!

How Do I Invoke Thee? Let Me Count The Ways: The Physical Object Model

micahel — Fri, 03 Jun 2005 19:30:00 GMT

Most test cases that manipulate a user interface are tightly tied to the current details of that UI. What the test case is doing often gets lost in the details of how it is doing it. We abstract away these implementation details by wrapping our application UI with a Physical Object Model that lets us reference the UI without having to know the details of how it is accessed.

We take this a step further by using a controls abstraction layer to smudge UI details into similarity. If a check box in the UI is changed to a radio button, everything that knows that check box is a check box now has to be updated to talk to it as a radio button. We don’t want to have to make that change, especially when a control simply changes to another type without affecting its semantics. If you really do care about a control’s type you can to talk to it as such, but we have found those occasions to be few and far between.

Show Me Yours: Application Internals

micahel — Wed, 01 Jun 2005 19:30:00 GMT

Details regarding how data is pulled out of our application’s internal data structures and other parts of the system environment are factored into an Application Internals model. This isolates knowledge about where data is found and how it is accessed in a single location. We use the same user-centric organization here as we do in our LFM, so locating these getters is a simple matter of determining to what feature the particular piece of data corresponds.

Did You? Did You Really? Loosely Coupled Comprehensive Verification

micahel — Mon, 30 May 2005 19:30:00 GMT

Verifying that a test case’s actions had the expected result is perhaps the most important part of testing. Every test case does something at least a little differently than every other test case, so the expected results are often a little different. These minute differences make it difficult to factor verification out to shared code and so verification code tends to be embedded in and duplicated across each test case.

Intermixing test case execution code with test case verification code further complicates matters. Initial state data necessarily must be gathered before individual operations are executed. Expected state can be calculated anytime after initial state is recorded to just before actual state is verified. Verification that actual state matches expected state must of course be done sometime after each operation is executed; often immediately after, if subsequent steps in the test case will destroy the current actual state. All of this makes it difficult to differentiate between execution code and verification code.

Separately, the set of properties that are typically verified is nowhere near the complete set that would be necessary for truly comprehensive verification (that is, verifying every property after every operation). The copious amount of work required to do so is generally deemed not worth the trouble. This is especially true since for any particular operation most properties will be unchanged. Experienced testers, though, will recognize that this is exactly how the most insidious bugs are manifest by changes in something that should be completely unaffected by the operation.

We have bypassed these problems by decoupling verification from execution. Loosely Coupled Comprehensive Verification is easy to explain and almost as easy to implement. Just before a test case or LFM method executes an operation, it notifies the Verification Manager that it is about to do so and also provides any relevant details. The test case or LFM method next executes the operation, and then finally it notifies the Verification Manager that it has completed the operation. That’s it as far as the test case or LFM method is concerned!

When Verification Manager is notified that something is about to happen it baselines current state and then works with a set of Expected State Generators to determine the expected state. Upon notification that the operation has completed Verification Manager compares actual state against expected state and logs any differences as failures.

This very loose coupling between verification and the rest of the system makes it very flexible. If the details regarding how a particular expected state is calculated change, the corresponding Expected State Generator is the only entity that has to change. Similarly, if the set of properties being verified changes nothing outside the verification subsystem needs to be modified.

Another benefit we get from this scheme is a dramatic reduction in follow-on failures – failures that occur solely because some previous action failed. Because we baseline expected state before every action it is always relative to the current state of the application, so a previous failure that has no effect on an action won’t fail that action just because the verification code expected that previous action to succeed. This eliminates “noise” failures and allows us to concentrate on the real problem.

Because verification details are decoupled from execution, the set of properties being verified can start small and expand over time. Helping this to happen is the ability to say “I don’t care” what happens to a particular property as a result of a particular operation. Any property with such a value is ignored when actual state is compared to expected state after the operation has completed. Once the expected result is known the tester simply updates the Expected State Generator appropriately and suddenly every test case automatically expects the new behavior.