Our single biggest misstep in our test approach last version was that we failed to sufficiently change our testing methodologies to fit the Agile environment under which we developed our V1 product.  As we had done in our previous years here at MS, we placed our emphasis on system-level, UI-based test automation, which had served us well in the past when working under more of a waterfall model.

So, what did our test approach look like heading into V1?  Well, let me try to encapsulate a book-sized chunk of info into a succinct, semi-directed list:

• Receive the initial feature specs
• Review and provide feedback on the spec
• Create a comprehensive test plan based on the spec.  The focus of the test plan is at the system level, meaning that it is abstracted more or less at an end user action level
• Determine the supporting test libraries, both UI and non-UI, to be authored
• Author supporting libraries as you author automation.  This is a labor-intensive process, particularly for the UI libraries as they are not only complex, but also fragile, being highly succeptible to product changes
• Author test automation
• Excecute recurring test runs in a test management environment upon which our automation was entirely dependent
• Repair automation as product changes occur
• Expand and further abstract the supporting libraries to enable model-based testing tools to generate broad suites of tests to be used as static regression tests.

We found that following this traditional, system-level testing approach our automation was consistently about a sprint behind feature implementation.  This gap led to many unintended consequences:

• For about a sprint, there was little to no consistent QA automation backing up some features, meaning that serious breaks could go uncaught for far too long
• QA and Dev were at different points in the feature development timeline, severely dampening critical QA/Dev person-to-person conversations.  Simply, Devs were focusing on implementing new features while QA was still looking at the 'old stuff' in an effort to get automation in place
• Changes made to the product meant that QA would, at times, spend time working on supporting libraries that were now out of date
• QA constantly felt behind, putting a damper on the amount of time that many QA feature owners devoted to critical hands-on testing.  Getting the automation in place and keeping it running was soaking up too much time.

As we moved on to V2 and looked back at the lessons learned, it was clear that we needed to make a number of bold steps:

• QA and Dev owners of the same feature must work in a far more collaborative fashion and develop strong interpersonal relationships and a sense of joint ownership of their feature.  This was accomplished by some owners in V1, but we needed it to be more broadly true in the organization.
• QA cannot afford to focus all, or even the majority, of its testing efforts at a system level as can be done in a waterfall environment.  Instead, much of QA's efforts must be at the subsystem level.
• When a new feature comes online or is significantly altered, QA needs to get some high-pri tests automated immediately before moving on to broader testing.
• The overhead of test automation must be made far lighter to free up time for additional QA hands-on testing early in a feature's life and QA management must ensure that such testing is a highly valued priority.  To lighten the overhead, use of UI-driven automation is avoided, instead leveraging API testing and DTE testing.
• To further strengthen Dev/QA synergy, all Dev unit tests and QA tests are run together on a nightly basis.

The P0 test:

The initial focus of test automation is what we call the 'P0' test.  This is the quick-turnaround automation that ensures that QA and Dev are in synch, gets regression automation in place quickly, and seeks to push QA/Dev to stay in synch through the cycle.  Here are the basic requirements for P0 tests:

• They should be automated and in place within 1 day after a feature or feature change appears in the build.  In order to make this happen, P0s must be few in number (this is by design) and Dev must give QA a heads-up on the changes being made before they are checked in.
• P0 tests must be associated with the dev workitem(s) that they cover.  This is recorded in the tests' metadata
• P0 tests must have been agreed to by both Dev and QA before they are implemented.  Both should agree that failure of the test represents a critical failure.  If not, the scenario envisioned is likely a P1 or P2 test (I'll cover those in another posting)
• P0 tests must not duplicate Dev unit tests but, instead, be at a bit more abstract level, what we call 'API integration testing'.  What does this mean?  Well, if you have methods A, B, and C, Dev should have unit tests for each; however, QA P0 tests will cover how A, B, and C interact.  B may appear to work correctly and C may appear to work correctly, but if C is dependent upon upstream action from B and that integration point is faulty, QA should catch it.
• All P0 tests are run on a nightly basis.  Many P0 tests are also labeled as being 'check-in' tests and must be run by Dev before any code check-in takes place.

This is a 100k foot view of something that we've spent a lot of time on and, as I read the post, can see that it may be confusing in many ways on its own.  As I follow-up with more info on other testing that we are doing, though, I hope that the place of P0 testing is put into proper context