Our single biggest misstep in our test approach last version was that we failed to sufficiently change our testing methodologies to fit the Agile environment under which we developed our V1 product. As we had done in our previous years here at MS, we placed our emphasis on system-level, UI-based test automation, which had served us well in the past when working under more of a waterfall model.
So, what did our test approach look like heading into V1? Well, let me try to encapsulate a book-sized chunk of info into a succinct, semi-directed list:
We found that following this traditional, system-level testing approach our automation was consistently about a sprint behind feature implementation. This gap led to many unintended consequences:
As we moved on to V2 and looked back at the lessons learned, it was clear that we needed to make a number of bold steps:
The P0 test:
The initial focus of test automation is what we call the 'P0' test. This is the quick-turnaround automation that ensures that QA and Dev are in synch, gets regression automation in place quickly, and seeks to push QA/Dev to stay in synch through the cycle. Here are the basic requirements for P0 tests:
This is a 100k foot view of something that we've spent a lot of time on and, as I read the post, can see that it may be confusing in many ways on its own. As I follow-up with more info on other testing that we are doing, though, I hope that the place of P0 testing is put into proper context