[Rob] 

General IDE testing cycle status points (iterations on/between any two points…)

1. Specification review
2. Test Plan Design
3. Test Plan Review
4. Developer Implementation design
5. Developer Implementation design review
6. Test case design [exploratory, user feedback, regression]
7. Automation Design
8. Automation Implementation and review
9. Manual Test Pass
10. Automation run
11. Project status evaluation and reporting

Since the organization is broken up between disciplines (Program management, Development, and Test), cross discipline communication helps deal with the implied dependencies in the eleven points above.

A typical cycle contains zero or more of the following obstacles based on the above points:

1. PM (program manager group) delivers spec(s) late, preventing review by schedule
2. test plans delayed by changing specs/parallel product cycle effort lowering priority
3. test plan reviews are cursory, resulting in missed holes fund later which cost more to correct
4. the design is the code
5. well, we can review the code…or just use the feature…
6. test cases ended up being designed without complete specifications…requires revisions regularly at added cost
7. automation gets pushed back due to UI/feature churn making tests obsolete an hour before they are checked in
8. Automation harness issues; changing requirements; dogfooding delays (a good thing, but causes progress trade-off) [dogfooding:  using VC drops to develop automation for testing VC drops…]
9. Manual Test Pass: can always count on them, but they take too long if automation not strategic or complete (temptation is to delay automation to get complete manual coverage creating an ever deepening hole…bite the bullet and do the automation right so manual is only what can’t/shouldn’t be automated).
10. matrix explosion (platforms * SKU’s * architectures * permission levels;  pair wise testing save us!)
11. What do you mean, “make it green”?!  [balancing quality driven vs. date driven product requirements]

Some random elaborations:

There’s a lot of overlap at the beginning/end of product cycles.  This really prevents any one discipline from getting a ‘fresh start’, as wrap up work often takes priority over new work (reverse that for PM).  Thus, the new work gets delayed quite often.

A note on ‘priority’:  Everything is ‘pri 1’, because if it wasn’t, it was cut.  This is because there isn’t enough time to do everything that ‘has to happen before we ship’.  So, ‘priority’ has no meaning.  Just ignore it.  Treat it as someone saying, ‘um…’ or ‘er…’.  I’m making it a priority to remove that word from tester vocabulary….

At one point, our team had fallen significantly behind our goals for automation development.  We found ourselves in a position where we needed to keep testing up on implementation churn by providing immediate testing of any dev patch/check in.  This constant impromptu testing continuously delayed automation development between formal test passes.  We are in a much better state at this time because we followed through on the difficult decision (made with the dev org) to forego the interrupt driven manual testing completely for several milestones and focus on automation (formal test passes still occurred, of course, to cover all test cases for each milestone).

One angle which has been used with automating IDE features can be summed up with a call to “do everything as the user would”.  Reality set in as conflicts with shared UI continuously broke tests over multiple groups.  Since feature innovation can’t be stalled just to maintain an automation test bed, it became necessary to follow an academic rule:  “a test should test only what it tests”.  In other words, if you aren’t specifically testing the UI, if at all possible, the test should not fail because of a UI change.  Praise be the DTE!  (http://msdn.microsoft.com/library/en-us/vsintro7/html/vxgrfAutomationObjectModelChart.asp).  For visual studio, feature area automation authors can often bypass UI that they don’t need to directly test and get at what does need to be addressed with the current test.  This has made automation faster and more reliable for our current test harness.

[This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.]