I. M. Testy

Hi Stefan,

Spot on! I didn't mean to suggest that GUI automation should be avoided, but that we need to be smarter about how and when to do it.

I absolutely concur that GUI automation is a great approach to simulate end user interaction; especially if we model the UI,  define use cases as actions, and use a model based testing approach to simulate an even greater number of scenarios.

Good class design should separate the core logic and functionality from the GUI layer. But, of course it is easy to write poorly designed code, which is why there will always be jobs in testing. :-)

As you say there are some automation frameworks that provide an abstraction layer to help 'ease the pain' or improve ROI a bit, but timing and balance are key.

Hi Shrini,

Agreed. In my opinion many people make too many incorrect assumptions about the value a good automation effort provides to an organization.

I would also say that automated GUI testing can be useful in identifying certain types of usability issues. For example, missing key mnemonics, tab order, layout errors such as control overlap, truncation, etc. can be highly automated. (So, I guess I am that someone. :-)

But, I also agree there are other aspects of usability testing such as the 'ease of use' perspective may never be automated, nor should it be.

Hi Shrini,

Reducing test cycle time with automation occurs within the overall product cycle. For example, when I wrote the automated BVT for the international versions of Windows 95 it ran approximately 1000 functional tests on 4 different language versions simultaneously within approximately 30 minutes. This automated BVT/BAT test suite saved more 15 hours per build in BVT/BAT time which then provided the test team additional time to design and execute an increased number of tests to expand depth and breadth of coverage.

There is a common myth that test automation will reduce the product development lifecycle. However, this is a foolish assumption for 2 reasons.

First, there are a greater number of all possible tests in any complex system then can be feasibly accomplished within any reasonable timeframe to make exhaustive testing virtually impossible. (The single biggest challenge in testing is to reduce the nearly infinate number of tests to a very small subset of tests that is going to provide the decision makers with the appropriate qualified information that will allow them to better analyze risks and make more informed business decisions.)

The second reason increased test automation typically does not impact the total development lifecycle is a result of Parkinson's Law; "work expands so as to fill the time available for its completion." Test automation may indeed reduce specific testing cycles within the overall development lifecycle; however because the potential test burden is so large the overall schedule will unlikely be effected to any large degree.

Shrini,

I really like your statement "with good automation, you will do more testing (accomplish more) than before." I couldn't agree with you more. Well-designed automated tests can provide accurate evaluations of functionality and allow a tester to design and/or execute different tests to increase overall coverage.

Some of those 'extra' things in an automation cycle that you mention are also in manual testing. For example setup of unique environments, and of course I there will always be a maintenance cost for formalized test cases (tests that are written).

Of course, there are some breakthoughs in test automation and manual testing to reduce setup costs such as virtual machines. Also, a test automation framework can distribute tests across the wire to any networked computer that matches a particular setup, or recreate that setup from an image (if it exists) much quicker than manual testing. For example, at Microsoft an SDET may have anywhere from 2 to 6 machines in his or her office. However, once thier automated test gets checked in to source control it can be distributed to any computer in that team's lab (and with approx 120,000 lab machines around the company that automated test has the potential to execute on a lot more hardware and softwaer configurations than can be accomplished via manual testing within the same period.

We are also doing a lot of work with smarter deterministic and declarative type oracles, and also automated test result analysis. I talked a bit about this last year at some conferences. And we are starting to see these areas mature and providing increased value to our testing efforts.

I am well aware of the glorious stories told by automated test tool hucksters. In some cases, these tools provide what the customers need (and that is good), but in some cases I think customers are mislead, or perhaps the customer doesn't really understand testing, or perhaps some managers are attempting to use a tool to perhaps solve a much larger systemic problem.

Oh, and I also agree with you that any comparison between manual and automated testing is usually vague and meaningless. Manual testing gives us information that an automated test can't, and there is some information that an automated test can provide more accurately, or more timely as compared to a manual test. We must learn to use both approaches to their greatest benefit rather than view them as competing approaches.