Eric writes:

There's a lot of debate over automating software testing (ie is it worth it, will it get used, how flexible can an automated tool really be, etc.), and I was wondering if you had any firsthand experience.

Automating software testing is certainly worth the definite effort involved - as long as you automate the right things. Where "right" is entirely dependent on your context. If your application is a thin UI wrapped around a comprehensive model (i.e., set of APIs), you can easily write oodles of automated tests against that model. Automating the UI might be simple - if you can replace the application model with your own test model, then testing the UI is simply a matter of verifying that each UI action calls the model correctly. If you can't mock the application model, or the UI contains embedded logic, or the UI is undergoing drastic change, or the UI is dead simple and hardly ever breaks, then you might decide to not automate UI testing. Or you might decide to automate a few basic tests and do the rest manually. It all depends on your context.

Any test for any application can be automated. Some tests are easy to automate. Other tests would take a team of crack testers and developers years to automate. Some automated tests are easy to make reliable and stable and performant. Other automated tests are reliable but take days to run. Others are fast but not very reliable.  How do you choose? It all depends on your context.

In every case the feature team must decide what their priorities are. One hundred percent coverage of code, or data, or features, or some other measure? A fast gauge of the application's state? Near complete certainty that specific portions of the application are defect free? Each of these can be the right answer - or exactly the wrong answer. It all depends on your context.

Context, context, context: these are the three most important considerations of testing. If you are a one-person shop testing your own code, you might decide to automate everything because you know you become blind to your own mistakes, and so you build tests for every bug you find to ensure you do not make a similar mistake again. Or your customers might be clamoring for frequent updates and be tolerant of bugs and so you let them do your testing for you. If you are a giant corporation who will be servicing the release for years to come, you might opt for large quantities of automated tests with a bit of manual testing thrown in, so that hotfixes and service packs can be managed with a very small team. Or you might decide to not do any what-people-normally-consider-testing and instead conduct comprehensive code inspections of every last line of code, because your historical data shows that code inspections find more bugs faster and cheaper than does testing. It all depends on your context.

In the contexts I have experienced, I have found that automating most tests - starting with comprehensive unit tests and continuing with integration tests written against a model-based-ish automation stack - is a good use of my/my team's time. You may find something different. It all depends on your context.


*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great coding skills required.