JW on Test

PingBack from http://www.easycoded.com/the-manual-v-automated-testing-debate/

Debating manual v. automated testing is like debating eating v. drinking.  You can't survive without either one.

Automated testing frees the tester to do more manual testing.  It also helps the tester stay passionate about the product she is working on.

cant the same be said of others involved in the product - devs get too interested in using the latest programming techniques/languages, UI designers the latest cool trends and forget the product itself ?

I want to respond to the comment on comparing manual testing and automation testing to eating vs drinking while pointing out a subtlety or a distinction based on what is being automation.

If ur automating a set of API's or driving an object's methods to get test coverage I can see a lot of code(and coding) being involved. In such cases, i can see how a tester could get carried away with the testing code more so than the product itself.

On the other hand, if you are automating user clicks or end user actions then really I don't see how you could automate without actually having manually exercised the application/user actions or at least explored the application and made decisions on what you want to automate.

Such approach to automation may not exercise all the methods, but only the ones exposed by the end users actions. It ofcourse does have a down side.

From the POV of automating end user actions, isn't manual testing a sort of a pre-cursor to automation (of user actions). If you got a brand new app to automate, what would you start doing?

Ofcourse, I'd eat and drink for sure before embarking on any work.

It's not just automation, and it's not just testers.

Everyone on the team needs to remember... the whole point is to ship the software.

I've seen testers who feel an immense sense of accomplishment at creating a piece of elegant test automation which never ends up finding any bugs.

And I've seen testers who feel that just writing a tidy Test Plan is enough - actually executing that plan is someone else's burden.

I've seen developers who pride themselves on implementing that latest piece of technology, while ignoring the ever-growing bug list.

I've seen Product Managers proudly publishing their latest Business Requirements document, but who cannot be bothered to discuss the missing pieces with the testers as preparation to creating an effective Test Plan.

All the elegant automation, state-of-the art technology and poetic writing isn't worth much if you don't end up shipping a high-quality product.

This is a great blog entry - I'm really glad you did this one.

The behavioral aspect is really intersting. We often talk about candidates' ability to "break things" when we put them through interviews and then happily jettison that attitude in favor of having them writing comprehensive suites of regression automation (when do we find the bugs to regress?). More and more, software testers are judged on their ability as developers than as thinkers (not to say developer don't think, they just don't think as users) - and thinking is what testers need to do most.

Good Automation will find fewer new bugs than good manual testing, but there's a place for both. All too often, managers and their teams see automation as the most important piece of the testing and damn I'll agree it's an important one. However, as you've mentioned in other blogs, requirements, exploration, etc are easily as important and the more we hire people for development skills, the more we move towards behaviors and attitudes that are in line with logic flow than with customer flow (a very peculiar thing indeed!).

When approaching a testing project, I like to break things down as much as possible by the deisgn patterns of the system under test. If that's, say an MVC pattern, then the test patterns will try to map best to that through both their chronology and what they actually are. API level, object model and general automation test work really well when testing a statistical model (or even a testable view). When testing a controller or set of controllers, testers can rely less on automation and need to begin looking to requirements and scenarios.

This is where I've seen two testers have completely different results. I've seen a tester that wrote a supposedly incredibly robust suite of automation for a major (very UI centric) feature that missed 90% of the user scenarios (and nearly had himself fired). I've seen another tester not write a single line of test code and release a quality piece of software, but spent weeks and weeks retesting that code.

The balance you speak about is very difficult to achieve. I often like to theorize about how the behavioral aspects of developers and testers are contraindicative to one another. As I've become more of an automator and less of a black box tester I feel like I'm losing my grip on my creativity. I feel like I'm removed from the user and I'm at one with my developers.

In large organizations like Microsoft, this has become commonality - the black box tester is somewhat of an anachronism. If I go to one more review calibration and hear some test lead tell me that "he's great, he writes really good perl", I may descend into a hideous mire of depression, for good.

Well, that's a can of worms! In fact, that's a JW blog post I'd love to read: How to measure testers...

I'll go one better than telling you what makes me happy based on my outlandish comment above:

I love:

"She writes average perl, but she wrote some really great scripts that really cut to the core of the user requirements and we'll be able to use for years. She made great use of automation libraries, contributing to them when needed, and her automation for this feature has become the standard for testing a feature of this kind. Her perl will grow, but her testing skills are right on. Her perl will improve, but it's not vital for what she's contributing right now"

I also kinda love:

"He writes incredible perl and has used that skill to help us grow our toolset and point out where we can do things we've wanted to do before but lacked a clarity and skillset to do so"

I don't love:

"She's a good coder, thank god we hired her"

In many respects, I feel that calibration reviews are more a measurement of the managers' than the individuals. By seeing how managers calibrate, what they value, what the don't, what they get and what they struggle with, what they fight against and what they applaud, we get an insight into what kind of person those managers will hire. And to me, that is the crux of why I think your post (well, really, your posts) is so important.

I love automating, but I think I'm bad at it. Like most automators, I get hypnotized by the peacock of cool smart code, and I forget to look for really interesting scenarios. I'll often begin with bvt tests, then move through the categories. By the time I get to "scenario based tests", I'm either out of time or I'm bored. At that point, I've probably identified the need for a really cool, well factored, multipurpose automation library and I'd just love to do that instead.

It scares me that we hire for this skill-set and that it's a skill-set that I've been encouraged to grow. It's not that the skills aren't important, née vital, it's just that the problem is that we use that to measure if somebody is a good _tester_, and it really doesn't tell us that.

Traditionally, it's very difficult to measure good testers. In the past, we'd use things that had supposed "business impact" (not test or user impact) to measure because that was easy (things like they owned a big feature or they created a good website). Then, of course, we also used the usual flawed metrics (bug numbers, validity, etc), because that was easy. Now we use development skills. And interestingly, that's not easy, but it is easy to misuse or misjudge - especially when those skills are sometimes lower in the managers adjudging them.

Manual testers don't have to be black box testers, and automators don't have to be white box testers. We're all way confused about what it is we really want. I guess the hardest part to understand is how to see if someone can apply a testing pattern to a problem and be versatile enough to then use whatever form of testing is appropriate.