JW on Test : General comments

tour of the month: the exit-stage-right tour

James Whittaker — Thu, 21 May 2009 23:11:00 GMT

All tours much eventually come to an end and thus it is with my tour with Microsoft. I have resigned my position and am leaving the company. It was a great ride.

But the tours will continue. My book Exploratory Software Testing: Tips, Tricks, Tours and Techniques to Guide Manual Testers is in press and will appear through Addison-Wesley sometime this summer. I am truly thankful for the many wonderful testers at Microsoft who contributed wisdom, thoughts and even case studies to the effort. Special thanks go to Nicole Haugen, Geoff Staneff, David Gorena Elizondo, Shawn Brown and Bola Agbonile. Microsoft is full of great testers and even here, these guys manage to stand out.

I imagine that I will not be long in setting up a new blog as I have very much enjoyed this experience and being the only tester in Developer Division's top ten bloggers was quite an honor. For that I thank you.

In case you are interested in my landing place, I can imagine that one or two of the more popular testing blogs around town will be talking about it.

Wish me luck ...

testing sucks

James Whittaker — Fri, 03 Apr 2009 00:11:00 GMT

Bet that got your attention. It's true, but let me qualify it: Running test cases over and over in the hope that bugs will manifest sucks. It’s boring, uncreative work and since half the world thinks that is all testing is about, it is no great wonder few people covet testing positions. Testing is either too tedious and repetitive or it’s downright too hard. Either way, who would want to stay in such a position?

What is interesting about testing is strategy, deciding what to test and how to combine multiple features and environmental consideration in a single test. The tactical part of testing, actually running test cases and logging bugs is the least interesting part.

Smart test managers and test directors need to recognize this and ensure that every tester splits their time between strategy and tactics. Take the tedious and repetitive parts of the testing process and automate them. Tool development is a major creative task at Microsoft and is well rewarded by the corporate culture.

For the hard parts of the testing process like deciding what to test and determining test completeness, user scenarios and so forth we have another creative and interesting task. Testers who spend time categorizing tests and developing strategy (the interesting part) are more focused on testing and thus spend less time running tests (the boring part).

Testing is an immature science. There are a lot of insights that a thinking person can make without inordinate effort. By ensuring that testers have the time to take a step back from their testing effort and find insights that will improve their testing, teams will benefit. Not only are such insights liable to improve the overall quality of the test, but the creative time will improve the morale of the testers involved.

So all the managers out there need to ask themselves what they've done lately to make their testers more creative. If you don't have an answer, then testing isn't the only thing that sucks.

live webinar next week

James Whittaker — Wed, 25 Mar 2009 23:23:00 GMT

I am doing another webinar on March 31 which will be hosted by uTest. This is the second webinar I have done for them and I quite enjoyed the first. I have a lot of new material that I have created (and am still creating) for this one which I've called "5 Insights to Revolutionize Your QA."

The link to register is http://blog.utest.com/?p=122. Note the use of the words 'sucks' in the blurb. I love it when I get marketing people to say sucks...

links you've been asking for

James Whittaker — Wed, 25 Feb 2009 20:26:00 GMT

I’ve gotten enough requests for links to my interviews, lectures and pointless videos that I am finally tired of responding individually. Here are a few recent ones I have been pointing people to:

A video interview on Channel 9 about software testing

A video interview on Channel 9 about Visual Studio Team System

A radio interview on Dot Net Rocks with two funny guys (I had a blast with this one and it will be one reason I never run for political office in the future)

The Future of Testing webinar for uTest

No, “Piece of Crap” is not included. Sorry but it’s time we put that one to sleep. Lee Copeland is widely known to play it at his lectures so try and catch him on his speaking circuit if you just can’t resist. That video is another reason I can’t run for political office. Neil Young has my sincerest apologies.

Some internal links for my MS readers:

The Future of Testing

An Update on the Future

Online Security Testing Course (this is officially a classic since it is the last time I taught this material)

Sorry to post internal links that many of you can’t get to. But if you do manage to get to them, report the bug you exploited so I can include it in my future talks!

getting away from it all

James Whittaker — Sun, 01 Feb 2009 22:39:00 GMT

When you’re on vacation do you think about work? Not thoughts of dread, worry or angst but reflection, planning and problem solving. I just did. Last Sunday I awoke in Seattle to freezing temps and a dusting of snow. By midday I was building a sandcastle on Ka’anapali Beach, Maui in 79 degree sunshine. If that’s not getting away from it all, I don’t know what is.

Yet my mind wasn’t really away. In fact, I thought about work all the time. Given that software was everywhere I looked, it’s not hard to see why. My entire trip was booked online, even the taxi to the airport. Not a single person besides myself took part in the process. Just me … and a load of software.

The taxi cab itself contained software, as did the airplane. The baggage carousel, the espresso machine, the car rental counter (no person there, just a self serve terminal) and even the surveillance camera that watched my son juggle his soccer ball while I packed our bags in the trunk. All alone, except for the software. Even the frozen concoction machine had software that helped it maintain the right temperature. (It broke, incidentally, making me thankful that I am a beer drinker.)

Is it possible for anyone in this field to really get away from it all? (Don’t get me started on the motion sensors that control the air conditioning in the hotel room. I’m all for turning them off when they are not in use, but apparently sitting still and being cool was not one of their end-to-end scenarios.)

The truth of the matter is that getting away from it all just isn’t necessary for me. I like seeing software in action and I enjoy brooding over problems of testing it. Vacations free my mind from the daily grind and leave my mind to question things that back home I might overlook. Does this make me work obsessed or just indicate that I really like what I do?

Vacations have always been like this for me. When I was a professor, two students who led my research lab, Ibrahim El-Far and Scott Chase, actually avoided me when I returned from a trip, afraid of the work my new insights would bring. They never quite managed to successfully do so.

Which brings me back to the motion sensor in my room. The problem isn’t so much a poor tester, rather poor testing guidance. The sensor does exactly what it is designed to do and testing it based on those requirements got me in the sit-and-sweat loop. The problem is that no one thought to give it a field try … what I call ‘day in the life’ testing. Had the tester thought to take the sensor through a 24 hour cycle of usage they would have identified that problematic ten hour period (yes, ten, it’s a vacation after all) when motion is low and the desire to be cool is high. But what tool gives such guidance? Modern tools help testers in many ways, but helping them think of good test scenarios isn’t one of them. They help us organize, automate, regress and so forth, but do they really help us to test?

That’s the tool I want. Tomorrow, when I return, I am going to direct someone to build it for me. Ibrahim and Scott, you are off the hook this time.

new year's resolutions

James Whittaker — Mon, 05 Jan 2009 22:53:00 GMT

Welcome to the new year!

2009 will be the year I publish a new book on testing and the year I ship my first testing tool since Holodeck oh so many years ago.

I’m thinking about calling the book exploratory testing. Believe it or not the title isn’t taken. Any thoughts on that title? Am I too cheeky by claiming it? By happy coincidence the material I am preparing is on manual testing of a strikingly exploratory nature.

I don’t get to name the tools though as they will be shipping as part of Visual Studio 2010 and are currently code named after various islands in the Puget Sound. My boss (yes, there is some poor unlucky soul here at the empire who is forced to claim me as a direct) is in the process of blogging about those tools our team is building. Allow me to introduce Amit Chatterjee at his corner of MSDN. Please send along your condolences for him having to be the one to write my review.

Also, based on reader feedback, it will also be the year that I expose more insider details about testing culture and practice at Microsoft. Those are the most read of my blog posts and I’m going to take that as a hint. Although, much of the thunder has already been aired by my colleagues in their new book How We Test Software at Microsoft. That book's a good read and a high bar for me to match when my own book comes out in a few months.

poor promotion prospects for testers? (dirty MS laundry)

James Whittaker — Fri, 21 Nov 2008 19:22:00 GMT

As a result of my last post, I got this email from a Microsoft tester whom I work with fairly regularly and admire deeply:

"Yet how many Partner or Distinguished Engineer ICs in test do you see? How many testers get onto bench programs vs dev/pm?

Throughout my career at microsoft I've seen the cream of the crop of dev and PM rapidly rise through the ranks, but the cream of the crop in test gets shafted every time - even on teams that do covet innovation and provide leeway to discover and learn."

My last post was aimed more at keeping junior testers in the discipline and I still think innovation and mentoring is the answer to that. This comment is aimed at the higher levels where promotion concerns come to the forefront. Is there a glass ceiling for testers that doesn't exist for managers and developers? If Microsoft, who has a reputation for celebrating the test role, has such a ceiling must the prospects industry-wide be that much worse?

I'm not the most qualified to discuss this since I earned my level outside of Microsoft and did not rise through the ranks. Plus, talking publicly about our promotion policies is sending alarms bells through my head, I can hear HR rumbling now. So let me say this: we testers have a second class legacy to overcome. We are hampered by a history of not being appreciated. This is going to be one of the most important trends that we have to reverse as a group. All any of us can do is look down the ranks and make sure we are pulling people up behind us.

Before you stop to complain about your own promotion path, ask yourself how many people below you have you helped get promoted? Are you doing everything you can to ensure that the ICs two and three levels below you are getting noticed and getting promoted?

I think we're doing this pretty well at the lower ranks of Microsoft because promotions occur pretty briskly for the talented. But to get past that glass ceiling, we're going to need people at the higher levels to take mentoring more seriously. I wonder if those people are reading this blog ... if so, the guy who wrote the quote above is waiting to hear from you!

I posted some additional comments about this at the uTest blog in case you'd like to read more. Mentoring an promotion are important topics. I'd like to get others in the community to share their ideas around this.

keeping testers in test

James Whittaker — Fri, 21 Nov 2008 01:56:00 GMT

I did a webinar for UTest.com today and got some great questions. One question seemed to really resonate: how do you keep good testers from moving to development.

I hear this question a lot. Many engineers see test as a training ground for development. A testing job is just a foot in the door for a quick move to development. Sigh.

Let’s be honest, this is not a bad thing. I think that the more developers we have trained as testers is categorically good. They’ll write fewer bugs, communicate with test better and generally appreciate the work their test teams do on their behalf. I think the real sadness comes from the fact that Test as a discipline loses so many talented people.

I am not convinced that the folks who leave are really doing so because of the developers’ greener pastures. After all, there is a lot of code to write as a tester and it’s often a freer coding atmosphere. I think people leave because too many test managers are stuck in the past and living just to ship. Everywhere I see testers move to development I see teams that lack a real innovative spirit and the converse is most certainly true. The happiest, most content testers are in groups that covet innovators and provide opportunity to invent, investigate and discover.

Want your testers to stay? Give them the opportunity to innovate. If all you see is test cases and ship schedules, all your testers will see is the door. Can’t say I blame them either.

software tester wanted

James Whittaker — Wed, 19 Nov 2008 00:46:00 GMT

Position requires comparing an insanely complicated, poorly documented product to a nonexistent or woefully incomplete specification. Help from original developers will be minimal and given grudgingly. Product will be used in environments that vary widely with multiple users, multiple platforms, multiple languages and other such impossibilities yet unknown but just as important. Performance, security and privacy are absolute must haves and the applicant is expected to teach us exactly what that means. Post release failures are unacceptable and could cause us to go out of business so some knowledge of appropriate quality buzzwords is necessary.

no more testers?

James Whittaker — Sat, 15 Nov 2008 03:58:00 GMT

I gave a keynote at EuroSTAR on the future of software testing where I began by painting a picture of the promise of software as an indispensible tool that will play a critical role in solving some of humankind’s most vexing problems. Software, I argued, provides the magic necessary to help scientists find solutions for climate change, alternative energy and global economic stability. Without software how will medical researchers find cures for complex diseases and fulfill the promise of the human genome project? I made the point that software could very well be the tool that shifts the balance of these hard problems in our favor. But what, I asked by means of a litany of software failures, will save us from software?

Somehow as I painted my predictions of a future of software testing that promises a departure from late cycle heroics and low quality apps, some people got the impression that I predicted ‘no more testers.’ How one can latch onto a 20 second sound bite while tuning out the remainder of a 45 minute keynote is beyond me. The US elections are over, taking sound bites out of context is no longer in season.

This blog is replete with my biases toward manual testing and my admiration for the manual tester. If you read it and if you managed to listen for more than a couple of minutes during my keynote you’d have to conclude that I believe that the role of the tester is going to undergo fundamental change. I believe that testers will be more like test designers and that the traditional drudgery of low level details like test case implementation, execution and validation will be a thing of the past. Testers will work at a higher level and be far more impactful on quality.

I quite imagine that the vast majority of testers who actually listened to my full message will rejoice at such a future. I invite the others to take a second read.

speaking of Google ...

James Whittaker — Wed, 22 Oct 2008 19:24:00 GMT

Actually, it is more like speaking at Google as I am headed to GTAC tomorrow to give the newest version of my Future of Testing talk. Hope to see you there.

I’ve received tons of feedback on my blog posts about the future. So much so that I spent most of this past weekend integrating (or stealing, I suppose you could say, depending on your perspective) your insights, corrections, and additions. Thanks to all of you who have discussed these things with me and shared your wisdom.

If you happen to miss GTAC, I’ll be giving a similar but darker version at EuroSTAR entitled The End of Testing As We Know It in The Netherlands on November 11. Yes I was drinking and listening to REM when I made the presentation.

I think it helped.

the empire at star west

James Whittaker — Fri, 26 Sep 2008 18:25:00 GMT

If you only attend one conference on testing, I recommend STAR (East or West). It remains the biggest conference and trade show in the industry and all the best speakers are on their keynote circuit. Unfortunately, I am going to miss next week’s event in California. Fortunately, the empire will be well represented in both keynotes and track talks.

Tara Roth is giving a keynote on her experiences as a test executive with Office. Tara and I serve together on Microsoft’s Test Leadership Team and I have a lot of respect for the work she does on Office. Tara is a Partner here (which means she’s a big shot, makes big decisions and big bucks too) proving that Test is a discipline with no career ceiling. If you want to talk to one of the most successful career testers on the planet … she’s one of them.

David Gorena Elizondo is speaking on exploratory testing. He gave the talk two days ago here on campus and it got great reviews. The bugs he shows and the next generation test methods he details are getting a lot of usage around the company and are finding important bugs. Don’t miss his session. You can also visit David in the Microsoft booth in the expo.

Keith Stobie needs no introduction. He’s a staple on the circuit and one of our most well known test personalities. Whenever I am feeling overly smart, I like to strike up a conversation with Keith to bring me back to earth. This guy knows his testing. Theory, practice, manual, automated, interface, protocol, API … it doesn’t matter. He’s ten testers in one. Do yourself a favor and talk testing with this guy. You’ll walk away better off for the experience. Keith’s talk is about model-based testing.

Grigori Melnik is speaking alongside Jon Bach (one of our favorite consultants) on acceptance testing. These guys ran a project to define acceptance testing and included many of the best test thinkers in the company. I’ve read their report and my expectations for this talk are high.

Enjoy the conference and please send along any information about good sessions.

on certification

James Whittaker — Thu, 11 Sep 2008 16:23:00 GMT

How do you feel about tester certification? I’ve heard all the arguments for and against and looked at the different certifications and their requirements. Frankly, I have not been impressed. My employer doesn’t seem impressed either. I have yet to meet a single tester at Microsoft who is certified. Most don’t even know there is such a thing. They’ve all learned testing the old fashioned way: by reading all the books and papers they can get their hands on, apprenticing themselves to people at the company that are better at it than they are and critiquing the gurus and would-be gurus who spout off in person and in print.

Simple logic tells me this: Microsoft has some of the best testers I have ever met. So in my experience there is an inverse relationship between certification and testing talent. The same is true of testers at other companies I admire that I meet at conferences and meetings. The really good testers I know and meet just aren’t certified. There is the occasional counterexample, but the generalization holds. (Whether the reverse is true, I have little data with which to form an opinion.)

Let me repeat, this is my experience and experience does not equate to fact. However, the reason I am blogging about this is because I met three office managers/administrators recently who are certified. These three are not testers, but they work around software testers and they hosted a certification course and thought it would be helpful to sit in and understand what the people around them did day in and day out. They sat the courses, took the exam and got their certification.

Hmm.

Ok, I’ll grant they are smart, curious and hard working. But there is more to testing than that triad. They readily admit they know little about computing, even less about software. From the time I spent with them, I didn’t get the impression that they would have made good testers. Their skill is elsewhere. I doubt they would pass any class I ever taught at Florida Tech and I imagine they’d find the empire’s training a bit too much for them to digest as well. Yet they aced the certification exam without breaking a sweat.

What am I missing? Isn’t the point of a certification to certify that you can do something? Certify is a really strong word that I am uncomfortable using so lightly. When I hire a certified plumber, I expect said plumber to plumb beyond my uncertified ability. When I hire a certified electrician I expect that electrician to trivialize the problems that vexed me as an amateur. If I hired a certified tester, I would expect them to test with a similar magnitude of competence and skill. I wonder if an office manager of a plumbing company could so easily get certified to plumb.

Well I checked into it. Plumbers (at least in Seattle) are indeed certified but they don’t get that certification by taking a course and an exam (although they do both). They serve time apprenticing to a master plumber. You better believe that by the time they get that seal of approval, they can plumb till the cows come home.

I realize testing isn’t plumbing but the word certification gives me pause. It’s a strong word. Is there something more to tester certification that I am missing? Is it simply that you understand the base nomenclature of software or that you can converse with other testers and appear as one of the crowd? Or that you simply sat through a course with enough of an open mind that some of it sunk in? What value does this actually bring to the discipline? Are we any better off because we have these certifications? Are we risking certifying people who really can’t test and thereby water down the entire discipline?

I don’t think these certifications are really certifications at all. It’s just training. Calling it a certification is over selling it by a long shot. In my mind a certification means you have a seal of approval to do something beyond what an amateur/tinkerer can accomplish. Otherwise, what has the certification accomplished?

I am proud of being a tester and if I seem arrogant to be that way then so be it. What I do and what my compatriots do is beyond a single course that an office manager, no matter how smart, can just pick up.

However, if I am wrong about certification, I’d like to be enlightened. For the life of me, I don’t see the upside.

the poetry of testing

James Whittaker — Mon, 28 Jul 2008 21:02:00 GMT

God Save the Queen! (A curious statement … from my American point of view. But given what history has recorded of certain of England’s Kings I’ll grant the gender bias. Anyway, Save Her all the same as she presides over a country of such glorious breweries!)

If you haven’t guessed it already, I’m visiting England. I’m also in a pub (you probably guessed that too). And I just met with a half dozen or so local testers who convinced me (with the offer of free pints) to meet for a book signing. I almost never turn down a signing and I never turn down free beer, especially at the current exchange rate.

Upon parting, they urged me to turn our conversation into a blog post. Here it is. I hope it doesn’t embarrass me in the morning.

A single signature seeker was a developer. When I asked him why he bought my book he answered that he wanted to make sure his testers weren’t successful with the “tricks” I preached in it. He intended to frustrate them by writing code that wouldn’t fail that way.

I smiled and told him that if this was a soccer, excuse me … football, game I would rip my shirt off and celebrate my goal. He looked at me funny. I think the testers got it. I bet you do too.

He went on to describe why developing code was better than testing it. He talked about the challenge of wrestling with the compiler and deftly parrying the attempts of the IDE and the operating system to thwart him on his mission. It was a battle to him, a conquest. He was a Knight, fighting for User and Binary.

It was a great story, and I didn’t get permission to identify him so I won’t but his passion was fantastic and the world is a better place because he’s in software development.

But if developers are the fighters, I think of myself and my fellow testers as the bards. Testing, to me, is poetry. As I test software I have visions of inputs mingling with data, some are stored internally; some are used temporarily and discarded. I hear music playing as inputs move through the app and find their way to a data structure or get used in some computation. It helps me to think about the inputs in this way, it helps me understand what the application is doing with the input I give it and that in turn helps me to think of ways to break it. Every potential sour note represents some possible way the developer may have screwed up. Imagine your app processing input. Listen to the poetry it recites, it will tell you when it’s going to fail.

I find this especially true of testing web apps. I envision in my mind the formation of SQL queries that my inputs cause the application to make. I form impressions in my mind of the HTML traffic that is transmitted from client to server and the response back again. What is the application doing? Where is the data going and with what purpose? These are deep, existential questions worthy of the bard in all testers. And they find bugs. The more I can picture the internal processes going on in the application, the better I am able to understand how the developer might have made a mistake.

The music of the pounce is what makes it all worthwhile. That moment in which it becomes obvious that the software can do nothing but fail. It’s euphoria, the equivalent to scoring a winning goal. But, please, keep your shirt on. That’s a caution-able offense in football and we don’t want developers to be brandishing yellow cards at us.

the european test community

James Whittaker — Fri, 25 Jul 2008 14:21:00 GMT

I had the extreme privilege to speak to a crowd of test practitioners based in the UK last week. The event was hosted by Transition Consulting (TCL) and boasted some of the UK’s top consumers of testing services. A list of those companies is posted here but you’ll have to scroll down a bit because Stewart Noakes has been an active blogger recently.

One comment that some of my American readers might not like: European test audiences tend to be a lot more aware and a lot more involved in this discipline of testing. Everyone seemed familiar with my writing and the writing of people like Beizer, Kaner and Bach. I was especially surprised at the discussion of the history of the Kaner and Beizer schools of thought in the early 90s and the general knowledge of both industry and academic testing conferences and publications. There seems to be more eagerness to delve into the field and its history here than I generally see in my own country. These folks are really well read!

Proponents of certification might point to that as a reason since certification of testers seems far more popular in Europe. Does certification help spark people’s passion for testing? Test training in general seems more popular in Europe.

I think it might have something to do with the American bias toward test automation, particularly Microsoft’s. Most in our test community are SDETs and approach testing from a very developer-oriented perspective. (The High Altitude Developer explained this well.) They may be less inclined to think of themselves as testers and less inclined to involve themselves in its culture and history. That’s a shame. (Obviously there are many counterexamples at Microsoft but I think this is generally true among the population of tens of thousands of us.)

I am probably going to get in a lot of trouble for this post. But now that I’ve mentioned certification I have a hankering to blog about that now. I can almost guarantee that what I have to say about certification will draw some fire.

I do have a gig in London next week, we'll see if I change my mind. Certainly, I am not brave enough to post about certification before I face that crowd.