notes and rants : Testing

PNSQC Slides

alanpa — Tue, 27 Oct 2009 20:10:23 GMT

Thanks to everyone who attended my talk today. Feel free to fire more questions here if you have them.

Slides are available here.

Words and Status (and rants!)

alanpa — Wed, 20 May 2009 06:41:00 GMT

Words are funny things. We use them to convey meaning - but there's more to words than that. We also use words to try to change the way people view things. In real estate advertisements, for example, you see words like "rustic" used instead of "old", or describe a home as "having character" when it really means the house is practically falling apart.

I find it interesting (or amusing, I can't decide which) that testers play this game too. I know testers who avoid and replace words entrenched in the testing vocabulary with words that they think sound better - or in their "words", words that convey a better meaning than the original. Other than diluting the testing vocabulary, I don't really have a problem with this. If you want to call a boundary a fence-post or an edge, that's fine. If you want to call it a termination endpoint, that's fine too.

What really bugs me (believe me, I've had to edit the "bad" words out of this rant more than once already) is when people change the meanings as an attempt to raise the value of test, or show that test "has a seat at the table", or is a "first-class citizen". I have to tell you, I'm sick and tired of testers whining about not getting respect. If you want respect, you have to give respect and you have to earn respect. Using fancy words isn't going to do it for you, but if you do your job and show value and stop trying to convince people you're more valuable than you truly are, you'll be fine. I know plenty of testers who do these things, and guess what - they have respect (and even admiration) from their peers in development. This is a huge peeve of mine, and unfortunately, one I deal with too frequently for my own mental well-being. Here's a message for all testers - you aren't going to earn respect by whining about the fact that you don't have respect. Nothing, in fact, could be less effective. Knock it off and try something else. I'm this close to impersonating SteveB and throwing a chair through the window of our lovely test excellence offices.

I named this blog many years ago when I was "just" a test architect on the windows CE team at Microsoft. Finally, as I get to this point in the post I realize that I've finally written a post that reflects the blog title - or at least close if you consider a rant a word.

Terms and other things we can’t agree on

alanpa — Sun, 08 Mar 2009 06:21:16 GMT

I was in a meeting last week where we had a long drawn out discussion of some terms around scenarios, use cases, and design. I knew the ambiguity was there, but watching the conversation unfold reminded me how important a common understanding of terminology is in order to have a conversation that goes somewhere.

The meeting was great, but I walked away depressed. I was depressed because as great as the meeting went, it reminded me that communicating about test is much, much worse. Quick, tell me – what is test automation? What’s a test harness? What’s a test case? Your answers are probably different than mine, and our are different from the next person to read this post. Testers can’t have deep or broad conversations about testing because we can’t understand each other. I’m racking my brain right now to think of one word or phrase about testing that would mean the same to all of the testers I know.

Normally, I like to present my solution right now, but I don’t have one. The profession has had a few common terms in place for decades, but some sections of the profession want to define their own terms. I completely support having different approaches to the craft of software testing, but I wish it didn’t come at the expense of losing the chance of a common vocabulary.

So, you tell me. Name something all testers agree about. Or, tell me why we don’t need to agree on terms. If you think we do need to agree, tell me how we can get there.

Or, just make stuff up.

Metrics at STAR Conference

alanpa — Sat, 17 Jan 2009 18:22:48 GMT

I think I forgot to mention that I'll be speaking at STAR this spring. You'd think I'd be pimping hwtsam, but instead I'm speaking about something barely mentioned in the book at all - metrics. Software metrics are an old passion of mine - something I used to be heavily involved with, but much less so in my current role.

That, of course, doesn't stop me from talking about how teams can use (or misuse!) metrics to make critical decisions. I think it will be a fun talk, and sometime between now and May, I'll figure out how to try and sell a few copies of the book too.

I'm betting that as the conference approaches that I'll have a few more metric related posts to share.

DRE

alanpa — Wed, 28 Feb 2007 04:26:13 GMT

I know I'm overdue for a post when Bj starts lapping me on posts.

This week, I've been finishing up my presentations for STAR and PSQT (for those who haven't presented at a conference before, presenters are expected to have their presentation complete 2-3 months before the conference date). The two conference presentations are barely related, but the piece that brings them together is something that I have been meaning to post here for a while.

For anyone who has come across this post looking for information about Dr. Dre, all I can tell you is that I'm a big fan of the Ben Folds cover of a Dre song that I can't post the title to here (and certainly not lyrics!). For everyone else, this post is about Defect Removal Effectiveness.

I talk a lot about preventing bugs and everyone has heard ramblings on moving quality upstream, but nobody really seems to know too much about what it means. I believe that most software defects can be prevented, and for those bugs that cannot be prevented, that they should be detected as early as possible. DRE is simply a measure of how effectively defects (bugs) are removed at each stage of the product cycle.

Note: now that I've mentioned stages of the development cycle, some scrumbut is going to proclaim me as waterfall man. Please consider "stages" any serial activity in software development. Even agile purists want to have their note card user stories in place before writing code, so please apply this concept to whatever flavor of water-spiral-v-w-x-xp-xyz-iterative-prototype-rup-nm model you choose to use.

Hold on tight for a moment while I attempt to explain DRE without the use of any fancy formatting tricks such as tables, scrolling text or lightweight ajax controls.

Say, for example, that you find 10 bugs during requirements review (ambiguous statements, conflicting statements, etc.) Note yet again, that you can find these same sort of bugs reviewing user stories for consistency. Say that throughout the rest of the product cycle that you find another 15 bugs that relate to requirements. This means that during the initial stage / phase of the product cycle you found 10 bugs of the eventual 25 that were there at the time. Grade school math tells me that your effectiveness during this phase was 40% *10/25). Is that good - I don't know, I'll tell you in a minute.

Now, let's say that while the devs were coding they found another 10 of the requirement bugs, as well as 10 errors in their coding (due to unit test, buddy test or sheer luck). Let's also assume that 15 additional bugs were found in the testing phase which were attributed to developer error during coding.

This means, that during the coding phase there were 40 bugs latent in the product (15 requirements defects, 10 dev errors found and 15 remaining). 10 coding errors were found, along with 5 requirements errors. Grade school math (which again comes in handy) says our defect removal effectiveness was (15/40) 37.5%.

The numbers in this example are pretty close, so we can't say whether we're significantly better at one phase vs. the other. However, if you track this sort of metric throughout the product cycle, it allows you to do two important things.

Measuring DRE helps you target where improvement needs to be made. If, for example, developers are introducing a huge number of defects while writing code and finding very few of them, it would point to the necessity of additional detection techniques during that phase
Measuring DRE helps validate any sort of improvement programs. Say, for example, your development team is implementing unit tests. If they track the number of defects found they can validate the effectiveness of the time invested in writing unit tests.

The big caveat (if you haven't thought of it already), is that if you don't track when a defect was introduced in your bug management system, you can't track DRE. Hardly anyone I know currently tracks this, but I get more converts to this sort of thinking nearly every day.

ed. 9:20 pm. formatting

**boy I hope my grade school math holds up.

Exploring Exploratory Testing

alanpa — Sat, 30 Dec 2006 04:09:18 GMT

Exploratory testing seems to be the “it” word of software testing these days. Everyone from high profile consultants to newbie testers are talking about exploratory testing. My nature (similar to that of many testers) is to be a bit of a naysayer, and because of that, I’ve often tried to find holes in exploratory testing theory. However, I’m not one to provoke argument just for the sake of it, so I spent some time over the past few days analyzing my thoughts on ET. I’ll warn you in advance that this will likely be a long post, but I hope you take the time to read to the end.

My only real beef with ET is that everything I see written about it is from a black box approach. ET proponents describe it as “simultaneous testing and learning” In short, try something, see what it does, let the result guide what you try next. Yep – good stuff. The problem I see with many exploratory testers (including sometimes with the experts!) is huge amounts of inefficiency in applying the learning to the testing. I don’t get a lot of chances to observe exploratory testers, and conference presentations and videos may not do their skills justice, but it doesn’t matter since I absolutely agree that you should learn while you’re testing, and what you learn should directly influence what you do next.

I think most good testers will agree that the most effective testing is a combination of black box and white box testing, so the next thing I did was challenge my assumption that ET is a black-box approach. One common white box technique is code coverage analysis. When I’m looking at code coverage data, I learn about how my tests are executing and apply that knowledge to writing new tests. If I played with words long enough I may be able convince myself that ET can be applied to code coverage analysis, but I certainly wouldn’t convince anyone else. Strike one. How about code reviews? Nope – strike two. I took a step out of the metaphorical batter’s box to see if there was a time when I was using an exploratory approach when doing something inherently white box.

POW – base hit (and thankfully, the end of the baseball metaphor). I’ve written about debugging before (heck, I’ve written debuggers!). I almost always use an exploratory approach when debugging(let me watch the state of this variable and see if it tells me anything interesting…what happens if I change this variable…). Another thing I often do when testing a new component is to exercise the entire module in the debugger. Before I even write test cases, I write a few basic tests (scripted or unscripted), and use the debugger to understand how every code path is reached and executed. I note where boundary conditions should be tested, and where external data is used. I typically spend hours (and sometimes days) poking and prodding (and learning and executing) until I feel I have a good understanding of the component. The more I thought about it, the more I realized that these debugger sessions were exploratory sessions, and I love doing them. Assumption resolved – I love exploratory testing!

This reminded me that the stance of the ET zealots is not ET vs. white box testing (that was my own erroneous assumption). The ETers generally compare ET to scripted testing. But what tester would write a bunch of automated / scripted tests without taking time to understand what they were testing? (rhetorical question – hopefully none). Good test writers are going to use an exploratory approach to help define their test cases (manual or automated). Depending on the audience and support cycle of the product, ET approaches, in fact, may be enough (white or black box approaches). I definitely don’t have a problem with that. For some products, a combination of scripted and exploratory approaches are needed, but exploratory approaches can definitely help in designing better scripted tests. Case closed – ET is a valuable testing approach. (Note that the arguments against scripted tests are valid. Model based testing combats some of these arguments, but there’s a whole other post brewing in that discussion).

I thought I was done, then I thought some more…

Good testers use exploratory techniques. They may use them exclusively, or they may use them in conjunction with another approach, but in short, good testing == exploratory testing. Good testers need to be able to experiment, learn as they test, then apply what they’ve learned to the current and future testing sessions. In fact, I can think of dozens of situations where I’d prefer ET over scripted testing. Even in a “standards” based testing organization, I think there is tremendous value in exploring as a method of influencing scripted test case design. I’m starting to love exploratory testing more and more…

Aww crud – my brain wouldn’t stop there. I love to cook too. When I cook, I experiment with ingredients, learn from my experiments and apply what I’ve learned to the current and future cooking sessions. I like to mountain bike. When I’m biking, I experiment, learn from my experiments and apply what I’ve learned to the current and future biking session. I like to golf. When I golf, I experiment, learn from my experiments and apply what I’ve learned to the current and future golf session. When I …

Live search failed in an attempt to find anything on exploratory cooking, mountain biking or golfing. If you’re not so pissed off at me by now that you’ve stopped reading, you realize exactly what I came up with. Exploratory testing is just… testing. I would argue that those that don’t simultaneously execute and learn aren’t really testers (or certainly aren’t good testers). In fact, I’ve found that many articles on ET can safely be read with the word “exploratory” removed.

Here, I’ll show you an example in one sentence. I love ~~exploratory~~ testing. Try it yourself :}

SDETs at Microsoft

alanpa — Thu, 14 Dec 2006 02:49:40 GMT

Some time ago, MS began hiring primarily people with a programming background (industry or educational background) for test positions. As far as I know, there was no press announcement, but the idea seems to have worked its way into the overall global test community. Of course, anyone not close to information draws their own conclusions on this direction and what these people do (Bj has a post on this (with even more links) here). The interesting point (for me at least) is that MS has always hired "coders" for test positions – just not necessarily in the ratio we do today. The most common conclusion drawn is that we hire these coders for test positions because we want to automate everything and eliminate manual testing. While we do want testers who can write effective automation, that's only a small part of the equation. Testers who understand programming concepts and computer architecture typically have the analysis skills that testing requires. They can also (typically) find bugs earlier, and understand the root cause so that they can quickly find other similar bugs and implement early detection or prevention routines. I used the word "typically" in the above two sentences because not every good coder makes a good tester (not every "coder" makes a good developer either).

I often hear the argument that good testers are creative and are experts in their area. First of all, creativity (including creative problem solving) is a key attribute of just about any profession. Good programmers are creative too – for that matter, so are authors, secretaries, and zookeepers. The creativity argument is lost on me. As for the second point, the "expert" role and the "programming" role certainly are not mutually exclusive. In fact, I have known good testers who didn't know anything about programming. I have also known good programmers who couldn't test to save their lives. However, the truly fantastic testers I've met in my life fit the middle of that Venn diagram. Any good tester can find bugs. Today's software is in a state where finding bugs is pretty much like shooting fish in a barrel. We need testers that can find the bugs that exist today, but also implement quality improvements that will prevent those bugs from happening in the future – then devise techniques for finding (and preventing) the remaining bugs, and deploy these techniques efficiently across complex systems. Testers who understand programming are, in my opinion, are far better suited to these tasks.

Should testers without a programming degree be fired? Are you kidding – of course not. Any thoughts along that line are completely unfounded and irresponsible. However, if you asked me if testers should have a knowledge of software architecture and programming concepts, I would undoubtedly say yes. To me, finding bugs is passé and frankly, easy. Tomorrow's testers need to have the knowledge and skills that will help them find and prevent those bugs efficiently and effectively.

I could go on, but I think I've made my point.

The happy path should always pass

alanpa — Fri, 01 Dec 2006 10:14:10 GMT

I was in a discussion with some other test architects at work today. We were talking about challenges in testing, and at one point in our discussion someone mentioned his frustration in finding bugs in the basic functionality of an application or interface.

Many, many years ago, I thought it was cool to find basic functionality bugs - "hey - when I select this menu item, nothing happens" or "when I call this function with nominal parameters, it crashes". I remember thinking, I'm so glad I tested this so I could find this bug. That minimalistic euphoria lasted about 36 seconds (give or take a few years) before I realized that finding bugs in "the happy path" was a big fat waste of my time. I ~~hate~~ despise receiving code to test that doesn't work in basic scenarios. If all of the basics don't work when I get the code, the developer should probably be fired - but at the very least embarrassed if not suicidal over handing code to test that does not work for the most common scenarios.

I know many great developers who are embarrassed when a tester finds any bug in their code - even convoluted "a customer would never do this" bugs. These developers write great code - plain and simple. On the other hand, there are a ton of developers who blindly throw code "over the wall" to testers. Testers spend several seconds testing the freshly caught code, then have to throw it back when they are blocked because basic functionality doesn't work.

Here's a message for these developers who happily deliver bad code to testers: I think you should consider doing something else for a living. You have no business making software. If you don't see the idiocy in creating software with this mindset you are probably too stupid to get another job, so you may want to consider moving back in with your parents and getting a paper route (apologies to intelligent paper delivery people everywhere).

Sheesh - the happy path should always pass.

About Certifications

alanpa — Tue, 21 Nov 2006 19:18:09 GMT

I was listening to talk radio on the way home from work last night, and the topic of discussion was the recent crane accident a few miles from Microsoft in Bellevue. The focus of the discussion was on the capability of the crane operator, and one of the hosts kept asking if the operator was "certified." He was convinced that because there wasn't a certification program for crane operators that the operator wasn't qualified to operate the crane.

I almost called in to comment, but fortunately, a few trained crane operators were up to the task, and convinced the host that, although the operator wasn't "certified", that he had been sufficiently educated through required training programs, and that lack of operator training wasn't the cause of the accident.

The conversation got me thinking about testing certifications. Up front, I'll tell you that I have zero testing certifications - I have, however, had a lot of training on the subject of testing (on the job, seminars / conferences, and a lot of reading). How do I compare with certified testers? I bet some know more about testing than I do, and some know less. A certification doesn't make you more qualified to do something - the preparation for the certification is where the knowledge is potentially gained. A testing certification doesn't guarantee you a job or pay raise. In fact, a computer science degree doesn't even guarantee you that. What matters in any job is your ability to apply the knowledge you have to effectively accomplish work.

I certainly have nothing against certification programs, but they are a situation where you get out what you put in. If you study the bok (body of knowledge) with the goal of passing a multiple choice test, your certification is pretty much worthless. However, if you study the bok with a mindset toward applying the principles to your own software testing work, you will probably gain a breadth of knowledge that you can apply throughout your career.

Great testers can be certified...or not. What matters is that you are passionate about the subject, are open to trying new ideas, and are able to draw from a large toolbox of testing techniques and methodologies.

Thoughts from the week

alanpa — Sun, 12 Nov 2006 23:30:55 GMT

I'll post this tomorrow, but currently I'm somewhere over the northern US or southern Canada returning from a week in the Boston area. I spent the last week talking to testers from Groove and Softricity. Technically, I was delivering training, but all I really did was talk about testing all week. I don't claim to be an expert in testing, but I think I do know enough to give most testers something new to think about, or a new way to think about something they already know about, and I think I was able to achieve that.

I don't typically blog about classes I teach, but this last week is worth saying something about. Simply put, I had a great experience, and it was probably one of my favorite weeks of teaching since I quit teaching in the public schools 15 years ago. The fact that I enjoyed the past week has nothing to do with any feats I pulled off - I was just impressed with everyone I talked to. Given that MS has acquired these two companies relatively recently, I think the testers may have been a little worried that some flunky from "corporate" was going to come and tell them how to do their jobs (I'd probably feel the same way). Instead, I just talked about the testing they were already doing, and gave them a few new things to think about. I always worry when I teach experienced testers that they will get bored and not challenge themselves and say "we already know this". Instead, they pretty much dove into and dissected every scenario I gave them. I love to see people think and challenge themselves, and I was impressed with the effort they put into courses. I drove them pretty hard with six to six and a half hours of class a day and a minimal number of short breaks, but they kept on working and listening - even when I was tired and a little fried at the end of the day.

Another unique (and cool) aspect of the week is that I saw nearly the entire test team at Groove (plus two testers from Softricity). It's not often that you get a chance to have influence over an entire organization like that, and that certainly added to my satisfaction this week.

That said, about a week away from home is about all I can handle, so I'm looking forward to touching down in Seattle and sleeping in my own bed.

Where do they get this stuff?

alanpa — Sun, 12 Nov 2006 23:30:31 GMT

In the "if it's on the internet, it must be true?" category is this article and quote from it-director.com (Malik Arjun originally brought the article to my attention)

The ratio of testers to programmers is now 3:1. This has given Microsoft another headache as they try and recruit good testers with the technical and personal ability to winkle out software bugs. Lack of resources in the traditional heartlands of Redmond, WA, have lead to the off shoring of a lot of this work to Microsoft in India.

I know that at many companies, that three devs to every tester is a typical ratio, but that's not what the ratio is at Microsoft. But, if you read the above carefully, it says that MS has three testers for every developer - and that ratio is also quite wrong.

The ratio varies slightly across the company (there's no set policy), but company-wide, it's about 1:1. Even with a 1:1 ratio, that's a lot of testers. I'm not even sure what the work a team with a 3:1 test to developer ratio would look like.

On a side note, this could be the first time I've seen the word "winkle" used in the same sentence with "technical" and "software".

Building better software with the NM model

alanpa — Mon, 09 Oct 2006 22:24:58 GMT

In the course I teach for senior testers, we spend a few hours talking about different approaches to software development (waterfall, agile, v, w, rup, etc.). At the end of the discussion I, of course, tell them which approach is best.

Of course, I'm kidding - I hope anyone reading this realizes that. There are better ways to make software, and the hard part is figuring out which approach is best for the given situation and personnel. Steve Yegge posted a follow up to his "Good Agile Bad Agile" post this weekend. Like all of his posts, it's long, but well written with good points. If you have been under a rock and haven't read his agile post it's a good read whether you are a fan of agile or not.

Let me preface my rant by stating that I like Agile. An Agile approach may not be the best solution for every software problem, but many of the approaches that live under the agile umbrella are make a lot of sense for creating good software. One thing that bugs me (and it may just be what I've seen) about the devout agilists is the notion that non-agile approach to software development is inherently waterfall based. This is like saying any condiment that isn't mayonnaise must be ketchup. Ok - maybe not exactly like that, but I'm writing in a hurry :}

Waterfall (and it's brother and sister v-model and w-model) have been used to ship a lot of software. And...I guess that's about the only good thing I have to say about it right now. The idea of entirely completing any phase before starting on the next phase is ok in theory, but rarely works in practice. This is especially problematic in just about any project longer than six months or so (completely made up number - I don't have any empirical evidence to support that figure - it just seems accurate to me). Most projects I've worked on have been closer to spiral model (break a large project into milestones and take time to review the process and progress at each checkpoint) than anything else. Sometimes, it has worked, and sometimes it hasn't. Usually the teams I have been on have learned from their mistakes, and sometimes they haven't. I think those observations could be said of waterfall or agile (or even rup) projects as well.

Now let's discuss the Noodle Madly** model of software development. In the NM model, you look at the scope of the project, the people assigned to the project (including their experiences), and the goals of the project and design the engineering model with those things in mind. For a small internal project, a model based on Agile methodologies may be adopted. For a larger project with hundreds of people working on it you may choose something closer to spiral - or - you may choose a hybrid model. For example, because you have a 7 year support plan in place, you may be locked into creating complete design and functional specifications. However, you see short iterations as beneficial, so you choose to have milestones end every 6 weeks. You find that the engineering team is reluctant to have a team room, so you keep people in their individual offices - but you get every sub team to agree to work in a team room for one week every milestone. You may or may not decide that pair programming or pair testing will be part of the plan - it depends on the team and the work they need to do. That's the beauty of the NM model of software development. It's not based on blind faith that the plan will work for everyone - it's a plan created by people who understand the system and the components that make it up. It's a model that allows for reflection and compromise, and a model that can be used to create great software no matter what the audience is.

But what do I know?

** anagram of 'any old model'

[ed. 3:15pm - Yes - I realize that in theory, agile promotes individuals and interaction over process, but that's not what I see agilists practice. I see individuals and interaction over process and tools as long as the processes are purely agile based. ymmv]

Technorati tags: Software Engineering, Agile

Testing for correctness?

alanpa — Wed, 04 Oct 2006 23:01:04 GMT

This morning I watched a short video where the presenters looked at a UI element and asked questions on determining if it was "correct". It's a short (six minute) video, so please take a minute to view it for context. The overall theme is that rather than focus on whether something is correct, testers should determine if a problem exists. Those two concepts are highly related, and it's not worth (imo) trying to determine which is the focus of testing - especially since I think I could make a reasonable argument for either.

What got me thinking was the question the presenters were trying to answer - Is this arrow graphic in the drop down list correct? They discussed whether it looked right, looked similar to other drop down list arrows, was it the right shape, is it something I recognize, etc, and most importantly, is there a problem? Most of the video elaborates on these points.

I don't have a problem with the ideas presented in the video - however I do think that a better example may have made more sense. In my mind, as soon as the question "How do you know if this arrow is correct" was asked, I would have written a test to grab the window and either compare a snapshot against a known good bitmap, or pull the id directly from the resource section of the relevant dll (in this case, mso.dll), and compare directly against the resource (yes, the resource could be wrong, but it would help me isolate the problem). As far as knowing what the error should do, I could compare against the spec - or, if I was working on an agile team, against the behavior described in the relevant user story.

I suppose I like the ideas - I suppose I would just approach things differently given this example.

Tags: Software Testing, Testing

Debugging 100

alanpa — Mon, 02 Oct 2006 23:35:42 GMT

Universities typically title their introductory courses XYZ 101: Introduction to XYZ. Often, the 101 courses require that students have some basis of knowledge in the area (e.g. pre-algebra for a math 101 course, or some level of writing skills for an English 101 course). Where I went to school, these "pre-intro" courses were "100" courses - e.g. Math 100: Pre-Algebra.

I was recently asked to prepare a short course on debugging for a group of recent college graduates. The catch is, that none of them graduated with a computer science degree. They know some programming, but probably (imo) not enough for me to dive right into a laundry list of debugging tools and techniques.

As I was thinking about this, I realized that a lot of developers are too dependent on the tools. I have observed dozens (if not hundreds) of debugging sessions where developers stared at the code (or assembly) in the debugger looking for a hidden clue, or stepped meticulously through the code, only to miss the moment when their precious variable changed values. I've worked with developers who refused to look at a bug (or even think about it) unless it was trapped in the debugger. Debugging tools are one part of debugging, but I decided that I will talk about the other parts of debugging.

Simply put, debugging == isolating the problem. That means that to improve debugging skills, you need to improve skills on problem isolation. If your car won't start, what do you do? You could outsource (call a mechanic), or you could debug. If you decide to figure out why your car isn't starting, you are going to try to isolate the problem. You will want to examine the symptoms (is the engine turning over or not), the environment (is it cold, hot, do I smell gas, do I have gas), and historical variables (when was the last time it started, has this problem occurred before?). Once you've thought about these things, then you should bust out the appropriate set of tools (jumper cables, gas, oil, tire iron, ...) to help narrow things down. To me, these steps make perfect sense when thinking about a car that won't start, but I see developers (including test developers) ignore this when trying to track down a bug.

Tags: Debugging, Software Quality

Kaner's five year plan

alanpa — Thu, 21 Sep 2006 20:14:00 GMT

I usually don't write commentary about other peoples blogs, but Cem Kaner had an interesting post a few weeks ago that I wanted to talk about.

The first thing that jumped out at me was this quote:

I think there are a lot of good, diligent, motivated black-box testers who want to learn how to program and how to apply that to their testing. I think that the next generation of testers will have to have programming skills. Test-first might be the way to help some of our generation develop skills that have eluded them.

The first half of this quote summarizes the direction we have taken over the last few years in hiring testers at Microsoft. We want testers with programming skills. More importantly, we want programmers who have had the matematics and other analyitical training that a computer science (or related) degree requires. Many people think that the only reason for testers to know programming is so that they can write more automation. While creating automation is a part of why I think testers should have a CS background, testers with a technical background are also much better at creating test cases, and are also often better at exploratory testing. Testers that understand the system (application, OS, web, etc.) they are testing are able to create better test cases (written and exploratory), and be much less prone to over testing or under testing the system.

Some testers argue that creativity is the key skill that testers need in order to be successful. I woud be the last person in the world to argue that creativity is unimportant, but creativity is extremely important in practically any field of engineering. Of course creative skills are going to make someone a good tester, but a knowledge of computer science will make them a great tester. Do I think testers without a CS degree need to go back to school to get one? No - but they should put some effort into learning programming and related sciences (math, logic, etc.).

The second half of the statement reinforces my thinking about TDD as a test design technique. I think it's the way we should teach everyone to program.

A few paragraphs later, Dr. Kaner has this to say:

The problem is that we need a new generation of software testing architects and we don’t know how to foster them.

This, is exactly the focus of much of my work at Microsoft. We are hiring great testers, and we have a roadmap that shows that testers can be promoted as fast and as far as any other discipline, but there is still much more we can do to show how much MS values great testers. I believe in the roadmap - I was asked a week or two ago if I ever considered working for the 'G' company or a large local online book store - but I don't think either of those companies - or any company in the world for that matter has the career path for testers that MS has. Those of you that know me will know that I am just about the last person to stand up and rah-rah for MS, but I honestly believe that I can grow farther as a tester here than I can anywhere else in the world. Now I just need to convince 7500 testers the same thing.

12:02pm - fixed typo (crow -> grow)