AdamU's WebLog : Testing

Myth Busting Testing #2

adamu — Sun, 26 Jun 2005 00:07:00 GMT

Myth #2: Testing does not, by itself, improve software quality.

I love this one! I must have heard it like a gazillion times. And taken at its most literal, it is entirely correct. Huh? Yep, it's entirely literally correct. So why am I calling it a myth? Because it implies that testing and quality are not related, and that is completely false.

Reality: Without testing, you’ll never be able to improve your software quality. In order to develop better, you must test constantly.

Here’s a great quote from one of the best books on developing software, Code Complete by Steve McConnell:

“Testing by itself does not improve software quality. Test results are an indicator of quality, but in and of themselves, they don't improve it. Trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often. What you eat before you step onto the scale determines how much you will weigh, and the software development techniques you use determine how many errors testing will find. If you want to lose weight, don't buy a new scale; change your diet. If you want to improve your software, don't test more; develop better.” - Steve C McConnell, Code Complete: A Practical Handbook of Software Construction

McConnell’s quote above is a fantastic one; it makes great copy. However, I completely disagree with the conclusion that implies that testing and developing are at odds (at the best the tone probably implies that they are orthogonal), I believe that the quote actually shows just how important it is to test more, not less. Because in order to achieve improvements in your software by 'developing better', you simply must test more. Test has to become inextricably entwined in the development process. Here’s what I mean…

You are sitting in a restaurant somewhere, perhaps it’s a Saturday, and you’ve decided to take a ride to explore highway 2, and you found a little greasy spoon diner, and since it’s lunch time, you stop in to grab a bite. As you’ve mostly polished off the blue plate special, the spatial relationship between what’s left of the open faced sandwich and the side of fries gives you a moment of elucidation: you’ve hit upon an idea to solve a problem you were stuck on for most of the week. As you design, before it’s even going down on the napkin with the crayons that are supposed to be for the kids, you are testing your ideas. You form them up, write them down, and then somewhere in version 27, after you’ve done this iteration of design and test, design and test, you eventually translate those ideas to the napkin. It’s even mostly readable, although the crayon tore the napkin a couple times. You pocket the napkin, pay the bill, and you are happily on your way again, seeing the sights on highway 2, and they seem rosier than before lunch, and they probably are, because now your mind is free from the problem it had been background processing on.

When you return home that evening, you bust out the laptop, and carefully unfold the napkin, and you start to translate from napkin to code. Hey, you are a geek and love writing code, what else are you going to do on a Saturday night? And somewhere in the iterative process of translating from crayon wax to electrons, you’ve designed something that you believe will solve the problem. At which point, what do you do? Why, you hook this thing up to the rest of your system, you create some test data, and then you test it. You build even more test data, and try it another way. You find a corner case you missed, you fix it, and you test it again. You throw garbage data at it, and you fix a couple bugs that were there that weren’t validating some inputs. And because you are such a great developer that your ego never gets in the way of creating great code, you run through all those same tests again to make sure that the changes you made didn’t break anything.

When you get back to the office, you schedule a design review. Yep, you test it again. Once you’ve made changes to the design from that review, you implement the code, and then you have a peer do a code review. You see, you’ve measured (tested) your coding proficiency, and you know that for every 100 lines of code you write, there are probably 3 bugs. And sure enough, you find 4 during the code review (yep, you tested it again). You are ready to check it in to the source tree, so of course you run the standard set of check in tests on it. And then, because you are a great developer, you write unit tests for it so that when someone else checks some code in, or makes a change somewhere else in the code base, there's an early warning system that can signal that there's a problem.

As part of the check-in, you write a test release document on the code you’ve checked in, and you fire it off to your test partner. You know that the software design engineer in test (SDET) who will read it, will probably think of how this integrates with the larger system under test, and write scenario based tests for it. If he’s really good (and he probably is, because great developers always want to work with great testers, they have no ego problems) he probably has already had the scenarios coded up, and now he can go back in and fill in the gaps that your document has pointed out as potential areas of concern. Wow, you tested it yet again. You review the unit tests with your SDET partner, and he finds a corner case you missed, and he says he'll go ahead and add it to the test suite, and then he adds them to the check in tests and daily automation run. He can do this because the development and test teams use the same test harness for unit tests, checkin tests, functional tests, so any test that dev writes can be leveraged by test in the test lab, and any test that an SDET writes can become part of the dev unit or check in tests.

So if you want to lose weight, you do weigh yourself all the time. Why? Because you want to know where you stand. You want to make immediate course corrections, not wait a month and find out that you’ve put back on 3 pounds. You want to do it far more frequently. And you don't use just one scale. You measure yourself with a tape measure, and when you put on your jeans. You measure yourself when you look in the mirror. And while some measurements are more qualitative and/or subjective than others, all of them are useful and help you towards your goal of losing weight.

Think and Test, Design and Test, Develop and Test. Fix and Test. Fix more and Test. Then you hand it to someone else and help them test it. It’s what great developers, and great development teams, do.

Better development is better testing. If you never measure your code’s quality by testing, you'll never know if the quality is sufficient.

I will say it again (because I can, it’s my blog, and because it’s important)

Without testing, you’ll never be able to improve your software quality. In order to develop better, you must test constantly.

Do you have that kind of relationship (communication and tools) with your test partners? Shouldn't you? At Microsoft's engineering excellence days during the last 3 years, we've talked alot about 'pushing quality upstream.' This is half of what I think it means. There's more on what test and pm can do as well. What do you think 'pushing quality upstream' means?

Myth Busting Testing and Test Automation

adamu — Thu, 23 Jun 2005 13:27:00 GMT

I've been reading a lot lately, and wow, there are so many opinions on the advantages and disadvantages of writing test automation. I thought I’d share my observations as to the common misperceptions that anyone in the software development business (engineer or manager, development, test or pm) needs to understand about testing in general and test automation in particular.

Myth #1: There are a set of tests that I would execute manually, but would never write test automation for because I'd only ever run them once.

If you would only run a test once, then you've written the wrong test, or you haven't started testing soon enough.

I am going to say it again (because I can, it’s my blog, and because it’s important)

If you would only run a test once, then you've written the wrong test, or you haven't started testing soon enough.

Quite simply, if there is a test that you’d only run one time, your test design is wrong. The test should be more generalized, perhaps use data sampling to provide variability to the test. Well designed tests are the key to testing, whether that testing is automated or manual.

OK, Adam, assume that I believe you about writing more general tests, but what do you mean about testing soon enough? I thought that writing test automation too soon is a bad thing. It breaks all the time, and my maintenance cost is too high.

My point here is that if you’ve waited to test a component such that if you run all your tests and, assuming that there were no failures, it’s ready to ship, then you haven’t been involved in the development process early enough. I think that this is a great topic for a future more detailed post, so for now, I'll focus on the test design part of the problem.

A quote from James Bach from Test Automation Snake Oil:

Once a specific test case is executed a single time, and no bug is found, there is little chance that the test case will ever find a bug, unless a new bug is introduced into the system. If there is variation in the test cases, though, as there usually is when tests are executed by hand, there is a greater likelihood of revealing problems both new and old. Variability is one of the great advantages of hand testing over script and playback testing.

And another from Brian Marick on When Should a Test be Automated

The fact that humans can’t be precise about inputs means that repeated runs of a manual test are often slightly different tests, which might lead to discovery of a support code bug. For example, people make mistakes, back out, and retry inputs, thus sometimes stumbling across interactions between error-handling code and the code under test.

While I completely agree with what James and Brian are saying regarding variability, I completely disagree that you can’t have the same variability in your automated testing as your manual testing. It’s harder to design and implement test systems that solve the variability problem and yet can be reproducible, but hey, that’s one of the reasons we build software: to solve complex problems. It does mean that typical techniques that James mentions (scripted and playback testing) are not useful mechanisms for more effective test automation.

Now there may be a set of tests that may be too expensive to automate, but automating because of expense is an entirely different reason, and one that I will discuss another time.

Here’s another great quote from James Bach’s snake oil:

One day, a few years ago, there was a blackout during a fierce evening storm, right in the middle of the unattended execution of the wonderful test suite that my team had created. When we arrived at work the next morning, we found that our suite had automatically rebooted itself, reset the network, picked up where it left off, and finished the testing. It took a lot of work to make our suite that bulletproof, and we were delighted. The thing is, we later found, during a review of test scripts in the suite, that out of about 450 tests, only about 18 of them were truly useful. It's a long story how that came to pass (basically the wise oak tree scenario) but the upshot of it was that we had a test suite that could, with high reliability, discover nothing important about the software we were testing. I've told this story to other test managers who shrug it off. They don't think this could happen to them. Well, it will happen if the machinery of testing distracts you from the craft of testing.

Again, I agree entirely with the sentiment, but I completely disagree when he indicates that automated testing was the culprit. This is not a test automation issue, it’s a test design issue. You could just as easily have run 450 manual tests that were poorly designed that would also tell you nothing useful about the code under test.

That it’s harder to write well designed automated tests is a given, but heck, as I stated above, why do we write any software? To solve some highly complex problems that would take us longer to do manually. Test automation, while having its own set of unique problems, is still, at the end of the day, just software that is attempting to verify the state of another piece of software at any given point of time.

When tests are correctly designed, and you are involved in the process early enough, there should never be a test that you’d only run once.

Engineering Excellence and TrustWorthy Computing

adamu — Wed, 22 Jun 2005 08:28:00 GMT

For the last several years, Microsoft has had days devoted to Engineering Excellence (EE), and to TrustWorthy Computing (TWC). Much of EE days in the last couple years have been about pushing quality upstream, which I interpret as getting more testing done sooner in the process, whether it is design reviews, code reviews, or unit testing, but ultimately before code is checked in. TWC has been about the pillars of privacy, security, reliability, business process integrity and transparency.

This year, they combined the two events into a single 3 day event, with dozens of in depth talks on a broad range of subjects related to EE and TWC. You can learn more about TWC here.

The keynotes were good, and the final keynote was by SteveB, who was as fun and spontaneous as ever. It is amazing to see how far we've come as a company with regards to these two key initiatives in just 3 years.

The most interesting thing I saw today was on Team Software Process (TSP) and Personal Software Process (PSP) that the BUIT has been using with good results. I am going to read a couple of Watts S. Humphrey's books on this and see if we can apply it to what my team is doing. More importantly, it got me to think about what we are measuring with regards to productivity (lines of code written) and defects injected. There was a quote used, something along the lines of "If you aren't measuring, you aren't engineering. I don't know what it is, but it is not engineering." If anyone knows the source of the quote, and the exact quote, please let me know.

Test Automation, plus what is it like to be in Test at Microsoft?

adamu — Thu, 16 Jun 2005 06:39:00 GMT

It's not sexy, but it sure is fun.

Santosh recently had a great post on what it's like being in test at microsoft. For those who read my posts from last year on campus and industry recruiting at microsoft, I would encourage you to also read this to get the sense of why being an SDET at Microsoft can be so rewarding.

Test Automation

Michael Hunter, aka the Braidy Tester, has had an excellent set of posts on my team's approach to test automation. It is essential reading, as will be his upcoming example series.

Preparing for technical positions at MS: part deux: Industry route

adamu — Wed, 07 Jul 2004 04:32:00 GMT

If you've been working (or idling) for a year or more since you hung up your full time academic cap and gown, the microsoft recruiting machine considers you an industry candidate. Many of my comments in the prior post about campus candidates will apply to industry candidates as well. In fact, I would say that ALL of it applies. They still do spare time coding projects outside work; they are constantly trying new technologies and reading to keep themselves technically sharp. The pick up on technology trends and can expound on where they think they should go. They are smart AND hard working. But for industry, the bar goes up (and it should!); they now have practical experience that should translate to a more polished set of skills.

Preparation

So back to how to prepare yourself. If you are a developer and you want an SDE or SDET position at microsoft, the best way to prepare youself is to write code, read code, dream code. There's no secret here. Have it reviewed by others so that you get better. You'll want to take a 4 pronged approach to what you code:

A solid general knowledge across the board in areas like file I/O, 2d/3d graphics, string manipulation, databound apps, whatever you haven't tried, broaden your horizons. You never know what kind of interview question you will get, and having a broad knowledge from which to drawn on will help you.
At least one area of deep expertise. Something that you've written tons of code in, and have much deeper knowledge of the problem space such that you could be a strong resource for your team in this space. You should be, above all, passionate, about this area, whatever it is.
Use multiple languages and learn the pros/cons of each. When you're asked one of my favorite questions “What's your favorite coding language“, you'll be able to answer most correctly “that depends on what the problem is.“ It goes back to the old saying, if you only have a hammer in the toolbox, everything looks like a nail.
Become an expert on your tools and leverage tools to make yourself better. Think about a master trade craftsman. If you asked him why he bought his DeWalt (or Ryobi, or Snap-on, etc, etc) tool, what kind of answer would you expect. Ask him what the advantages are with it over other tools he could be using. Why does he need a special diamond tipped wet saw? Because it ensures clean cuts on hard ceramic tile. For a software engineer, tool knowledge equates to efficient coding. Understanding what your tools are good at, and where you might upgrade demonstrates that master craftsman approach. Why do you use Lutz's Reflector? Why do you write all your code using EMACS? What do you hate about intellisense? How does late binding affect your coding? What 10 features do you wish you could add to your coding environment?

Interviewing

During the interview process, you'll be asked lots of questions, from coding problems to open ended questions designed to test your creativity, to fundamental questions like math/logic based puzzles designed to evaluate your approach to problem solving.

Coding: I would tell you to try writing code on paper or in notepad or on a whiteboard. For today's rad tool environments, intellisense coding is so helpful that it can be crippling you if you need to interview.
Creativity: I don't know how to develop this, either you have this or you don't.
Problem solving: I don't know of a single way to prepare for these type of things, except to have a solid mathematical foundation and be both creative and inquisitive. If you don't feel your math skills are up to par, upgrade them! Generally coding across a broad space can keep you sharp enough to solve these kinds of problems.

And of course, don't be afraid to ask what kind of question this is. Why is a man hole cover round? Is it a logic problem, or a creativity problem? That really depends on who is asking it.

Preparing for Technical Positions at MS: part 1: college campus route

adamu — Tue, 22 Jun 2004 21:23:00 GMT

Matt asked about how to prepare for a technical position at Microsoft. Since there are two separate recruiting engines at Microsoft, I’m going to break my comments up into college candidates and industry candidates to match the paths that people typically enter the company.

College Campus Candidates

I interview a lot of people at college campuses as well as during full interview loops at Microsoft since I took my current position 2 years ago. I've visited 7 college campuses and two foreign countries (Bulgaria and Romania) and interviewed around 200 candidates during those trips, and I have had about 20 or so campus candidates interview for positions during that same period. Obviously, a BS or MS in CS at a school with a good CS program is probably the best way to prepare yourself technically for any technical position at Microsoft. It probably goes without saying, but I'l say it anyway: regardless of your degree or your GPA or your class rank, not just getting the grade, but developing a deep understanding of the curriculum is what will best serve you. That GPA on the resume that’s reviewed by a screener at a recruiting fair, or seen by the schools recruiter, may get you 30 minutes with me, but you'll never get back to Redmond for a full loop if I see that you came out of school with a degree in CS and have gaping holes in your fundamental knowledge.

So back to how to prepare yourself. Here’s some of the things I see in the best candidates:

They are involved in coding activities outside school projects. Coding competitions, cross engineering projects like robotics with a mechanical engineer and electrical engineer, interns writing code for a company, highly involved in local programming user groups, teachers aide for coding classes so that you see a lot of other peoples code, freeware/shareware projects to learn about a new technology – perhaps building an app for a pocket pc to understand how it works, etc.
They keep up with today’s technology, are passionate about it, and can form opinions on where it is headed. A couple common interview questions are
1. “If I gave you $5000 to spend on any technology related items, what would you spend it on?” Be prepared to talk about all the hardware components in a PC if you want to buy one, what software and why you’d buy, and the advantages of your printer or camera of choice over others in the market.
2. “Given $10,000,000 dollars to invest in the tech sector, what would you invest it in?” Wireless, 3d graphics, display technology, what trends do you see, or breakthroughs are you dying for that you think would have a broad appeal?
I’m not sure how you go about changing who you are, but I’ll tell you that the best candidates I see are hardworking and fundamentally smart. I see a lot of 3.8+ GPA folks, and they work really hard to get the high GPA. But too many of them seem to be so focused on getting the grades that they failed to get an education, or they just aren’t that fundamentally smart (or it was a bad day; it happens to everyone), because in my interviews many don't do too well. I’ve talked to other recruiters for other companies, and this is a major difference between Microsoft and other companies. They tell me that the GPA will tell them that they know their stuff and can get things done. Well, for me, that’s half of it. I want that, sure, and I want to hire the person who will actually make us better as a company with their ideas in addition to their energy and effort.

What I look for when someone applies for STE

I mostly look for fundamentally smart people with a real passion for technology who can generate test cases and understand enough about how computers work to understand what code is doing. I also look for determination and inquisitiveness. Then I look at if this person can grow their coding skills. OK, they've never done any OOP, but can they code functionally? When I am making the decision for my team as the Hiring Manager, every STE I hire I believe that they have the coding skills to eventually move to SDET/SDE. Hey, I am in developer tools, I want every person I hire to be an expert on our product, and so I look for the potential to become an expert, which would mean being a developer. That is not the bar across the entire company, so when I am interviewing for Microsoft on campus, I look for the first part (fundamentally smart, real passion, understands computers at a moderate to high level), never gives up, and an inquisitive nature. I generally ask some testing questions, but because there are really few classes that teach how to test software, I am looking for aptitude, not ability. Red Flags: when I see someone come through with a degree in IS rather than CS, I drill very hard on their technical skills. Most seem to make that move because they aren't enjoying or just don't get the coding aspects. If this is the case, you will have a tougher time with me in making the STE bar. Not impossible, but it is more difficult.

What I look for when someone applies for PM

The best PM candidates have a MS/Phd in CS, excellent oral and written communication skills, understand the underlying technology, can build prototypes for proof of concept work, and are great negotiators. Pretty much, the best pm’s could be in Dev or QA if they wanted to be, but they have plus communication skills and a strong customer focus that they would rather focus on tying all the features together at a higher level rather than drilling into one area of technology. Warning signs: When I see someone with a BS in CS, then masters in business, and they want to be a PM. While I think that I understand the logic of this course, it rarely pans out well for candidates. They've lost any tech edge they may have had (they haven't taken a technical course in 2 years), and can’t talk about fundamental coding or design concepts any more. The other red flag: when people haven’t done the leg work to understand what the PM role is. You mean I can immediately manage people out of college? Wow, I want that job! PM's manage features and process, are not people managers (lead PM's would do both). Customer focus isn't enough in this role as well (I see many BS in IS trying for the PM role); you have to know your technical stuff too.

What I look for when someone applies for SDE/SDET

These people love to write code, live it, breath it, bring their laptop to the interview and are probably coding while their waiting for me to be done with the person in front of them in the interview queue. OK, that might be a slight exaggeration, but really, the best way to get technically proficient at a SDE/SDET core competency of writing code is to, well, write code. Lots of it. All the time. Candidates don't need to have perfect syntax, I'm not expecting that, but do they have a firm grasp on the concepts? Do their eyes light up as we talk about optimizations on a simple but elegant solution to a coding problem? Do I see that passion? For SDET, the bar goes up slightly in that I am looking for an additional skill. I look to see if they can test their own code. Sure, everyone tests their own code, right? It’s just part of the development process, right? Well, not exactly. Can they step back from the fact that they wrote the code, and look with a critical eye on their implementation? There are 4 types of developers from a QA perspective.

1. Can write code, can’t test other people’s code, can’t test his own code.

2. Can write code, can test other peoples code, can’t test his own code.

3. Can write code, can test other people code, can test his own code.

4. Can write code, can’t test other people’s code, can test his own code.

#1s don’t last too long at Microsoft. I characterize them in a very general way as either not enough experience or smarts to go beyond just writing code, or that they have such a huge ego that they believe that they never have bugs in their own code, and anything else being written doesn’t deserve their time. #2, they enjoy poking holes in other peoples designs, but have too big an ego to believe that there are holes in their own code. Ideally, we want to hire all #3 types for SDE or SDET, but we really need them in SDET roles. Why? Well, for test code, there is no additional organization that test’s the test code to ensure that it is working correctly, so these people are the last line of defense between the bugs in their code that cause test code to not uncover flaws in the shipping code. #4s are ok as SDE as well, they can test their own code, they just don’t help their peers that much via code reviews. Mostly, these people see one solution to a problem, and when people throw a different algorithm at them to solve a problem, they struggle grasping it, so reviewing someone else’s implementation is challenging for them.

I'll talk more about the industry recruiting route next time around.

You've got Questions, I've got answers.

adamu — Fri, 18 Jun 2004 12:14:00 GMT

I got some questions on my last blog post, so I’ll do my best to answer them now:

Q. How hard is it to get a job at Microsoft?

Well, that’s a tough question to answer. There are hundreds of variables there. If you know someone and they submit your resume as a referral because they know your work, and you do legwork on open positions that you feel would be an excellent match (and you carefully chose which ones you are best suited for, because recruiters dislike nothing more than someone who applies for 27 different open positions just hoping to get any of them), then you’ve significantly improved your chances at getting at least a phone call from a recruiter.

It also depends on if you are in school still, or have been out of school for more than a year, because the recruiting process is handled differently for ‘industry’ candidates and ‘campus’ candidates.

No matter which bucket you fall into, you should read Gretchen and Zoe’s blog to understand all about how recruiting works.

Q. How does it work at Microsoft for a job change to occur (such as Test Engineer to Development Engineer)?

The short answer: you would interview for the job. The long answer: one of the great things about Microsoft is the ability to not only move between teams working on various technologies, but move between job functions as well. All the job openings that we have (did I mention that I have 3 openings on my team?) are posted both internally and externally. If you were to stay within a functional discipline, you would most likely go through what we call an ‘informational’ interview process to see if the job is a good fit, etc. If both the hiring manager and the person applying for the position wished to continue the interview process, then typically a formal interview loop with several people would be scheduled. Sometimes this is waived, but that is strongly discouraged. For functional changes, a formal interview is mandatory.

Q. Is there a central body that determines if someone can code well enough to take a development job, or is up to the individual managers for the different teams?

Coding, while a very valuable skill to have, is just one of many skills necessary to succeed in many jobs, but you really shouldn't focus down strictly on coding, as you need to meet the bar across a broad skill set to succeed. There are people that can code, and have the interpersonal skills of a yak. But to get back to the question, there is no central body; the Hiring Manager for every job makes the final hire/no hire decision at the end of the interview.

Q. What does someone have to do to show he can code?

Again, that’s generally demonstrated during the interview process.

Most of these questions are focused primarily on a move from Test to Development. That can be a common path, but by no means is there any ‘norm’. I’ve seen dev to pm, pm to dev, test to pm, pm to test, dev to test, writers to qa, even moves into recruiting and marketing. Gretchen (and Zoe, who you should read if you are really interested in understanding recruiting and interviewing at Microsoft) made a post on how she loves what she is doing, and that she’s certain that there is another career here at Microsoft for her.

Of course my focus is QA, and as a company, we are working on retaining more senior developers in quality roles. In fact the Technical Lead in Test and the Test Architect positions within QA organizations are becoming more frequent as we build even more complex systems and the technical demands in QA grow proportionately with the complexity of the systems being tested. While the path into management is one road for growth in QA, (and I will readily admit that for some time it was the only path for growth) that is no longer the case.

I have a personal jihad of sorts about the type of people we need to hire into QA. I’ve spent my career at Microsoft in developer tools. In order to be an expert tester for a developer tool, you really have to be a developer. So we’ve always hired a lot of developers on the teams I’ve been on. On my current team, I hired all developers who can test (software design engineer in test, or SDET). Every single person on the team writes code; in fact, they write a lot of code. Sure, we write test plans to define what code we will write; we write design documents for our tools, and we do the occasional buddy test/manual test/bug bash thing. Mostly we focus on test automation for a variety of reasons. There’s lots of opinions on efficacy of automated vs manual testing, and I’ll leave that for a future post topic.

Why QA? Why Microsoft?

adamu — Wed, 02 Jun 2004 21:05:00 GMT

I came to Microsoft 8 years ago, happy but a little frightened about a salaried job, happy not to be travleing, and most of all, amazed that I got paid to beat on new technology and make it break, but my goal was to eventually be a developer here. I always loved building software for myself, and testing it would be a great way to get in the door, but eventually, I knew that I'd want to make the jump to development.

Yet here it is, 8 years later, and I'm a test manager. Why? Well, it wasn't an accident.

See, after about 2 years working with my Program Management and Development counterparts, I had an epiphany of sorts. I loved what I was doing; I mean I really loved my job. I was writing code, lots of it, as much as I wanted. My job was writing the code that tested the code we shipped. I got to be PM, Dev, and QA all in one over all the test code. Plus I got all the opportunities I could ask for in impacting the features we shipped. And as a Test Engineer, I'd written tons of specifications and dramatically impacted the shape of the features just by using the thing and giving my opinions, and by leaving the implementation details up to someone else, I was broadening my scope of influence. I'd even been asked to write shipping code a time or two because I was the last remaining expert on an existing code base.

So back to my epiphany. I realized that my job could be whatever I wanted it to be. If I took the narrow vision of a software test engineer or software design engineer in test's job and boxed myself into a job description, then perhaps I would have moved to development. But I realized that the job description is just a minimum bar; it's the stuff that has to get done to ship. But here is an employer that wants employees to go above and beyond. No, that's not quite right, Microsoft actively encourages them to do so; in fact, the entire compensation system is based upon you doing more than just 'your job'. I had the freedom to make my job more than just what was in the job description. You have great ideas, you have a strong vision and are willing to stick your neck out a bit (and not afraid to be wrong from time to time because your coworkers will let you know in no uncertain terms that you are wrong), the sky is really the limit.

So you're saying, Adam, that's great, but why is it that test can expand their job beyond their job scope, but dev and pm can't? Well, they can, but it's not as in sync with what they are already doing, so it's a tougher battle. For Dev, they have to get so technically deep in one area to code the features that they struggle stepping back to see the 'forest for all the trees' in most cases. When they can do this, typically the don't spend too long as a developer. They move to PM or to a Dev Lead. There's a well known saying at microsoft; if you want to know how a product works, ask a tester on the team. And it's mostly true. Dev generally know the feature the coded really well, but rarely step back and see how it integrates into the whole. And they are generally quite arrogant about it. Think I'm wrong? Check out this post by Jon Avner

PM, well, if you get PM to actually install the product, they could tell you how it was supposed to work when they wrote the spec. OK, so I am being a bit facetious; I have worked with some great PMs who were the first people to check a bug on the product when it came in. But largely, during the milestone while dev and test are coding, they have moved on beyond what's being coded and working on the next set of specs; using the product day to day and challenging the ideas they already put down on paper and got everyone to agree to is probably an akward step for them.

So, OK, Adam, I get that QA can have a broad impact, gets to write lots of code, and get's paid to poke holes in other peoples code! So why Microsoft?

Challenges abound. Just working at Microsoft is challenging; but working with some of the brightest people in the world every day is invigorating and fulfulling as much as it is challenging. For me in my current role, building and driving a QA organization and the entire product unit to stand for quality in every respect is both challenging and rewarding.

So I could work for lots of companies, but Microsoft is different. Here they encourage me to go above and beyond. While I've spent my entire career in developer tools, I have an impact on the quality of Office and Windows. I could move to games or portable devices, or tablet, or exchange or sql server and have a rewarding experience in any of those groups. The opportunities for moving between organizations to become reinvigorated by a product, well, I don't think there's anywhere in the world quite like Microsoft.

So here I am, 8 years later, doing my QA job, and still at Microsoft. And I wouldn't want it any other way.