Test Guide : Becoming A Tester

How We Test Software At Microsoft (Sometimes, Somewhens)

micahel — Wed, 22 Apr 2009 19:30:00 GMT

Some time back, Alan Page sent me a review copy of How We Test Software At Microsoft (HWTSAM), which he, Bj Rollison, and Ken Johnston published late last year. I was curious to read this and discover whether their view of testing matches mine. Now that I have finished HWTSAM I can say it more or less does.

Overall I liked this book, enough so that I believe every tester should read it. HWTSAM is chockablock full of details regarding fundamental testing techniques, strategies, and processes which I believe every tester should be familiar with (even if you disagree with the utility of some of them). HWTSAM is also chockablock full of guidance regarding when each of these tends to be useful and not so useful. And HWTSAM is chockablock full of insight into how many teams at Microsoft test many of their products, insight which has been gleaned from a plethora of successes and failures in testing said products and is likely to have at least some relevance to whatever you happen to be testing or developing. If you do not come away from this book without at least one new idea to integrate into your testing, I will be astonished.

I also disliked HWTSAM. For one, I continually stumbled over the editing (or lack thereof), to the point that I eventually started wondering whether it had been edited at all. Typos, seems-incorrect-to-me grammar, and "Why is that a note?" and "Why is that note here?" questions all took my attention away from the information Alan, Bj, and Ken were presenting. Similarly, as I read the "Software Engineering at Microsoft" and "Software Test Engineers at Microsoft" chapters I wondered what they had to do about testing; some time later I realized that HWTSAM is as much about process as it is about techniques, and that the way Microsoft's Test orgs tend to be structured and mesh with the other discipline orgs is an important factor in how we test and our successes and failures in doing so effectively.

I also found a wide disparity between the levels of coverage of various topics, with some discussions going into reference-book-levels of detail while other topics were given barely a paragraph. The sidebars often confused me as well, not so much via their content as by prompting questions regarding whether their were real or made up - while many of the stories were signed by their submitter, many others weren't, and I felt some of the unsigned stories made their point too well to be true. Finally, while the software-plus-services chapter presents lots of information regarding the testing environments Microsoft uses, my hopes for a similar level of detail regarding testing software-plus-services applications were not fulfilled.

In a few cases my dislike flared into violent disagreement. One cause of these flareups were the many absolute statements I noticed throughout this book, both about the utility of particular techniques, strategies, and processes and about how we do things here at Microsoft. I tend to bristle at the idea that anything always works or is always best, especially when it comes to testing, and especially when I know that statement to be false. Interestingly, other parts of the book clearly state that no technique or strategy or process always works or is always best, and no technique or strategy or process is used everywhere in Microsoft - another instance of bad editing perhaps.

Despite all of these dislikes, I enjoyed reading HWTSAM. I loved the stories about testing Microsoft software, the techniques used to do so, and the bugs those techniques found. Thanks to the "About Microsoft" chapters I finally understand our organizational structure here. I laughed with delight as I read how we now add so many servers to our datacenters every year that we have our suppliers deliver them in shipping containers which we simply drop in, plug in, and turn on. And I was happy to see at least one statement that manual testing is both useful and used here, giving lie to the common belief that Microsoft hates manual testing.

While other books go into more details regarding many of the topics this book covers, and other books are more even in their coverage of said topics, and other books provide bibliographies and other pointers to more details about said topics, I do not know of any other book that weaves all this information together into a story that describes how testing is done at a real software company. That story - the overall view - is the reason I believe every tester should read this book. Once you have, let me know your thoughts about it: michael dot j dot hunter at microsoft dot com.

Banking Expectations

micahel — Wed, 11 Mar 2009 19:30:00 GMT

For weeks now I have been seeing ads for a local bank. These ads consist of a vast expanse of white space disturbed only by the phrase "(This space intentionally left blank)". These ads baffled me.

Yesterday, however, I realized that the last word in these advertisements was not "blank" - it was "bank"! Now I understand the ads to be touting the bank as liberal, with all of the goodness (or not) associated with being left-leaning. Seeing the word that was actually there rather than the word I expected to see made all the difference between confusion and comprehension.

This and the string of other expectation-busting experiences I have had lately got me to thinking about the importance of expectation-busting in testing. It can be easy for us to fall into a complacency of expectations with regards to our products and our teammates. We expect that we can continue our testing from where we left off yesterday because nothing important has been checked in. We expect Mary to continue handling localization correctly and Martha to not, James to continue testing his features thoroughly and Jim to not even bother to compile his. We build piles of these often-invisible expectations about our world, and we act upon those expectations. Many times this makes our lives simpler; other times it trips us up.

I find it vital to be aware of my expectations. While expectations help me filter my world and allow me to pay attention to only what is new and different, they can also help me ignore newness and differences that does not fit my expectations of what new and different will look like - that is, to miss defects which do not appear where I expect them to. Every great tester I know is aware of where their expectations might be blinding them and continually questions those expectations to determine whether they are still valid. They allow their expectations to guide them; they never allow their expectations to blind them.

Becoming aware of our expectations and how they are affecting us can be difficult. I find that asking myself "What haven't I thought about today?" helps me uncover expectations I did not realize I had formed, a vital first step towards releasing them.

What expectations do you have about your product, your team, yourself? How do they help you or hurt you? Let me know: michael dot j dot hunter at microsoft dot com.

Testing Curiosity

micahel — Wed, 25 Feb 2009 20:30:00 GMT

What drew you to testing? What keeps you here? What would cause you to leave?

I fell into testing by accident. Autodesk needed testers for a new version of AutoCAD, I had registered copies of AutoCAD, they asked if I was interested, I said "Oh yeah!", they sent me test builds, I sent them bugs. I did not really know what I was doing, although I did know I was having fun!

This led directly to a position testing IntelliCAD for Visio, which was then still an independent company. My first day on the job my boss handed me a copy of Testing Computer Software and told me to get started testing. Scales fell from my eyes as I put what I learned from that book into practice. I discovered that I had been testing my entire life, and that I was good at it! Promotions came fast and furiously, as did bugs, and praise from my developers.

What drew me to testing originally, and what still draws me to it, is curiosity. Curiosity about how my program works. Curiosity about how my app is implemented. Curiosity about what was going through my developers' heads as they designed and wrote and tested their code. Curiosity about how well they translated their dream of our product into reality, and where their dream does not match with what our customers mean to accomplish. Curiosity.

Curiosity drew me to testing and is one reason I stay here. Curiosity about what interesting bugs I will find today. Curiosity about how I can help my team test better, and more effectively, and more efficiently. Curiosity about what makes my teammates tick and how that interacts with and affects my product. Curiosity about what makes me tick and how that interacts with and affects my product. Curiosity.

What would make me leave testing is realizing that I am no longer curious about these things. While I find that I am no longer interested in proving I am a great tester, I do still enjoy helping other testers become great. As I work with them we generally find bugs, which I still enjoy doing. And I still enjoy hearing about the excellent bugs my teammates and friends find. Curiosity is still present for me I believe!

What drew you to testing? What keeps you here? What would cause you to leave? Let me know: michael dot j dot hunter at microsoft dot com.

Falsely Responsible

micahel — Wed, 11 Feb 2009 20:30:00 GMT

Newbie testers often believe they can actually stop their product from shipping. Product teams often foster this belief by forcing their testers to sign off on their product before shipping it. Can you imagine the result if we decided to exercise our alleged power and elected to not sign off?

CEO: Why haven't we shipped yet?
Test Manager: Michael hasn't signed off on the build yet. He feels it needs more testing, that it still has bugs lurking in it. He says it will take him at least two more weeks before he will know whether he will be able to sign off, and when he will be able to do so.
CEO: I'm gonna sign Michael off!

There are so many things wrong with this picture!

Why are we testers supposed to sign off on the product when no one else has to do so?
Why does management pretend to give us testers authority we do not really have?
We do we testers let ourselves be put in this untenable position of false authority?
Why aren't we testers paid commensurately to our alleged power and responsibility?

The first I have never understood. The second, it seems to me, occurs when management cannot make a decision - at least not until someone else makes one for them to overrule! The third I find to be that we do not know better, or we do know better and are afraid to do anything about it. The fourth I believe ties back to the second: while management is happy to pretend to give us this authority, they aren't about to even pretend to give us commensurate remuneration, let alone actually give it to us!

If you are in this situation, you have at least the following options:

Exercise your purported rights and see what happens.
Refuse to exercise this responsibility and see what happens.
Join a different team, one that understands how to treat and use their testers.
Stay put and attempt to educate your management that making this type of decision is their job, and that your job is only to provide them with the information they need to make it.

If you are in this situation, what are you doing to get yourself out of it? If you ever have been in this situation, how do you get yourself out of it? Let me know: michael dot j dot hunter at microsoft dot com.

Heuristical Madness

micahel — Wed, 04 Feb 2009 20:30:00 GMT

Recently I was thinking about the plethora of Test Heuristics floating about the blogosphere. HICCUPPS, CRUSSPIC STMPL, FCC CUTS VIDS, and on and on and on. Eventually I became overloaded and started searching for a heuristic to help me sort through and remember all of these heuristics!

Then I realized I already had a sorting heuristic: me! I am a sometimes fallible solution to the many different problems I encounter every day. Sometimes I work in solving those problems, and sometimes I do not - I am fallible. So it does not make sense to get down on myself when I fail; heuristics fail sometimes, and I am a heuristic. It does not matter if I do not always bring the results someone else is expecting; heuristics do not always bring the results we expect, and I am a heuristic.

Wouldn't it be grand if CRUSSPIC STMPL could speak up and say "Um, actually, San Francisco DePOT would be more appropriate right now"? If all of these testing heuristics could walk in to our offices and test our products for us? That is in fact what they do, in the person of me, and you, and everyone else on our teams. If we are heuristics, then so are our developers, and our program managers, and even our managers! Which means each of us and all of us will fail at times.

Anyone who expects CRUSSPIC STMPL to find every bug is mad. Likewise, anyone who expects you or me to find every bug is mad. Even if that expecting person is you or me!

Heuristics fail sometimes - that is their nature. As each of us is a heuristic, we will fail sometimes. We will be asked to do things for which we are not well suited. We will be expected to bring about results which we are unable to do. And we will fail. No big deal, such is the life when one is a heuristic.

As a self-aware heuristic, it is my duty to speak up when I feel I am being misapplied. It is also my duty to help those using me to understand how to best apply me. As I do, I must remember that those using me are also heuristics, and so they will sometimes fail in their roles as well. If I expect perfection from them, then I am the one who is mad.

What are you doing to discover how you are best applied, and to help you teammates understand this? Let me know: michael dot j dot hunter at microsoft dot com.

What Do You Do?

micahel — Wed, 10 Sep 2008 19:30:00 GMT

Do you know the unique value you add to your team? I used to think I did, until a series of unfortunate events showed me that I did not. That set me off on a splurge of introspection regarding what I do and why I do it. Now I believe I really do understand the unique value I add to my team. Here is how I reached this point, and how you can reach it yourself if you are not already there.

Some time back I realized I was unhappy with my job. I was not enjoying it anymore, except for a very few parts. I always intend my job to be play, and I never see any reason to continue in one which has become work, so I started looking for a new job. After searching awhile I found a job which I felt fit the bill and I moved.

After a while, though, I realized I was unhappy with my job. I was not enjoying it, except for a very few parts. I had not been there for long, and I wondered what had gone wrong. Had I not understood what I was getting into? Had the job changed under me? Had I changed under the job? I decided to find answers to these questions before I started yet another job search, lest I jump to yet another new job and find myself in this situation yet again. I am happy I did, because I discovered that what I thought I wanted in a job wasn't what I really wanted in a job!

I started by listing everything I liked and disliked about my current and previous jobs. Next I examined my past performance reviews and pulled out what seemed like key words, terms, and themes. Then I asked a small number of people whose opinions I trusted to send me the five words they felt best described me. Then I waited.

Each time someone sent me their five words I added the words to my Who Am I And What Do I Want To Do With Myself List, then went back to waiting. Once I had responses from everybody I entered a cycle of reviewing the list, mulling it over, reviewing it again, and mulling it over some more. Eventually I started to see similarities in and themes across what I liked about jobs I had had, what I received rave reviews for, and the various words people felt described me. I organized the data, reorganized the data, and rereorganized the data. Stronger themes emerged. I built affinity diagrams from the data. Yet stronger themes emerged. I realized that although I enjoy developing I am not a developer; that although I enjoy testing I am not a tester; that although I enjoy mentoring and coaching and leading and teaching, I am not any of those things either. I realized that I am a me, and that I work in ways that seem to be different from how people seem to expect me to work.

I decided to pursue this last observation further, on the theory that if people aren't expecting something from me, that something is probably a unique something I add to my team. I considered how important partnering is to me, how I prefer pairing with someone over working alone for most everything I do. I also contemplated how important alone time also is to me, how I need time away from everyone in order to process what I take in and synthesize it into something new. I started calling this my Cycle Of Conversation, and I realized that it was present every time I had enjoyed myself and that it was absent every time I had not. A-ha!

This and other key learnings eventually became my statement of the value I add to a team. Johanna Rothman told me this is a very NT statement. Michael Bolton said it helped him understand me yet did not at all help him understand what I do. What I do, I now knew, is not nearly as important to me as how I do it. As long as I am working with people to solve problems that matter to them, and doing so in the fashion I describe in my value statement, I am happy.

Michael's comment reminded me, however, that not everyone in the world has NT preferences. (Most do not, in fact.) Bearing that in mind, I drew up a (probably still rather NT-ish) list of the things I do. Double-entry bookkeeping helps accountants ensure they have not missed out a ledger entry, and describing the unique value I add in two different ways helped me ensure I had not missed out an aspect of my job that was important to me.

Several revisions later I felt these two items described me accurately. People who know me well agreed. At that point I applied these items back to the unhappiness I was feeling about my job.

I talked with my manager about everything he wanted me to do, everything I wanted me to do, and how each of those projects did or did not match the way I work. While my unhappiness with my job did not change overnight, I did feel better simply for having told my manager that I was unhappy, why I was unhappy, and how I wanted to change my job so that I (believed I) would be more happy. Starting that discussion was a first step towards making my job what I wanted it to be.

I talk with a lot of testers. Rarely do I talk with a tester who is completely happy with their job. As I help them explore their unhappiness and where it comes from, oftentimes we discover that they do not understand the unique value they add to their team, nor do they understand how they want to be adding value to their team. As we define these statements and find ways to change their jobs to better match what they want to do, their unhappiness with their job invariably starts becoming happiness with their job.

Which brings me back to the question I asked at the beginning of this post: Do you know the unique value you add to your team? Do you know the unique value you want to add to your team? What are you going to do about defining it, and then about making it happen? Take action and let me know what you learn!

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.

Games Testers Play

micahel — Tue, 19 Aug 2008 19:30:00 GMT

What are you favorite testing games?

I learned Headline News from Elisabeth Hendrickson many years ago, and it has since become one of my favorite games. The idea is to dream up headlines involving your application or feature which you really do not want to ever see, the more dramatic the better. For example: Chief Executive Impeached After Faulty Security Allows Millions To Read Her Snarky Comments About World Leaders. I find playing this game to be a fun way to identify the risks and issues about which my product team cares most.

Another game I quite enjoy is Why Would Anyone Do That?, which I find to be useful in uncovering hidden assumptions I have about my product. To play this game, pick some random statement, requirement, or fact about your product. For example: The Xygon234 cell phone will be powered by a standard BLL battery cell. Now spiff on that requirement to an extreme and consider the ramifications. For example: What if the customer removes the battery while the phone is in use? What if the customer leaves the battery in its charger for two solid months ? What if the customer leaves the battery out of the phone for two solid months and thus depletes the phone's backup battery? I find discussing why someone might do "that" to be useful even if we ultimately decide that no one ever will do "that".

A third game I like to play is Hit The Widget With A Hammer. If your product involves hardware, this might involve an actual hammer. More typically, however, it involves a (programmatic or manual) loop that hammers one particular input to your program hard. This might mean clicking a button thousands of times in quick succession, blasting millions of values through a particular API, or repeatedly setting and unsetting a particular configuration value. I find areas that fall over when I hammer them often also fall over when I use them more gently.

These are a few of the games I use to help me jump start my testing. What games do you play? Let me know!

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.

Tester Center Tester Spotlight: Ivo Manolov

micahel — Wed, 13 Aug 2008 23:30:00 GMT

I have another Tester Spotlight posted! Pop over to Tester Center - via Channel 9 - to see my interview with Ivo Manolov, Test Manager for the Windows Presentation Foundation team.

Nike Testing

micahel — Wed, 06 Aug 2008 19:30:00 GMT

When I started writing this blog I had no idea what I was going to write about. I don't remember that ever happening before - usually I know exactly what I want to talk about and more-or-less how I want to say it.

When I test, however, I find myself in a similar situation all the time: I sit down for a bout of testing and discover that I haven't a clue what to test or how to test it. For example, I currently own the entire Macintosh client for Live Mesh. How do I get started testing that?

If I don't know where to start, it probably doesn't matter where I start so long as I get started. Once I started writing this, I picked up momentum and eventually discovered that I had a pile of words - i.e., this blog post. Other times I discover that I have nothing, in which case I start over with a new first word. In both cases I learn about what I do or do not have to say about my topic.

Likewise, once I start testing I pick up momentum and eventually discover that I have a pile of defects to log. Or I discover that I have nothing, in which case I move on to a different area or switch to a different testing strategy. In both cases I learn about what I can and cannot say about the state of my application.

One reason I believe this Simply Get Started Already strategy works for me is that doing something - anything - gets me into gear, which I find to be a prerequisite for slipping into Flow. When I am in Flow, information flows effortlessly and I know exactly what to do, where to do it, and how to do it. If I'm not where I need to be, the information guides me there.

Until I start analyzing.

What is the best word to use here? Does that sentence flow as well as it could? Should this be one paragraph or two?

That isn't the functionality I meant to invoke. What's the proper order to go through this checklist? Would five objects be better than six?

Are these really the best examples I could give?

Are these repro steps really as short as they could be?

While I do not want to ignore these issues, neither do I want to sidetrack myself out of Flow. I find that editing myself is a surefire route out of Flow, whereas writing down the words as they come to me, and executing the tests as they come to me, without regard to their correctness or perfectness or Best Optionness, tends to take me deeper into Flow. By keeping these two states of consciousness separate, I can better focus on and stay in each one, and I ultimately make more forward progress and come up with an end result more to my liking than when I attempt to do everything all at once.

How about you? What do you do when you don't know what to do or where to start? Let me know!

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.

Rhymeless References

micahel — Mon, 21 Jul 2008 19:30:00 GMT

If you want to learn more about the Myers-Briggs Type Indicator, personality types, neuro-linguistic programming, and representational systems, I suggest you start with the following. (Full disclosure: I have no affiliation with any of these websites, companies, and authors.)

Personality Types

TypeFocus Careers has a free personality survey.
AdvisorTeam has several free and not-free personality surveys.
I find What Type Am I?, by Renee Baron, an accessible introduction to and explanation of the Myers-Briggs personality types. It includes a personality survey.
Please Understand Me II, by David Keirsey, describes the personality Temperaments Keirsey built on top of the Myers-Briggs personality types. It includes a personality survey.
Teaching Around the 4Mat System, by Bernice McCarthy, explains one way to integrate personality types into education.

Representational Systems

Discover Your Preferred Representational System has a free representational system survey.
Influencing With Integrity, by Genie Z. Laborde, briefly covers representational systems.

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.

Finally! The Finish!

micahel — Fri, 18 Jul 2008 19:30:00 GMT

So that is MBTI, and that is NLP.
Which still leaves the puzzling, mysterious A&P.
I know some about "A" and much less about "P";
Your doctor knows much more, that I guarantee!

"A" is for "anatomy", where you learn to see
How to tell your elbow from your hand and knee.
"P" describes your insides, your "physiology"
How it works and functions, why you smell and see.

If this introduction leaves you wanting more;
If these words, you feel, have opened a big door;
I urge you these topics to go and explore.
Only don't go too fast - your brain it might get sore!

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.

Trio Of Tastes

micahel — Wed, 16 Jul 2008 19:30:00 GMT

Visual people learn with their eyes.
They peek and they look and their world clarifies.
"I see what you're saying!" they exclaim with a cry;
Use picture words back else you'll be in disguise.

Auditory people learn with their ears;
Tho some choose to read rather than hear.
"Sounds good to me" lets you into their mind peer;
Use hearing words back and your meaning is clear.

Kinesthetics are the third group you'll find;
They touch and they feel life into their minds.
"You hit a sore spot" says one of this kind;
Use sensing words back else they will feel blind.

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.

New Lingo Please!

micahel — Mon, 14 Jul 2008 19:30:00 GMT

NLP is a science (tho some say it's not)
Which claims to explain how people come to a thought.
Neuro Linguistic Programming can help you avoid being caught
In the communication perils with which life is fraught.

This science holds that people think, say, and learn
In characteristic ways, and that you can in turn
Identify their style and go on to discern
Words to use with them that won't make them burn.

Listen to people's words as they speak.
Watch their responses when you open your beak.
Do they talk about hearing? Or taking a peek?
Or maybe of feeling. Then your words you tweak.

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.

Timely Play

micahel — Fri, 11 Jul 2008 19:30:00 GMT

"J" is for "Judging", though not the critical kind.
It simply describes the way we unwind.
Or don't, for we often put our play far behind
Our work which we want to be completed on time.

"P" stands for "Perceiving"; however I call it "Play"!
For us Ps care for naught but having fun all the day.
"Structure"'s a word we'd rather not say!
We don't mind it, you know, yet we keep it at bay.

Decisions! Decisions! We Js want them made now!
While delaying decisions is us Ps' solemn vow.
By working together we turn furrowed brows
Into great outcomes that make us yell "Wow!"

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.

Feeling Logical

micahel — Wed, 09 Jul 2008 19:30:00 GMT

"T" is for "Thinking", so called because we
Decide with our brains, not our feelings.
We use our logic to decide and to see
Exactly how we want our world to be.

"F", on the other hand, logic deplores
At least when it means pushing people down to the floor.
While "Feeler"s do use logic, it leaves them quite sore
For emotions and values are what form their core.

Ts, it would behoove us to come down from our tower
And come to know those feelings from which we so cower.
Fs, use your logic, and you'll find the power
To talk with us Ts without getting sour.

*** Want a fun job on a great team? I need a tester! Interested? Let's talk: Michael dot J dot Hunter at microsoft dot com. Great testing and coding skills required.