Test Guide

JMike's Favorite Bug

humbugreality — Thu, 09 Feb 2012 18:10:00 GMT

I found this bug while working as one of the first QA engineers at the MathWorks (makers of MATLAB and Simulink technical computing software) in the early 1990s.

ANSI standard 754 mandates rounding and such behavior for real finite-precision floating-point arithmetic. But there is no standard -- at least I can say for sure there was no standard in the early 1990s -- for extending those rules to complex arithmetic. So I did a fair amount of exploratory testing of MATLAB's complex arithmetic details on various operating systems, mathematical libraries, and machine architectures.

One basic mathematical rule is that a number times its complex conjugate is always real (has no imaginary part). So I wrote a little test that multiplied a bunch of numbers against their complex conjugates, and looked at the results. Nearly all platforms performed as expected. But the IBM RS-6000 would give Funny Answers: often there would be a teeny tiny little imaginary part, on the order of ten-to-the-minus-sixteen-or-so times the real part.

I showed the result to MathWorks chief scientist Cleve Moler, who speculated that it was a problem in how C's mathematical operations were compiled down to the RS-6000's instruction set. He sent the result off to the appropriate people at IBM, who quickly determined that that was true.

On most platforms, the part of the computation to determine the result's imaginary part compiled down to the following:

Multiply the first number's real part by the second number's imaginary part and store the result in standard 64-bit floating-point format.
Multiply the second number's real part by the first number's imaginary part and store the result in another 64-bit register.
Add the results of the two registers and store the result back in one of them.

But on the IBM RS-6000, the operation was fancier and exposed that they were carrying internal results around in longer (I believe 80-bit) registers:

Multiply the first number's real part by the second number's imaginary part, to 80-bit precision, and store the result in a 64-bit register.
In one shot, using the "multiply with add" instruction: multiply the second number's real part by the first number's imaginary part, to 80-bit precision, and add in the result from the register (which has been rounded to 64 bits), and return the answer.

See the problem there: the second half of the operation carried more bits after the decimal place than the first half did, so when you add them together, those low-order bits would be left, which would then come out as the imaginary part of the answer.

IBM fixed this issue quickly without a fuss.

This is my favorite bug because it demonstrated to me (and to others at the MathWorks) the value of hunting for bugs, it launched me into a long period of curiosity about finite-precision arithmetic (leading eventually to an advanced degree in numerical analysis), and, in retrospect, showed how a company could fix a bug that had significant impact on scientific computation without the widespread hue and cry that resulted from the classic "Pentium Bug" that dogged Intel a few years later.

-- J. Michael Hammond

Do you have a bug whose story you love to tell? Let me know!

Mohan Sagar's Favorite Bug

humbugreality — Thu, 15 Sep 2011 00:33:00 GMT

I am working as a SharePoint QA. We developed the Status Reports project. It is basically used to collect the weekly status for all the projects. You are allowed to submit the status for any project only if you are a member or owner of that particular project. Each project has a project category dropdown list. We successfully deployed it into production and got some change requests. After changing some functionality, I tested it in the dev environment and in production as well. But the real users were getting errors when they were trying to submit status for some projects. I just verified it in the dev environment, it worked well but in production it is still there. I started testing it in the production environment thinking the issue is with the production site.

Every time I have to follow below steps while testing in production:

Edit the Project list item and add myself as a Project Member/Owner
Submit the status
Remove my name after testing

I followed the same steps for the project for which the users were unable to submit status but I didn’t get any error.

ISSUE:

Actually the issue was, during the recent deployment in production, the developer messed up the Category field because of which the category had become empty for some of the projects. I could not catch this issue even though I tested in production as every time, I edited the project list item to add myself to the Project Member/Owner, then as the category is a dropdown list it defaulted to “X”, that “X” category was assigned to that particular project, and I didn’t get any errors while submitting the status.

After a detailed analysis I found it.

-- Mohan Sagar

Do you have a bug whose story you love to tell? Let me know!

CASTing About

humbugreality — Sun, 14 Aug 2011 01:30:00 GMT

Last week found me at CAST thanks to Jeff Atwood on behalf of Stack Exchange (now with a tester-centric site!). Thanks, Jeff!

You might remember I went to CAST last year and came back way impressed; this year's CAST managed to actually surpass my expectations! And it failed to meet them as well.

Location. CAST 2011 was up in Lynnwood, WA, just up the road from Seattle. In my mind this is starting to get into the boonies, but really it's still well in civilization. Plenty of four star (according to Bing) restaurants of all makes and models (including, apparently, a preponderance of Really Good Korean), its very own conference center, and hotels across a range of price points and amenities.

The conference hotels were fine, nothing fancy, not horrid either. The biggest problem here was that plural: there were *two* conference hotels, a 20-minute walk from each other. Not a horrible distance (I walk way more than that everyday), but far enough apart that it took coordination to decide when to meet who where. Just one hotel - or multiple in closer proximity - would have been nice. (CAST 2012, please take note!)

Food. I was impressed by the conference center's food: much better than I've come to expect from conference center fare. This would be a solid "Awesome!" were it not for the distinct lack of protein in *some* of the meals. Like about half of them. While (I guess) some people get by fine eating mostly carbs, I need pretty much a 50/50 mix between carbs and protein. Sandwiches had plenty of protein, nothing else did. (CAST 2012, please take note!)

Program. The program this year was created rather autocratically, with presentations being by invitation of the program chair rather than the typical call for papers. This resulted in a bit of an uproar, so an Emerging Topics track was added: shorter, 20-minute talks (as opposed to the 60+ minutes of standard sessions), suggested by and voted on by the CAST community. The shorter length resulted in a somewhat lower barrier to entry - while the community input on the speakers may have actually resulted in a higher barrier to entry. This was a great idea I hope is continued in the future! (CAST 2012, please take note!)

Sessions. The sessions were uniformly great - which is not to say every speaker was incredibly polished, which is exactly as it should be. CAST isn't so much about "Come hear me wow you with my awesome speaker skills (that don't give you a chance to get a question in edgewise)" as "Come hear me talk about this thing, and then let's have a facilitated discussion that's as much about your thoughts as mine". So while more polish from some of the presenters might have been nice (like, the microphone's there to be spoken *into*, not to hold by your waist), the discussions more than made up for any lack of polish. And - in my ultimate test of a talk - I came away from every talk with at least one new idea or question. So awesome!

Tutorials. I attended Michael Bolton's tutorial on Test Framing, so I can't speak to the others. In a fractal of the conference, I think this tutorial both met and failed my expectations: I finally understand what Michael means when he talks about framing tests, and every encounter I have with Michael ends up making my brain hurt in a really good way; yet it also felt like a partial rehash of Rapid Software Testing, and I spent about a third of the tutorial puzzled how it had anything to do with test framing. Overall, though, a good use of my time.

Questions. The questions I most remember - and am still pondering - are:

"How do I decide how much debugging I should do v. handing over to my dev?"
"Why run a test I'm never going to use results from?"
"What place do emotions have in testing?" (one of many coming out of my lightning talk on "Emotional Testing"; more details in future blog posts)

Quotes. Oh the surfeit of fantastic quotes :) My favorites:

Pretty much everyone, at one point of another: "All models are wrong, some models are useful"
Michael Bolton, in his keynote: "A tester's job isn't to confirm that a product works as expected, but to discover how it actually works"
James Bach, in his keynote: "Doing something just because you're told to isn't being an engineer"
Ajay Balamurugadas relating how a company asked Weekend Testers to test its product, they did and replied with a plethora of bug reports - and then never heard from said company again
Pete Whalen, in his Emerging Topics talk, suggesting doing integration testing "like your system's a character in Pulp Fiction, because all those 'unrelated' systems actually are"
Paul Holland quoting Paul Gerrard's modification of a quote from Michael Bolton: "Why didn't I find that bug? Because the developers hid it so well even *they* couldn't find it!"
Nancy Kelln amazing list of ("truly awful", to again quote Paul Holland) testing-related metrics
Sherry Heinze's understated "It's hard to test if you don't think"

Am I glad I went? Definitely! Even with inexperienced speakers, protein-light meals, and half of CAST being in a different hotel, this was the most fun - and learning - I've had all year. The keynotes, Emerging Topics, and (I think) Lightning Talks will soon be posted at CAST's blip.tv archive; I encourage you to check them out and get a taste of what could be yours next summer when CAST 2012 comes to Silicon Valley - hope to see you there!

CAST Away

humbugreality — Wed, 11 Aug 2010 16:30:00 GMT

Last week found me braving the Grand Rapids, Michigan summer (heat! humidity! sun! loved it!) to attend my first CAST. While I propped it a couple years ago, I didn't actually make it until now.

The short version: CAST is awesome, conference proceedings will soon be up at http://www.cast2010.org so you can see what you missed, you should go next year!

Longer version: Monday I participated in an all-day tutorial by Fiona Charles on Speaking Truth to Power. Most of this session involved role playing scenarios about telling an unwelcome message to people in power. Some came from our real lives, others were invented; all came scarily close to real situations we had suffered through at some point.

I found this session the most helpful of any I attended. We could halt the role play whenever we desired and discuss what had happened so far, then continue on or replay it from the beginning. This let us experiment with different approaches and see how they affected a scenario's outcome. Highly valuable!

I had three big learnings that day:

"Let me get back to you on that" is helpful in a multitude of situations.
Our emotions will hijack us when we least expect them to.
Changing the physical relationship between the people in a conversation will change the conversation.

Each of the next two days started off with a keynote and then continued with a full day of sessions. While one speaker turned me off when he seemed to make fun of waterfall ("Look at how long it takes!"), it turned out OK; every other session was quite good.

While the talks were interesting, the best part of every session was the facilitated conversations at their end. At least a third of every session was devoted to these discussions, and the CAST staff ensured that even the shyest introvert had a chance to talk without being steamrolled by the decidedly non-shy extroverts who always seem to dominate Q&A periods.

Extrapolating from this you might guess that pretty much everybody talked with pretty much everybody else throughout the conference, and you would pretty much be correct. Before sessions, after sessions, and into the wee hours of the night I had any number of conversations to choose from, and I took full advantage of these opportunities to meet new people, chat in person with people I had only met online, and renew friendships with people I already knew. For a person like me who loves to talk with people about testing, this was nirvana!

So, like I said earlier, CAST was awesome, conference proceedings will soon be up at http://www.cast2010.org, and you should definitely go next year! It'll be in Seattle, and I will definitely be there!

Talking Clocks

humbugreality — Fri, 02 Jul 2010 17:00:00 GMT

We have a set of clocks counting down the time until our next big milestone, one where all the testers are, one by the devs, and one by PM. I just noticed that they tell a story:

PM reaches the end of the milestone first;
Dev reaches it next;
And Test reaches it last

Isn't that how it always seems to be when it comes to software development?

I'm A Twit!

humbugreality — Fri, 25 Jun 2010 16:30:00 GMT

I'm on Twitter now; @humbugreality if you're interested. No promises whether I'll post there more frequently than I am currently doing here; so far I've twittered two jokes and am wondering what to say next.

Judging from the tweets I've seen, Twitter seems to be a combination Facebook + instant messaging tool where friending someone isn't necessary before you can see their updates; I don't see the point in that yet.

The main reason I haven't been blogging is that I'm not sure what I want to say. One thing I learned in my recent job search is that the act of testing no longer excites me - I'd much rather spend my time helping people it does excite do their job. And to answer the obvious question: I still don't want to be a manager. All of which makes my current role of Toolsmith (which inevitably involves much process-smithing) a fit while I figure out what I *really* want to be doing with my life now. (Which may turn out to be exactly what I'm already doing...)

Binging my way through the TED podcasts is helping me decide what I want to do with my life; at least, many of the talks feel important to me in a way I haven't yet put my finger on. More details when I have 'em.

As I've mentioned previously, my primary task right now is building a new automation stack for my team (~60 testers; all of Windows Phone Test eventually I'm hoping). While my thinking has evolved somewhat since I last blogged heavily about automation stack design, my first version here looked pretty similar. The more I learn about my team and their problems, however, and the more I realize that many of the reasons I'd previously done things in particular ways no longer apply (e.g., a lack of good model-based testing tools), the more my design here changes. It's becoming simpler, mostly, which I take as a positive sign. Once my design settles down I'll post more details about the process and end result.

In the meantime, know that I'm busy writing lots of code and standards and guidelines in hopes of helping my team do their job better!

Book Review: Mistress of Molecules

humbugreality — Tue, 25 May 2010 19:30:00 GMT

Last fall Jerry Weinberg sent me a review copy of his scifi novel Mistress of Molecules. I pretty much read it straight through, and then didn't blog for six months so never posted my review. So here goes:

I enjoyed it! Although I guessed the major plot points and some of the details well ahead of their occurrence, I always felt like "Hah! I *knew* that was going to happen!" rather than "Well, duh - what else would have happened?" And I was surprised by other details, including the manner in which what I guessed would happen actually did.

Jerry says he switched from non-fiction to fiction in hopes of finding a wider audience for his ideas and teachings. While I don't know how he's doing reach-wise, I clearly see his teachings in the puzzles I saw everywhere as well as the situations in which he put his characters. As Jerry helps Libra and Andre deal with their various life and work problems, realize (some of) what is really going on, and discover new ways to resolve their predicaments, I see clear parallels to his non-fiction books, PSL workshop, and AYE conference. I wish he'd been writing fiction when I was a teenager! Young adults feeling like square pegs being stuffed into round holes struck home for me, and I like to think his teachings would have seeped into my brain and helped me deal with high school and college a little better! As I know is happening for me as I read his books now.

As for those puzzles I mentioned, every character and place name, as well as every sequence of numbers, had me puzzling out its significance. Sometimes this pulled me out of the story, and while I *think* I know the answer for some cases others still have me stumped. Jerry?

What I love most about the best science fiction is that it can be read on (at least) two levels: the surface story, and the underlying manifesto. Mistress of Molecules fulfills that definition of science fiction in spades. Recommended.

Reliable Intent

humbugreality — Thu, 13 May 2010 16:30:00 GMT

I've been super busy talking with as many testers and leads on my team as I can, gathering data on what is and is not working for them. I'm completely unsurprised to learn that, as with many other teams I've talked with, they're having problems understanding what tests mean to test, finding and using functionality from other areas, and getting reliable results from tests.

Over the course of these discussions, everything I heard started grouping into two buckets: Intent and Reliability:

Intent: Our tests contain only the details that matter.
Reliability: Test failures always mean a product bug.

Intent harks back to the stack I helped build on Expression, although I'm finding my thinking around that is evolving (more details soon!). Reliability will come in part from reorganizing our current stack and merging our multiple ways to do <whatever> into exactly one that lives in an easily discoverable location. Beyond that, we're going to put some metrics around reliability to help us identify the most problematic functions, make them reliable, then move on to the next set. I'm sure we'll find problems in our test code, in our product code, in other teams' product code, and in the underlying device code. We have good working relationships with all of these groups, and a history of working through similar problems with them, so I know we'll get everything straightened out.

In related news, yesterday I purchased a data plan for my WP7 phone, so now I can read and respond to email even when I don't have wifi. I don't think I'm turning into a phonehead though - I haven't yet found a situation where I wouldn't have rather had a tablet I could ink on rather than use a tiny keyboard that makes my fingers hurt. Heeeeeeeeeeeere, tablet tablet tablet tablet tablet! (I hope I get a shark!)

In separate related news, I'm updating my You Are Not Done Yet checklist with mobile-related items and topics. If you have any, let me know!

Sometimes Sausage Making Is Beautiful!

humbugreality — Thu, 13 May 2010 04:30:00 GMT

Seeing Microsoft from the inside, I often don't understand what is going on. Sometimes, however, it's just beautiful!

Recently I discovered that VPN no longer worked for me, so I dropped by Security to see what was wrong. Therein ensued a journey of frustrating details, one I am still working through.

This evening I emailed the head of Microsoft HR. The _Head_, as in she reports directly to Steve Ballmer. I detailed my experiences and grumped about them.

She emailed me right back, asking for more details. And passed my mail on to the appropriate people asking for answers.

I love this company!

John's Favorite Bug

humbugreality — Mon, 26 Apr 2010 16:30:00 GMT

This particular one is more of a hardware defect than a software defect but it's a good story nonetheless.

I was working at a major company that had just gotten into the PC business. This was early in my career. We had been shipped a prototype of their new computer that was about to hit the market, so we were encouraged to "self-host" it by using it for daily tasks. Others had used it and had loved the experience, so I had sat in front of it in excitement, knowing that I was using a PC that wasn't even available anywhere else yet.

I started typing, but most of what came out on the screen was gibberish. For example, if I had tried to type "hello world", it came out as "je;;p wpr;d". How bizarre. I tried to open Notepad to do some testing, but of course I hit "run..." and ended up typing "mpte[ad". Hmm, time to find Notepad in the Start menu (which I never have had to do before!) Anyway, Notepad opened and I typed my usual touch-typing phrase:

Tje qiocl brpwm fpx ki,[ed pver tje ;azu dpg/

Whaaaa?! Was the computer possessed? I looked down at the keyboard and pressed each character deliberately. This came out:

The quick brown fox jumped over the lazy dog.

I scratched my head. Why would my usual touch-typing fail but when I pressed each character slowly, it worked? Was there some electrical flaw with the keyboard? Maybe a device driver couldn't keep up with fast keystrokes? I asked others who had used the PC, but they didn't have my problem.

Frustrated, I looked down at the keyboard and then, with a sense of dread, I saw it. My heart froze and I got a chill over me. I had known that thousands of these PCs were being manufactured at that moment, so I ripped the keyboard out of its socket and ran to our VP's office. I banged on the door and hurriedly explained the problem to our VP, who immediately called people higher up in the corporate food chain and got the manufacturing halted.

The issue? Every computer keyboard has a little plastic dimple over the "F" and the "J" keys. These dimples give a clue to your fingers where to rest your forefingers, so that touch typing is much faster. This particular PC had a defect; the keyboard dimples were over the "F" and the "K" key, not the "J" key. Naturally, when I touch-typed, every character on the right side of the keyboard came out wrong. But when I looked at the keyboard and hunted-and-pecked, the keystrokes came out right. The reason others hadn't had my problem is that they were hunt-and-peckers!

The moral of this story? Don't hesitate to blow a whistle and inform others of even the seemingly most innocuous flaw, no matter where you are in the corporate food chain. You could be saving the company millions of dollars. And for goodness sake, learn to touch-type! :)

-- John

Do you have a bug whose story you love to tell? Let me know!

Stressing The Matrix

humbugreality — Wed, 21 Apr 2010 17:46:00 GMT

Recently I saw a way to describe any particular stress test that I liked:

First, define the sets of qualities you care about:

Input
Execution entry point
Resources
Validation

for example.

Next, define the qualities within each set you are interested in; maybe:

Input

Concurrency of operations
Data quality / noisiness
Load / frequency of operations
Velocity / speed of operations

Execution entry point

Public APIs
Private APIs
User interface

Resources

CPU
Hard drive space
Memory pressure
Network flakiness
Duration of test

Validation

Application health
Assertions
Crashes
Data coherency
Deadlocks
Global expectations of system state
Live locks
Operation-specific expectations
System coherency

Finally, rate each quality from 0 to 10, where 0 means "This test does not target <quality>" and 10 means "This test is *all* about <quality>".

Now you have a point in a five-dimensional space (I'd give you a picture if I knew a good way to portray it!) that defines this particular stress test. Do this for all the rest of your stress tests and you can easily see how well you are covering the qualities you care about - and what tests you're missing!

I'm finding this framework helps me talk stress with people: Every person seems to have a different idea of "stress test", and this framework helps us clarify what exactly we mean by that. For example, I often hear dissension over the amount of validation stress tests should do: "Everything a functional test does!" "Nothing other than watching for crashes and asserts!" "Basic functional testing plus catching crashes and asserts!" These are three different types of stress tests, all of which can be useful - it depends on the intent of the test.

Which is perhaps the most important use of this framework: clearly defining the intent of that stress test you're about to write!

Denise Signer's Favorite Bug

humbugreality — Tue, 20 Apr 2010 19:00:00 GMT

Earlier this semester my capstone (= senior project) group wanted to print a 30-page Word 2007 document we'd been working on for a couple months. After struggling with the ink and paper issues of two printers, we finally held the document in our hands. But while we flipped through it, we noticed that two pages mysteriously were half blank. Page breaks had been added where there hadn't been any, which meant that our entire printout was messed up.

We looked at our screen and realized that those page breaks had been added to the digital version of our document as well. After reopening the document, everything looked just fine. When we printed our document again, the page break problem repeated itself. (Poor trees.)

We then took a closer look at where this was happening, and noticed that the page breaks were inserted where we had placed cross-references. Our cross-references turned from “<paragraph number> <paragraph text>” into “< 0.> <page break><paragraph text>”. Hitting CTRL+Z (or the undo button) fixed the error on the screen, but of course that didn't help us print. We ended up hand-typing the cross-references to get around this problem and meet our deadline.

I later figured out that the <paragraph number> was responsible for its own reset and the <paragraph text> had been inserting the <page break>. I could replace them with new automated cross-references and the problems went away.

-- Denise Signer

Do you have a bug whose story you love to tell? Let me know!

I'm Stressed!

humbugreality — Mon, 19 Apr 2010 16:30:00 GMT

David Treadwell once advised me to take a job where I knew what I was doing about fifty percent of the time and would be completely clueless the rest of the time, because that would be a good balance between comfort and extreme learning. Alan Page says to find the steepest learning curve. I’ve managed to satisfy both constraints!

The fifty percent I know is testing and solving testing problems. I’ve been doing both for a long time, and everyone (I’ve met so far) has the same problems – as well as utterly different ones. Which takes me to the other fifty percent.

Where my learning curve is completely vertical.

Devices, I am learning, is a whole different world from the desktop applications I am used to! A full build takes all eight of my cores four hours. Debugging managed code requires two debuggers. Missing ship dates means lots of device manufacturers and mobile operators lose revenue. My desktop has as much RAM as my device has total storage. And I am /way/ out of my depth!

One thing buoying me up while also dragging me down is Sync Stress. I always end up doing stress at some point, I'm plugging into an existing stress framework, and I’ve been doing (file) sync for the last three years, so I have some idea of what I need to do. The APIs I am writing against, however, are baffling me. In waves, no less. As soon as I grok one bit I discover another section that makes no sense at all. Then all of a sudden I grok that bit, and immediately discover another. So most days I go home with my brain aching from everything I learned that day, happy with what I accomplished that day and unhappy that it wasn’t nearly as much as I had hoped – and remembering I’ve supposed to have my strategy for our automation stack out for review tomorrow. Yikes!

My Very Busy CPU

humbugreality — Wed, 07 Apr 2010 21:30:00 GMT

Hmmm...I start a build and all eight of my cores gets busy - what's up with that? <g/>

How To Increase Your Chances Of Surviving A Microsoft Interview

humbugreality — Wed, 07 Apr 2010 16:30:00 GMT

I've talked with several people recently about how they can increase their chances of making it through their upcoming interviews at Microsoft. After noticing definite patterns in their questions, and in my answers, I decided to record them here where they may help a broader audience.

I'm so nervous! Yep! In my ten-plus years here at Microsoft I've been through forty-some informational interviews and close to ten full interview loops, and I still get nervous, and am sure I am doing horribly, and I am talking really fast, and . . .

Take a breath. Slow down. Be yourself. Yes, you may not have any idea how to solve the problem your interviewer just asked you. It doesn't matter. Unless you have a bad interviewer, what they are most interested in is how you approach the problem, not whether you come up with the best solution. More than once I've gotten stuck on "I know there's a better way to do this!", and so I'm standing there doing nothing rather than working towards a solution - *not* helpful in getting hired!

They keep asking me to write these algorithms I've never heard of! Tell them! I never studied computer science and so run into this all the time. Tell your interviewer that you aren't familiar with the algorithm, or concept, or whatever, and that you'll work through it as best you can. Remember, your interviewer wants to see how you solve problems. They can teach the particular programming language they use or the problem domain they are in as long as you can work through a problem you've never seen before.

I have to code?!? Maybe. It depends what job you are interviewing for. Dev or Test, definitely, in at least one interview and probably all of them. Program Manager, UI Designer, something other discipline, maybe - it depends on the group.

Will I be asked those stupid impossible-to-solve-unless-you-know-the-gimmick questions? Maybe. Personally, I *hate* them!

What else do I need to know? Be yourself. Ask questions. Explain your thought process. Ask for what you need. Remember: if it's the right job for you, you'll get it!

The New Guy

humbugreality — Thu, 01 Apr 2010 21:00:00 GMT

My first week on Windows Phone went great! Not that I did much of anything other than set up my new computer (eight processors! 6GB RAM! 1.3TB HD!), figure out how to build and run product and test code, and start getting acquainted with my new team.

My primary focus for the next year or so is our automation stack: figuring out how it works now, what doesn't work as well as we need it to (or at all), and developing a plan for incrementally taking us ever closer to the "ideal" automation stack my boss wants us to have. Everything I'm learning tells me the work I did on Expression and Mesh will be put to effective use here on Windows Phone. The first talk I gave had a warm reception, and I'm getting the feeling I have a receptive audience here - hooray! From what I've seen so far I think what is here already will flow nicely to where I'd like to take it.

In separate news, I'm shopping for a new cell phone plan so I can dogfood Windows Phone. My current cell phone is seven years old and most definitely *not* up to Windows Phone's hardware requirements, so I figure it's time for an upgrade. Any suggestions on carriers?

I'm finding it difficult to get back into a regular rhythm of writing - I stopped doing pretty much everything last summer, and while I remember having an endless supply of opinions to write about, my well has either dried up or not yet been primed enough. I'll be doing a plethora of writing and speaking as I roadmap our automation stack, however, which I suspect will get things flowing nicely!

I haven't played with Windows Phone enough yet to know whether it will transform me from a guy whose last cell phone bill was $0.16 into one of those people who never put their phones down; I'll let you know how it goes!

Raman's Favorite Bug

humbugreality — Fri, 19 Mar 2010 18:00:00 GMT

We were seeing weird issues for Microsoft Systems Management Server's (SMS) Software Update Reports like sometimes the reports would show a negative number of compliant machines or a negative number of machines where Updates had been downloaded. The customer requirement was that the number of computers in different compliance states should add up to the total number of computers that were picked up for a particular deployment and no negative numbers :) Now Software Update Data for the reports was coming from complex SQL triggers where we doing things like add 1 to # of machines to a certain state and subtract 1 from the machine’s previous state etc. The actual State Message generating and processing of state messages was all magic to us and another team owned that testing.

The challenge was that this issue would show up very infrequently and sometimes the negative numbers would become 0 or positive by the time we got a dev to my test machines; the repro for totals not adding up was comparatively easier. To dig deeper we wrote SQL Queries to find the anomalies in State messages that were coming into the SMS Server. There was our answer: many scenarios where Server got a message of Download Completed, but the Download Started message was dropped, as well as instances where a Success message would come before the Installing Update message. In the end we were able to write a query which returned 0 rows when life was good; anything else meant call the dev to investigate.

As it turned out, it was not just our feature. The underlying infrastructure bugs were found and fixed and that made SMS 2003 reports ready for End users.

-- Raman

Do you have a bug whose story you love to tell? Let me know!

Corey's Favorite Bug

humbugreality — Tue, 16 Mar 2010 18:00:00 GMT

My favorite bug was an issue that would cause a test hang on roughly 1 machine in 100 running overnight stress. All the test threads would get stuck making TCP connections, and we’d see the same memory signature in the kernel heap which indicated one of hundreds of kernel threads had written to a wrong memory location. Once in this state, a system reboot was required to get network connectivity again.

The developer for the area had tried instrumenting potential overflow candidates and as the tester, I had tried different testing angles to narrow down the culprit but we were making no progress. The product was a week or two away from shipping and general consensus was that we should probably just risk it.

Finally, in desperation, I just stared at the memory in hex for hours trying to discern what it might be and the answer came to me. It was a tick count and more than that, it was roughly the same tick count in all the failed machines. From that, we were able to pinpoint the source of the corruption in the DHCPv6 refresh timer, get the fix ready, and prevent hordes of customers from rebooting their devices to cure weird connectivity issues.

-- Corey

Do you have a bug whose story you love to tell? Let me know!

Status

humbugreality — Wed, 10 Mar 2010 17:30:00 GMT

2010 is going great for me!

2009, on the other hand, was the sort of year I wouldn't wish on anybody - and that I wouldn't give up for anything. The amount I learned about myself, why I do things the way I do, and what I could do about it - and what I have done about it - is directly responsible for this year going so well.

These changes have had ramifications in both my personal and professional lives. The personal changes have involved decisions I never expected to make, decisions I may talk about someday. Professionally, I realized how much I was taking advantage of work rather than respecting it, and how much I wasn't doing what my team needed me to do. I also realized how poor the communication between my management and me had been.

I turned all that around by remembering to ask "What are you expecting to see from this?", paraphrasing the answers I hear, and then actively updating my management of my progress against said work. As a result, I've completed an incredible amount of work, my features are looking good, I have our Windows functional, stress, and performance tests running against Macs, and - I'm leaving.

Not Microsoft (although I seriously considered that). Later this month I join the Windows Phone Communications Test Design Team, which is a fancy way of saying I’ll be helping them test new and revised ways to test WP’s messaging stack. I'll be mentoring some great testers, defining architectures to solve some interesting problems, and spinning up on devices - a brand new world for me. Woot!

I've been ignoring the blogosphere until recently, including answering mail. If you emailed me and haven't received a response - and still want one <g/> - email me again. Or find me at CAST - I'll be attending this year, and I'm looking forward to catching up with y'all!

Does this post mean I'm blogging regularly again? Not necessarily - let's see what happens, shall we? <g/>

John Mac Millan's Favorite Bug

humbugreality — Tue, 09 Mar 2010 21:00:00 GMT

I was testing a radar system that controlled the guns and missile launchers on a naval combat ship. The radar system consisted of a Fore subsystem, Aft subsystem and a Weapons subsystem. After several months of testing we discovered that if we started the subsystems in a certain sequence the Fore guns and Aft guns would reverse and point at the ship. The scary thing was this bug had existed for years without being detected! Fortunately all ships have mechanical interlocks to guard against any weapon pointing at the ship itself and committing suicide.

-- John Mac Millan

Do you have a bug whose story you love to tell? Let me know!

Sumit Kalra's Favorite Bug

humbugreality — Fri, 28 Aug 2009 02:30:00 GMT

Our application (web based) is built in .Net and Infragistics. It has 6 modules. One module is similar to product management, in this we need to enter product title, product details, etc. Newly created product names appear on the homepage. So while testing, in the "Product Title" field I entered this text: <script>alert ("test")</script> and then I saved the Products page. After saving an alert popup with text "test" appeared. And when I went to the homepage, the product name was not appearing but a popup with text "test" appeared. Whenever any user goes to the homepage, an alert popup appears. Then I insert a for loop in the product title and now the popup appeared 5 times. It was quite irritating =)

This is my favorite bug. Actually it is a security loophole (cross side scripting). Here one can call malicious scripts also.

-- Sumit Kalra

Do you have a bug whose story you love to tell? Let me know!

Thelma Whitehorton's Favorite Bug

humbugreality — Thu, 06 Aug 2009 03:00:00 GMT

The one that says that it is erasing my old email when it really isn't!

I like to erase old email. For some reason the web mail I am using will not allow me to delete my old email. One day I spent over an hour attempting to delete old email, when I realized the old email had not been deleted at all. "What in the world is this?" I spoke to myself. Then like lightning from on high, I realized my bug was human. What else could it be? Some one out there, wants my old email. So, I don't want it any more! Let 'em have it!

-- Thelma Whitehorton

Do you have a bug whose story you love to tell? Let me know!

Donny Liu's Favorite Bug

humbugreality — Wed, 29 Jul 2009 21:30:00 GMT

Our team uses a web-based tool to test our product. In one page, we can input tree-like structured data. In some conditions, we have to create a child under one node of that tree by clicking a drop-down list to select the type of the child object. The amazing thing is, we have to click many times to create the child successfully. We have to click, click, click, and don't know which time we can succeed, sometimes, it never works. After being confused for several days, one tester finally found the defect: just at the left side of the drop-down list, there is an invisible button! If we click that button by accident, we can create the child object. If we click carefully on the list, we will have no chance of success. Sometimes, carelessness helps!

-- Donny Liu

Do you have a bug whose story you love to tell? Let me know!

Sherry Chupka's Favorite Bug

humbugreality — Mon, 27 Jul 2009 23:30:00 GMT

In one application there is a small line of whitespace between a custom toolbar and the menu; clicking this whitespace causes an unhandled exception due to a null pointer. I found this by accident just trying to access certain options. This is my favorite because I like finding unexpected bugs.

-- Sherry Chupka

Do you have a bug whose story you love to tell? Let me know!

The Bug Craig Will Never Forget

humbugreality — Fri, 19 Jun 2009 19:30:00 GMT

Of all the bugs I've came across I'll never forget this one because it was so severe and easy to miss. Later, when I was a Test Lead it was the first thing I added to my sign-off checklist… A few years ago, our product was practically out the door when we learned that setup would fail if the computer clock was fast forwarded. Basically, a year or so after shipping the product would no longer install, because some time-bomb code remained in the setup custom actions. The team had performed timebomb tests. But did them by moving the clock forward *after*, and not before, installing the product. Easy mistake to make, if you aren't aware of the setup timebomb or that setup can have custom actions. For me, it's a good illustration of a common reason why teams miss bugs: knowledge/communication gaps.

-- Craig

Do you have a bug whose story you love to tell? Let me know!