David Fries' WebLog : Software Testing

Know the bugs in your area

davfries — Fri, 23 Jul 2004 20:23:00 GMT

This is a simple one, but easy to overlook. I'm going to make the assumption that you are familiar with bug reporting, and that you have some sort of bug tracking database.

Bugs (as in the bug reports, not the actual product mis-behavior itself) are just another way of communicating information. There may have been times in your past when you were not included in an email thread which concerned a feature you owned. In this case, you'd probably go to whoever sent the email, and complain about not being brought “into the loop”.

Let's say that someone else testing another feature finds a bug in your area. With bugs, there is only one addressee (the person that the bug is assigned to). As a tester, you don't really want this bug to get assigned to you while it's still active, but you would like to at least know about the bug. The solution? I use a program called Bug Watcher which looks at the bug database every 15 minutes or so, and tells me if there were any changes to any bug matching a given criteria. In my case, I want to know about NGEN bugs, so I look for any bugs that have the word NGEN in the title. This is great because I am able to really get a good feel as to what are the current issues in the area. Sometimes I can also play around in the area where the bug was found, and find another bug or two. This is also a great way to reduce the number of duplicate bugs.

Monitoring bugs in your area is a great way to stay “in the loop.”

Virtual Server 2005 makes software testing more effecient

davfries — Wed, 23 Jun 2004 18:18:00 GMT

While that sounds like a press release tagline, it's the honest truth.

For those of you not familiar with Virtual Server 2005, read this review or the Microsoft Virtual Server site

Some of these features are not unique to Virtual Server, but are useful nonetheless.

Simultaneous Virtual Machines: I can run many Virtual Machines at the same time, all on one Virtual Server, as long as I have the RAM available to support it. This is nice because as I'm doing various tasks throught the day (bug investigations, test development, test runs), I can have more machines than I actually have to do these tasks.
Web Interface: This is really handy for letting others use my virtual machine, as well as for accessing multiple Virtual Servers from one machine (I barely log into one of my machines. I just access the Virtual Machines it's hosting through the web interface). I had an interesting experience yesterday as I saw 2 different developers debug an issue on my machine. Previously, this meant the developer would be sitting at my machine, preventing me from getting any work done on that machine. Now it simply meant that I could watch them debug using the web interface, while I carried on business as usual on the REAL machine. It's easier to set up than remote debugging, as well.
Better than Imaging: As one who has tried many different solutions for maintaining multiple OS's on the same machine (dual/triple boot, Power Quest Drive Image), this is by far the most efficient. Not only is fast to switch between machines (since I can run them simultaneously), it's also great for having a wide array of OS's available at my fingertips, even if I don't have the RAM available to have it running all the time. And when I'm done dirtying up the machine with my tests, I simply roll back to the “pristine“ state. Goodness.
Fully Utilizing Processors: In my testing, I do test runs that only utilize one processor, even though I have a dual proc machine. I also have several different “flavors“ of this run, where different configuration options are set. Using Virtual Server, I can fully utilize the second processor by having the runs running on separate Virtual Machines. Virtual Server (and the OS) take care of the load balancing for me (although there are options to tweak this behavior)
Undo Disks: It's easy as pie to make updates to the Virtual Machine. Simply go back to the “pristine“ state, add your updates (say, from Windows Update?) and tell Virtual server to commit undo disks. Now your “pristine“ state has the latest updates. This is much less painful than maintaining a drive image with the latest updates.
Library of VM's: One of the test leads on my team has developed a Virtual Machine library, where I can simply copy a zip file to my machine, unzip it, set a few config options, and voila! I've got myself a new Virtual Machine. Multiply this savings by everyone who uses the Virtual Machine library, and it's easy to see how virtual machine technology can help your bottom line.

There is not much that I dislike about Virtual Server, but there are a couple of issues:

Lost Changes: Sometimes, when the machine is rebooted, or there's a power outage, all the stuff that I was working on is lost. The next time I start the VM, I will be in a “pristine“ state again. This can be pretty painful, depending on what you're doing. Maybe this was just a bug that I hit...
A Little Too Easy: I mentioned that it was easy to make updates to the “pristine“ state. Well, I once accedentally clicked on the wrong item in the menu, and my “pristine“ state was now not so pristine. I had to destroy the Virtual Machine and start from scratch (which was not so bad, thanks to the the VM Library). Virtual Server warns you every time you're about to delete data, but it doesn't warn you when you're about to update your “pristine“ state.

Overall, the benefits I see far outweigh the cons. I'm still a beginner when it comes to virtual technology, and I can't wait to learn more about it.

Pairwise Testing for Dummies

davfries — Fri, 18 Jun 2004 03:38:00 GMT

So, imagine that you're a tester, and your job is to test a pogo stick.

Yes, a pogo stick.

So, you sit down, and brainstorm (like I did just now) a list of things that you might try to do to break the pogo stick, while still within bounds of “normal” use. (Running over the pogo stick with a cement truck would certainly break it, but this scenario is not covered by the warranty)

Let's imagine your list of ideas looks something like this:

Pogo in the rain
Pogo off a platform (1 or 2 feet)
Pogo in sand
Pogo under varying weights (50, 100, 150 lbs)

Now that we have our list ready, let's look at our testing approach

A relatively simple approach would be to simply go down the list, and write a test case for each variation.

Pogo in the rain
Pogo in the sun
Pogo on level ground
Pogo off a 1 foot platform
Pogo off a 2 foot platform
Pogo on pavement
Pogo in the sand
Pogo with 50 lbs
Pogo with 100 lbs
Pogo with 150 lbs

This would certainly do a fair job of testing things, but what about testing for certain problems that appear only when certain combinations are used (Say, pogo in the rain off a 1 foot platform, or pogo into sand under 150 lbs of stress)

In order to exhaustively test every possible combination, you'll need to design a test matrix like so:

Weather	Height	Surface	Weight
Sunny	0 ft.	Pavement	50 lbs
Sunny	0 ft.	Pavement	100 lbs
Sunny	0 ft.	Pavement	150 lbs
Sunny	0 ft.	Sand	50 lbs
Sunny	0 ft.	Sand	100 lbs
Sunny	0 ft.	Sand	150 lbs
Sunny	1 ft.	Pavement	50 lbs
Sunny	1 ft.	Pavement	100 lbs
Sunny	1 ft.	Pavement	150 lbs
Sunny	1 ft.	Sand	50 lbs
Sunny	1 ft.	Sand	100 lbs
Sunny	1 ft.	Sand	150 lbs
Sunny	2 ft.	Pavement	50 lbs
Sunny	2 ft.	Pavement	100 lbs
Sunny	2 ft.	Pavement	150 lbs
Sunny	2 ft.	Sand	50 lbs
Sunny	2 ft.	Sand	100 lbs
Sunny	2 ft.	Sand	150 lbs
Rainy	0 ft.	Pavement	50 lbs
Rainy	0 ft.	Pavement	100 lbs
Rainy	0 ft.	Pavement	150 lbs
Rainy	0 ft.	Sand	50 lbs
Rainy	0 ft.	Sand	100 lbs
Rainy	0 ft.	Sand	150 lbs
Rainy	1 ft.	Pavement	50 lbs
Rainy	1 ft.	Pavement	100 lbs
Rainy	1 ft.	Pavement	150 lbs
Rainy	1 ft.	Sand	50 lbs
Rainy	1 ft.	Sand	100 lbs
Rainy	1 ft.	Sand	150 lbs
Rainy	2 ft.	Pavement	50 lbs
Rainy	2 ft.	Pavement	100 lbs
Rainy	2 ft.	Pavement	150 lbs
Rainy	2 ft.	Sand	50 lbs
Rainy	2 ft.	Sand	100 lbs
Rainy	2 ft.	Sand	150 lbs

36 possible combinations... OK, but here's the really bad news. In order to make sure these tests are conducted professionally, you need to hire a professional Pogo Jumper (Boss's orders). Unfortunately, the Pogo Jumper's schedule is quite full (as one would expect for a world-renowned Pogo Jumper!), so he can only stop by on Tuesdays, and it takes him all day to go through one scenario from above (with all the stretching, pogo calibration, and self affirmation exercises involved).

Hmm, this means it will take you 36 weeks to test the pogo stick.

Not good.

Not good at all.

Here's where the notion of pair-wise testing comes in... The basic idea is that most problems (bugs) manifest themselves with combinations of 2 factors. Once you start trying every combination of 3 or 4 factors, the test matrix quickly explodes (as we can see from the table above), and you don't find as many bugs as when you were testing combinations of 2 factors. Clearly a case of the law of diminishing returns.

So, let's take certain rows from out of the table, and make certain that each factor is matched with each other factor at least once. (Rainy should be tested with 0 ft, 1 ft, 2 ft, Sand, Pavement, 50 lbs, 100 lbs, 150 lbs at least once, and the same thing goes for Sunny. Then we must verify that each height is matched with each type of surface, etc. etc.)

Here's an example of a pairwise matrix derived from the exhaustive matrix above:

Weather	Height	Surface	Weight
Sunny	0 ft.	Pavement	50 lbs
Sunny	0 ft.	Sand	150 lbs
Sunny	1 ft.	Pavement	100 lbs
Sunny	2 ft.	Pavement	100 lbs
Sunny	2 ft.	Sand	150 lbs
Rainy	0 ft.	Sand	100 lbs
Rainy	1 ft.	Pavement	50 lbs
Rainy	1 ft.	Sand	150 lbs
Rainy	2 ft.	Pavement	50 lbs

So, now we only need to rent out the professional Pogo Jumper for 9 weeks. We've just cut our testing matrix to 1/4 of its original size! And we will still catch any problems that happen with any combination of 2 factors. This really is a great way to think about problems when there are just too many combinations to test exhaustively.

The best way to create a pairwise matrix is to use a tool, although it's possible to do it manually as well.

AllPairs is a free tool to do this.

Want a better explanation? Go Here: http://www.developsense.com/testing/PairwiseTesting.html

Still curious? http://blogs.msdn.com/micahel/archive/2004/04/28/122702.aspx

Want even more? http://blogs.msdn.com/nihitk/archive/2004/05/09/128821.aspx

And for those of you that are here for the Pogo Sticks: http://www.post-gazette.com/healthscience/20010528bowgohealth2.asp

Dave