Testing Testing 1,2,3

Check out this book!

This year fellow Microsofties who are photographer nuts from around the world spent five months assembling and publishing a world-class, fine art photo book.  The time and talent that went into this project makes every copy of the book a gift to its owner.  And guess what, one of my photos actually made it into the book. 

Here is the full book preview online (160 pages of preview!).  I should mention that if you purchase the book, 100% of the profit from sales goes directly to the United Way and you get a valued and timeless reminder of how you too make a difference for those in need.

You can preview the book, or order the book, from your desk.  Not only does this book make a wonderful addition to any collection, and a thoughtful gift, each copy you purchase contributes directly to the ongoing, important work of United Way.  This book is offered for purchase at the cost of printing plus $25.00. This additional amount is contributed directly to United Way in your name and is fully tax deductable as a charitable contribution.

Even if you don’t think you’ll buy a copy of the book, go take a look at the online preview anyway. It really is amazing!

Oh, and don’t forget to turn to page 114.  :)

As I was planning to do a team-wide presentation on Session Based Testing, I just thought I might as well use this opportunity to blog and share my own learning on Session Based Testing and at the same time prepare for the talk.  Killing 1 bird with two stones has been one of my repertoires (wait a minute….).  I do apologize in advance that I am not able to share out the actual slides since it contains some Microsoft Confidential data.  But I’ll do my best to summarize the main point here.

What is Session Based Testing?

Session Based Testing (SBT) is a form of Exploratory testing that is used for rapid testing, on project with tight schedule, or informal/incomplete/changing specifications.  It was developed by Jonathan and James Bach, and now being used in many QA organizations, including Microsoft.  Here is the original paper (PDF) written by Jon Bach defining SBT.  For our test team, Session Based Testing represents an “optimized, ad-hoc E2E testing, but at the same time very disciplined and focus”.  For us, it is being used to find bugs quickly and early in the product cycle.

Finding bugs very quickly, and very early is extremely powerful.  Let’s think about that for a minute.  Tradition test process which test plan needs to be written, follow by test cases, follow by numerous test passes, and then finally we start to see bug count going up.   Session Based Testing has a much more lower “cost of entry”.  It pretty much short-circuits the whole testing process with the goal of finding bugs.  The instantaneous feedback to the development team is extremely beneficial.  Now, everybody can gauge how well the product is performing event though it can still be under development.  Any bugs identified may also help shape the way code is written, or how the spec is defined.  It’s okay if developers do not want to fix these bugs right now (their usual counter-argument is always we are not code complete yet!).  But with these bugs staring at their faces everyday, it’s easy for them to realize that they need to crank out better code. 

I would want to point out that Session does not and should not replace the traditional test process.  SBT’s goal is to lower down the initial cost of  testing, and also increase flexibility and serendipity that makes Exploratory so useful.  IMO, the combination of Session Based Testing AND a well thought-out test planning is definitely a one-two knock out punches to the major bugs hiding within the code.  Every tester should definitely try it.

So, how can it be that SBT is ad-hoc and at the same time focused testing methodology?  This more or less bring us to the next topic – the difference between Sessions and Bug Bash.

Sessions VS Bug Bash

As I talked to a few people about SBT, the #1 question that everybody asked is -- “So… what is the difference between Session and Bug Bash?”  To the best of my knowledge, here is the similarities and differences between the two:

I am not saying that holding Bug Bash is a bad thing.  Bug Bash (and dogfooding) has been one of the staple diet for every testing organizations.  The more eyes looking for bugs, the better.  However, one of the main weakness of Bug Bash is that the whole process is very random and executed in somewhat ad-hoc manner.  Many times I would see everybody pretty much find the same bugs over and again.  In addition, since everybody is banging on the product at the same time, there is no good way to tell if the product has really been tested as thoroughly as it can, or if test executions are evenly spread out for all the features.  And in my humble opinion, the greatest sin of all is that testing knowledge is lost the moment Bug Bash ends. 

Session Based Testing is built on top of this same principles, but take it one step further.  Test owner should do a little bit of homework on defining what are the test areas that he or she would like to be tested.  And then the test owner will create a number of “sessions”, which typically last about 60-90 mins.  There is a clear goal or mission of what each session aims to do (e.g. “Test open file dialog in Notepad”).  But there is no execution steps defined on what steps to do, or how to go about doing that.  Tester essentially becomes monkey tester (with a focus) and let the magic of Exploratory testing kick in.

During session, the tester would need to note down what he or she had done.  More or less like micro-blogging or “tweet” if you know what I’m talking about.  :)  Could be something like this:

• I tried launching the Open File dialog using keyboard shortcut, and I found out that the hotkey to Open is “n” instead of the more traditional “o” (I just made this up).  We should keep the hotkey consistent with other applications.  Bug #123 filed.
• Dragging the Open File dialog window around, and this seems to work fine as expected.  Also drag the window outside of the screen and the window redraws itself back correctly.  (again just made this up)
• Tried opening file without extension, and Notepad crashes!  Bug #456 filed. 

Information such as these are quite critical because it provides a lot of insights into how well the product is doing.  Tester can report not only on just the issues that were identified, but also can report on the general feedback of the product itself.  This can even be opinions on what the tester thought, whether it works great, intuitive to use, great experience, or just plain out sucks.

Learning through Sessions

I think this is probably the biggest advantage that SBT provided – the knowledge that is associated with it.  Since every executed session will produce some form of data, we can then make use of those information to help us increase the quality bar.  Information gathered from sessions are really quite simple.  Here are some of examples:

• Number of sessions run
• When were they run
• Which target area were executed
• Number of bugs filed
• Any issues raised (that are not considered to be bugs)
• …

With this set of inputs, we can easily perform the following simple analysis.  In my team, we just plug the data into Excel and generate a Pivot Table out of it.  This way, slicing and dicing the data is just one click away.

• Bug density per each target area.
• Average bug yield per sessions
• Bug trend over a given period of time (Example: Total # of sessions and total # of bugs week by week for the past 3 month).  This is another way to pulse check on how good/bad the product is doing.
• Who ran the most sessions/who filed the most bugs

Somebody once said “A picture is worth a thousand words”.  So in order to save myself 2K words, let me present to you 2 pictures.  :) 

Analysis 1: Bug density per each target area

Row Labels Total Sessions Total bugs Feature 1 20 4 Feature 2 10 8 Feature 3 31 86 Feature 4 29 79 Feature 5 11 22 Feature 6 14 37 Grand Total 115 236

By looking at the graph, you can immediately tell that Feature 3 and Feature 4 are somewhat in trouble.  The total number of bugs are pretty high compared with the other features.  Using this data, you can start off investigating on why quality is lower than other areas. 

In addition, a lot of sessions were executed for Feature 1, but the bug yield is extremely low.  This might be a good indication that you might want to pull back on running more sessions this feature, and spend time on other area instead.

Okay, let’s do one more:

Analysis 2: Bug trend over a given period of time

Time Avg Bug Week 25 5.60 Week 26 4.31 Week 27 2.86 Week 28 2.68 Week 29 2.42 Week 30 1.32 Week 31 1.13 Week 32 1.50

This example just show the average number of bugs filed per week over a 2 month period.  It’s easy to see that the # of bugs are trending downwards, which is a good thing assuming that sessions are running constantly.  If you see spikes in the trend which is out of place, then it can be due to poor check in or some form of regression.  Inversely, if you see a sharp drop, it is something to look at as well.

I hope these 2 analysis help to illustrate some forms of measurement that QA team can derive and use.  Session Based Testing has certainly helped our team.  First and foremost, we are able to test creatively using Exploratory techniques, but at the same time, we are also disciplined in terms of relying on information obtained by running sessions to help us to be even more productive and efficient. 

That is a pretty bold blog title and absolutely had nothing to do with the Oscar itself.  But I had very little time last night to watch Oscar, and was somewhat curious about the winners and losers.  So I hit up Google and use this search term “Oscar”. 

Here is the front page search results.  Seems pretty plain old Googley.

So then I got curious and want to see how Live Search is doing.  And here is what is shown on the front page.  Immediately, all the major winner results are displayed, which is exactly what I wanted to see.  Impressive…

Oh by the way, congrats to Slumdog. 

I've been on many teams in my whole testing career, and for some reason, I always end up getting assigned to validate the email address field.  So after numerous times of hunting down my old email test cases from previous team, I am giving up.  I thought I am just going to list out all of them here out on the internet.  I know that I probably going to have to refer to this list again and again in the future.  Might as well help benefit others like me.

Here is the list of valid and invalid email format which can be used for testing.  The list is not comprehensive by any means, and is probably missing a bunch of edge cases.  However, it provides enough coverage for the majority of the allowed/disallowed email address according to RFC 2822.

BTW, if anyone else would like to contribute more cases to this list, or spot errors (fingers crossed), please feel free to let me know and I'll make sure the list is up-to-date.

List of people who help contribute to help solidify the cases

gabriel.lozano-moran

bernie halpin

Avdhut

1) Order t-shirts for the development team

2) Announce the release date

3) Write the code

4) Write the manual

5) Hire a PM

6) PM spec the software based on the code (to ensure the software meets the specifications)

7) Ship/Launch

8) Test with real users (free)

9) Bugs are potential enhancements

10) Announce the upgrade

Is your team practicing this?

My current team has been integrating this testing technique for a while now along with the traditional testing methods of manual testing as well as automation.  Most of the testing community that I have known seem to brush this testing technique off simply as “monkey testing” which doesn’t really provide any significant yield on bugs nor achieve high quality bugs.  However, I have been observing how powerful this testing method can be from my own perspective as well as by seeing the type of bugs that some of my team mates find. 

Recently, I went for a test talk on this testing topic and to my surprise there are a few other teams within Microsoft that actually use this testing technique as well.  Here are the short summary of what I did learn from that talk:

1. In summary, many testers still under-estimate the power of this technique and simply disregard it as another “ad-hoc” testing.  They think that this is simply ad-hoc testing or monkey testing where tester just randomly bang and shake the product until bugs are found.  However, this technique is actually proven to find critical bugs, cool bugs, and more importantly the kind of bugs that customers would encounter as they use our products.
2. On the other hand, if testers use Exploratory testing in unplanned, undocumented, random manner, then that can be a waste of time and energy. 
3. In order to maximize Exploratory testing, testers should sit down and come up with a strategy or plan on what testing techniques to use.  This helps keep the testers stay in focus, and also apply the right techniques to get the most out of an Exploratory testing session time.
4. Knowledge sharing is key to Exploratory testing.  Make use of historical data such as the type of bugs found in previous versions of the product, which features are buggy, customer-reported bugs, even knowing your own devs and their way of coding can also help uncover holes and bugs (human tend to repeat the same mistakes).
5. It’s quite difficult to determine what is the right balance between Exploratory testing and the traditional testing methodologies.  How much exploration should a tester execute in order to get the biggest bang for the buck.

Exploratory Testing Techniques

According to the speaker, here are the 4 key techniques that any testers can refer to and use for their exploratory testing.  Each component has its own strengths/weaknesses.  Tester do not have to stick with a single component, but should spread out and make use of all of them or a combination of them.  The trick is to know which one to choose and apply and when. 

1. Freestyle

This is basically Ad-hoc testing.  Hit the product from black box standpoint.  From the customer’s perspective.  Pros: Low cost.  Anyone can do it.  Encourage creativity.  Cons: May not yield the critical bugs.  Low quality bugs are typically found with lots of dupes.  Difficult to determine coverage.

2.  Scenario based

Using a pre-defined end-to-end scenarios, complete with actual reproducible steps and validation methods prior to testing.  Testers can use these scenarios as the base, and figure out new end-to-end path that will achieve the same goals.  Pros: Easy to trace the repro steps and follow.  Can determine test coverage.  Cons: Pre-defined scenarios may actually not represent what the customer really wants. 

3.  Strategy based

Testers do have some background of the product.  Thoroughly understand the architecture and flow of the product.  Tester can then leverage the product knowledge and combined with well-known testing techniques to go about testing.  More to come on the techniques later.  Pros: Test techniques are proven to be effective.  Testers just need to master them and apply them in the right situation.  Testers are also pretty much free to use their own instinct and creativity to go about testing the product.  Focus on learning.  Cons: Newer testers on the team may not have the sufficient experience or knowledge of the product to be completely effective. 

4. Feedback based

Start out pretty much same as Freestyle testing.  But as testing sessions are conducted, testers build up a history of test executions, areas covered, bugs identified, code churns, etc.  Very similar to our session notes.  Testing from historical data such as bug records, data from automations and scripts, application logs, customer reported issues, etc.  Use these data as input so tester can identify which areas may yield the most bugs.  Pros: Knowledge is not lost.  Make use of previous knowledge is very powerful.  Cons: Sometimes it’s very difficult to make use of existing data effectively (especially if there are a lot).  May actually spend more time gathering and processing these data rather than actual testing.

Okay, after a long sabbatical from writing blogs due to various reasons (over-worked, personal reasons, or just plain lazy), I am about to begin start blogging again.  My MSDN blog ranking has taken such an embarrassing tumble that I think I need to start hustling up again. 

So, to welcome myself back to blogosphere, what better way than to show off a bug in my code?  I just found it this morning, in plain sight. 

Okay, I know this is pretty lame, and probably not the first picture of a “bug” in the code.  In fact, the first ever recording of a bug in the software is probably this one here (according to various sources). 

This moth is probably the God of all bugs (well… it’s where the term bug came from).  Mine is just a tiny, teeny little baby spider, whom my next door office mate has now kept it in his jelly bean jar as the team mascot. 

Cheers, and hopefully, I’ll keep on blogging this time.

Stumbled upon this great one liner off Bash.org (via Reddit) just moments ago.  Made me LOL.

Today was an eye opener for me.  When I browsed through my Flickr account, I happened to notice the greetings-in-different-language on my Flickr home page.  You know the usual phrases "Kumusta ch4n!" (Tagalog), or "Hola ch4n!" (Spanish), or "Bonjour ch4n!" (French), etc. etc. 

However, I noticed something different today on that greeting.  "OH HAI ch4n!".  I just subconciously thought at first that's how Chinese greet one another, but then on second thought, that's actually not Chinese.  That got me to look closer at the text which told me what language this was and wham! it hit me.

It's LOLCAT language!!!  (not sure if the screenshot is clear enough)

What?!  The internet has evolved so fast that I didn't realize a new language was born completely out of weird cat pictures with funny-talk subtitles.   Happycat must be proud feline today.

Curiousity got a better hold of me and I start to Live search to see if this language really existed or was it just a bunch of cat-loving Flickr devs' doing.

That's when the second wave of shock hit me.  LOLCAT is truly a language on its own!  There's a English-LOLCAT translator, a CPAN Perl module which translate English to LOLCAT, and last but not least, a LOLCAT bible!!!

Incredible...

BTW, if you are wondering, this blog title is "Hello World" in uh.... LOLCAT.

Today is meant to be the power of two's day. 

2¹ => 2

2² => 4

2³ => 8

Putting together make 2/4/8 => 2/4/08.

Of course, back home in Asia and possibly in other places around the globe today is actually 2²/2¹/2³ which isn't as interesting.

k.  'nuff nerdinous for the day.

This year is going to be a memorable one since it's the last year that Bill is going to be with us 'softie full time.  In light of the upcoming retirement, I decided to read on his bio just to learn a bit more about him.  What I found out was somewhat amazing, especially his childhood years.

According to this Wikipedia entry (and also here and this book here), it seemed that our dear BillG actually started off his initial experience with computer as a   <drum roll>   tester!  Oh and he was a good one too.  Excerpt from the book.

Bill Gates, Paul Allen and, two other hackers from Lakeside formed the Lakeside Programmers Group in late 1968. They were determined to find a way to apply their computer skills in the real world. The first opportunity to do this was a direct result of their mischievous activity with the school's computer time. The Computer Center Corporation's business was beginning to suffer due to the systems weak security and the frequency that it crashed. Impressed with Gates and the other Lakeside computer addicts' previous assaults on their computer, the Computer Center Corporation decided to hire the students to find bugs and expose weaknesses in the computer system. In return for the Lakeside Programming Group's help, the Computer Center Corporation would give them unlimited computer time.

-- Wallace, James. 1993. Hard Drive: Bill gates and the Making of the Microsoft Empire

That's right folks, before Bill became a great entrepeneur, he was a great software developer.  Before he became a great software developer, he was a great tester.  (actually he was also a hacker in a sense).  The next time I hear a dev who say they never want to do testing, I'll shove this info up their throat.  (Watch out, you know who you are...)

:-)

BTW, here's the infamous CES 2008 video perporting Bill's last day.  I think everybody has already seen it at least once, but oh well.

 Happy Testing!

Some off topic from software testing here (as if I have a lot of 'on-topic', but well...).  Yesterday, a few friends and I decided to go crazy and wander around town for a quick photo shooting session.  The weather was cranky (cloudy and off-n-on drizzle) and cold, but since I just got a hand on my newest toy (D200 + VR 18-200mm zoom lens + 50mm f1.8 lens), I just had to go and test drive them out.

We ventured off back to Amazon.com HQ up on Beacon hill, where I was hoping to get a glimpse of the Seattle downtown with sun rise  in the background.  But since the weather was not co-operating, all I got was a typical gray and wet looking shot of downtown.  The pigeon was so curious that it had to check us out.

Then we decided to head up to Alki beach for even more gray and wet looking shots of downtown (but with the puget sound in between as a bonus).  The seagull was already there, but I was curious so I had to check it out.

And then came the highlight of the day, where we got smart and decided to go to Volunteer Park where there is an indoor flower Conservatory.  Temperature was a comfortable 70+ °F and the flowers were very pretty and fragant.  The pictures that I got were amazing (thank you D200!) and in the process I got to learn more about my new equipment.  Here are some samples:

More pics can be found at http://www.flickr.com/photos/65021437@N00/sets/72157603467276292/show/

Lessons learnt:

1. Initially when I got the camera, I was not able to get the sharpness out of the D200 compared with my trusty D70.  Another thing which I found out was also that the D200 gave a much more warmer and softer tone than the D70.  But then I finally took the time to study the manual and also happened to stumble upon a cheat-sheet at Nikonian.org.  This was a god send and gave me much more control over the camera.
2. The fixed Nikkor 50mm f1.8D prime lens was absolutely fantastic.  For the low low price of $109, it is definitely the best deal ever.  The picture was very crisp and sharp.  The only downside is I bumped into many strangers using my 'zoom legs'.  How did I live for so long without it was still a mystery to me.  If you own a Nikon DSLR, you gotta have it.
3. Hello ViewNx, Bye bye Photshop.  I have a CS3, but I just found it to be so heavy-weight.  My learning ability and concentration level has depreciated to that of my soon-to-be 4 year old kid and I just don't have the time nor effort to learn new software skills :).  I shoot everything in RAW format and with the ViewNx, I am able to edit my image in the same exact way I could do using the camera and I got to learn how to setup the camera (espeically with the white balancing) the next time I am in a similar shot.  In my opinion, post-editing using CS3 would be cheating :D
4. Patience is really a virtue.  I had to wait for almost 30 mins out in the cold for the sun to shine through the cloud for one of the shot with the statue looking directly into the sun. 

Any comments/feedbacks are welcome. 

I have always enjoyed watching motivational speakers giving out some speeches or tips on how to err... be motivated in life, or help others stay motivated, or maybe motivate others.  But most of the time the content of those speeches would stay with me for about a day, two at most, and then I would simply forget.

There's something different today.  I came across an article about a CMU professor, Dr. Randy Pausch, who's terminally ill with cancer (he sure doesn't look like it).  The professor gave his one final lector and very powerful lesson on life.  A little while later I managed to Live search around and found the video of lecture.  After watching it, I was a bit choked and I knew at the moment that this was going to be one motivational lesson that I would not forget.

"It's not about how to achieve your dreams, it's how to lead your life. If you lead your life the right way, the karma will take care of itself. The dreams will come to you."

-- Dr. Randy Pausch

What a statement.

-------------------

(update 11/21/2007)

The full lecture is at http://www.cmu.edu/homepage/multimedia/randy-pausch-lecture.shtml.  It's 1.5 hour long, but well worth it. 

I have been wondering for as long as I can remember why MSFT doesn't have a centralized forum for software testing.  But today, it seemed that testing god has answered that question as I happened to stumble upon this MSDN Software Testing Discussing Forum.  Although it seemed that the forum had just been started up quite recently, but the testing community both internal and external are growing quite rapidly.

Here's the excerpt of the goal of this forum:

"The goal is to create a community that answers test related questions and discusses test methodologies, tools, automation, best practices and other hot topics in the testing domain. We’re looking forward to constructive discussions where members share diverse views and experiences. These posts and discussions will ultimately help create a searchable knowledge base for the software testing world."

I really love the idea of cultivating ideas related to software testing among the testing industry.  But I am even more pleased that this forum represents another step towards cooperation between testers both inside and outside of Microsoft, helping one another to produce better quality software.

Thanks to MSDN and the forum moderators who made this happen.

 -----------------

Updated: Fix the URL to the brand, spanking new Tester Center.

I considered this evening to be pretty special for me -- it's one of those rare days where I could have dinner peacefully all by myself.  Not that I enjoyed eating dinner alone, but sometimes what a geek man wants is simply quiet time with a computer.  :D

At least that's what I thought to be until I notice that my CPU consumption start revving when I surfed through fine news aggregator site like Digg. 

I'm pretty sure I have nothing running in the background at all.  I'm staring at the CPU meter in the task manager start rising up like inflation rate.

(apologize for the sloppy screenshot)

As you can see, on my Vista desktop (w/ Intel Core Duo) I have reading from 3 different performance dashboards:

1. CPU meter (get it here).  This one read 53% for Core 1 and 42% for Core 2.
2. More CPU meters (default Vista gadget).  These two have the same readings -- 49% and 62% respectively.
3. Just so to balance things out, I also launch up ResMon (from Task Manager, go to Performance tab, click on "Resource Monitor" button).  This one displays CPU consumption of 50%.

So...what is going on?  Is something/someone trying to do something funny?  I tried to eliminate any doubt, so I quickly kill IE and boom, the CPU all went flat.  I repeated this exercise a couple of times to convince that this wassn't any type of worm or virus or trojan or swarm thing attack.  Each time I launched up IE and browsed to Digg.com, the CPU cycle started to take off again.  Going to any other sites had very minimal impact on the CPU.  Only Digg front page.

To cross reference and eliminate any remaining doubt, I tried Firefox.  The same high CPU usage pattern was observed again.

I looked deeper into the site, and the only suspicious thing about Digg is the new Sony ad (at least I thought it was new today).  Hovering the mouse over the ad revealed a bit more interesting functionality -- the mouse trails in a colorful droplet design.  Very cool indeed.

Unfortunately, it's probably this mouse trail effect which was causing the degrading performance due to its intense graphics (for its size) and event-based programming.  In fact, going to Digg Video section, which had different ad did not cause any resource utilization unlike the front page.

So, this message is probably for both Digg and the ad designer.  Having a cool ad is great, but please make sure it doesn't kill your (intended) viewers' machine.  Performance test is a good thing.