My team recently finished what we call a “bug bash.” That is, a period of time where we tell all of the test developers to put down their compilers and simply play with the product. Usually a bug bash lasts a few days. This particular one was 2 days long. We often make a competition out of it and track bug opened numbers across the team with bragging rights or even prizes for those who come out on the top of the list.
Bug bashes are a time when everyone on the team is asked to spend all of their time conducting exploratory testing. Sometimes managers will influence the direction by assigning people end-user scenarios or features to look at. Other times the team is just let go and told to explore wherever they desire. Experience has shown me that some direction can be good. Assigning people to explore an area they don’t usually work on gets new eyes on the product and with new eyes come new use patterns and new bugs. Recently I’ve also discovered that it can be helpful to track where people have spent their time. During our last bug bash we created a list of areas that should be explored and had people sign off when they had investigated them. This gives us a much better sense of just what the coverage looked like and allows us to ensure all areas received attention.
Conducting a bug bash can be expensive. There is a lot of work to get done and putting everything else aside for 2 days adds up to a lot of other work getting pushed off. Why do we do this? What is the return on the investment? There are three primary reasons that come to mind:
We have found that empirically, a bug bash flushes out a lot of bugs in a short period of time. Our most recent bug bash saw the number of bugs opened jump to 400% of the daily average. This is important because we frontload the finding of the bugs. The earlier we know about bugs, the more likely we are to be able to fix them. Knowing about more bugs also helps us make more informed triage decisions.
The second reason we conduct bug bashes is because they are likely to find bugs on the seams. Test automation can only find certain kinds of bugs. Exploratory testing is a much better way to find issues on the seams—where functional units join up. Sometimes these bugs are the most critical. Imagine if we could have found the Win7 MP3 bug or the interaction between playing audio and network throughput before shipping the respective products. These are the sort of issues highly unlikely to be found in test automation but which can be found through exploratory testing. We obviously don’t find all such issues through bug bashes, but we do find a lot.
The final reason we run bug bashes is to get a sense of the product. Most of the time we spend our days focused on one small part of the operating system or another. It’s hard to get a sense for the state of the forest while staring at individual trees. After spending several days conducting exploratory tests on the product, we can get a much better sense whether the overall product is doing well or if there are serious issues.