Software Sleuthing : Testing

Magic Numbers: Integers

joshpoley — Fri, 09 Oct 2009 20:37:00 GMT

After many moons of sitting in a queue somewhere, the MSDN Tester Center just posted my intro article on special numbers you probably want to think about when running tests. The paper addresses several interesting domains of integers (all the way up to 64-bits) and gives you samples of important values to use:

In testing software, we often miss an entire class of bugs just because we don't happen to think about one category of inputs and how they might adversely affect the system. To help combat this in with my own tests, I've pulled together a listing of numbers which are good to use in testing for various independent reasons.

http://msdn.microsoft.com/en-us/library/ee621251.aspx

Test Plan Template

joshpoley — Fri, 02 Oct 2009 20:51:00 GMT

While looking at some of the traffic statistics for the MSDN Tester Center, I noticed that a bunch of people arrived at the site as a result of doing a Google search for "test plan template". The primary starting point for a template is really the IEEE standard 829-2008 which breaks apart the various types of documents and goes into detail on what should be present in each. In the Xbox software organization, we've taken the Level Test Plan and tweaked it to better fit our needs, but a (generalized) version of it is posted for your viewing pleasure: Word Template (dotx) or PDF.

The Brute Tester

joshpoley — Mon, 01 Jun 2009 20:47:00 GMT

A co-worker of mine was working on a tool to watch our release share for the appearance of a test result file, for which he would then publish up to a TFS server. The "trx" results file in question is actually being created on the Xbox console by our custom test harness, and historically was around a 10 megabyte (Unicode) file for a typical daily regression run. Yet on a Friday afternoon (when it actually happened to be nice outside), I get the following email from the tool writer:

Any idea why daily\testResults.trx is 1.10GB? It’s too large to load and publish.

I'm thinking to myself: "One gig? Ay corumba! Did something go into a bad loop or something?"

I go out to the release server and start digging through the various log files, and sure enough, that thing is huge. Luckily we also produce a little rollup by component area which is written out to a text file and only contains the totals. The previous day's totals came out to 6422 tests for this suite. Today's number: an unorthodox 454,216. Yea, I guess adding 447,794 test cases will definitely blow out the size of a TRX file.

Now to dig a little deeper. The four hundred and forty-seven thousand test cases were just in one feature area. Looking at the test code it was definitely following a brute-force pattern over multiple axis, the first instance I noticed was O(n³).

Cranking through half-a-million test cases for a single feature is probably not the best use of your time (or the computer's). And I would hate to be a developer asked to run this suite in order to isolate a failure down deep in a nest of loops ("sorry Joe Developer, but it starts to fail around the 300,000th test case, good luck with that breakpoint"). Instead you should be trying to pick the fewest (reasonable) cases that adequately test the component. Here are some ideas to get you started:

· Use Equivalence Classes/Partitioning. You can even use Code Coverage to help determine groups.

· Use All-Pairs or Pairwise pruning techniques (Microsoft's PICT tool can be downloaded for free).

· Use data generation (e.g. Fuzzing) or Data-driven techniques to increase your coverage over time. You don't need to run every single variation all at once, instead you increase your surface area a little bit each day.

Upcoming Testing Conferences (2nd half of 2009)

joshpoley — Wed, 13 May 2009 20:20:00 GMT

Here is my semiannual post on testing conferences that will be occurring later this year.

CAST	July 13 - July 16	Colorado Springs, Colorado, USA
ISSTA	July 19 - July 23	Chicago, Illinois, USA
softec	July 20 - July 22	Kuala Lumpur, Malaysia
STANZ	August 24 - August 25	Wellington, New Zealand
STANZ	August 27 - August 28	Sydney, Australia
Practical Software Quality and Testing	September 14 - September 18	Minneapolis, Minnesota, USA
SSQC	September 15	Zurich, Switzerland
CONQUEST	September 16 - September 16	Nuremberg, Germany
SSQC	September 24 - September 25	Stockholm, Sweden
SSQC	October 5	London, England
StarWest	October 5 - October 9	Anaheim, California, USA
QA&TEST	October 21 - October 23	Bilbao, Spain
Pacific NW Software Quality Conference	October 26 - October 28	Portland, Oregon, USA
expo:QA	October 27 - October 29	Madrid, Spain
SSQC	October 28 - October 29	Melbourne, Australia
Great Lakes Software Excellence Conference	November 16 - November 17	Grand Rapids, Michigan, USA
EuroStar	November 30 - December 3	Stockholm, Sweden

Upcoming Testing Conferences (1st half of 2009)

joshpoley — Tue, 04 Nov 2008 19:56:00 GMT

Here are the links and dates for some of the testing related conferences which are approaching.

Future Test	February 24 - February 25	New York, New York, USA
Swiss Testing Day	March 18	Zürich, Switzerland
ANZTB	March 23 - March 25	Sydney, Australia
SEPG	March 23 - March 26	San Jose, California, USA
STP	March 31 - April 2	San Mateo, California, USA
ICST	April 1 - April 4	Denver, Colorado, USA
QUEST	April 20 - April 24	Chicago, Illinois, USA
CITCON NA	April 24 - April 25	Minneapolis-St. Paul, Minnesota, USA
ITNG	April 27 - April 29	Las Vegas, Nevada, USA
PSQT	April 27 - May 1	Las Vega, Nevada, USA
StarEast	May 4 - May 8	Orlando, Florida, USA
ICEIS	May 6 - May 10	Milan, Italy
CITCON ANZ	June	Brisbane, Australia
Better Software	June 8 - June 11	Las Vegas, Nevada, USA
CAST	July 13 - July 16	Colorado Springs, Colorado, USA
ISSTA	July 19 - July 23	Chicago, Illinois, USA

If you want to get a jump start on conferences in the latter half of 2009, you can browse through the ones that were around in 2008. And thanks to Alan's comment on that page, I'm now pushing these through the Tester Center site as well, so they should be showing up there in the near future.

Testing when "the code is the documentation"

joshpoley — Thu, 04 Sep 2008 01:03:00 GMT

Throughout the course of my testing career, I've often heard this dreaded statement. Despite the causes behind it, I think that there will always be times when you just won't get any formal documentation or specification on how a piece of software is supposed to work. Not to mention the equally evil (or perhaps more so) times, when the documentation is completely obsolete and doesn't represent reality.

When I run into this, my work commonly falls into the following pattern consisting of four different phases.

Investigation

Here I spend some time upfront to get an understanding of what the software currently does and how it acts. A good chunk of time is used reading through and reviewing the code itself. I will also typically write a very basic (straight line) test making it easier to trace through the code in a debugger, which gives me a better feel for the program flow and organization.

Write the tests

Once I have some understanding of what the code is supposed to do, I then start cranking out test cases. Throughout the investigation and while I'm writing and running the tests, I'm also communicating with the developer (and the PM if there is one) in order to answer specific questions about the behavior.

Conflict resolution

After I've written a chunk of tests, there are undoubtedly a number of areas where a bug may or may not exist depending on someone's point of view. Since I've written my tests based on my perceived conception of how the software should behave (and the developer has done likewise with the code) there will be areas where my test expects one behavior, but the program does something else. And without a specification to arbitrate, this often becomes a heated battle. Especially since the developer has already written the code (and is undoubtedly working on something else now), he/she will be reluctant to go back and change it based on the whim of a tester. Thisis where your persuasion and diplomacy skills come into use as you need to make a case and argue on behalf of the customer.

Documentation

After finishing up the tests and the stability of the software has solidified, I typically go back and provide some form of documentation. At the very least I ensure to comment specific weirdness along with my tests. But depending on the area, I may also provide some high level information to co-workers. For example, after working with the (undocumented) custom file system used on the Xbox, I created a basic table outlining the characteristics compared to other common file systems. This included information useful to my colleagues such as maximum filename length, maximum filesize, etc. At the other end of the documentation scale, you could always go and write the specification yourself, which would definitely help to improve the maintainability of the system.

Security Tools: Codenomicon

joshpoley — Wed, 27 Aug 2008 00:58:00 GMT

About three years ago, when the Xbox 360 was getting close to launching, we went through a security pass of the audio and photo playback capabilities. One of the tools that was recommended to us by another employee was the Codenomicon suite of test files. The folks at this company have taken the time to pick apart some of the typical file formats and produce an extensive library of files which target the specification and common implementations in an attempt to push the limits of your software. Unlike random fuzzing tools, these guys have gone in and hand crafted the files to exercise your implementation in a systematic and precise manner. And one of the major benefits over fuzzing is that you actually get documentation for each test case describing what is interesting about the file.

After running the files through the Xbox's code base, we actually found a couple of edge cases that weren't being handled correctly and warranted a fix. One of the interesting bugs was around an integer overflow that led to memory corruption. We would do some size calculations which resulted in a 32-bit unsigned integer overflowing to zero, which was then passed down to malloc. The cool (from a twisted perspective) thing about malloc(0), is that it actually allocates zero bytes of memory for you and hands back a valid pointer. But unfortunately for us, we would then go and try to write to that pointer which doesn't take very long before something horribly bad happens as we trash memory.

I think it is a fairly accurate statement that you can never do enough testing, and you should definitely try to use all the tools at your disposal. The Codenomicon suites are one such example, and something that not very many people I've talked to have heard about, so it is certainly worth investigating for your product.

Developer Tip: Add a blank macro so testers can inject code

joshpoley — Mon, 11 Aug 2008 19:39:00 GMT

While I was working on writing some API tests for a module that was being integrated into the Xbox Dashboard application, there was some rework to the class that happened in the middle process. In doing so, the developer decided to make a bunch of the functionality private, causing my existing test to fail to build (as they no longer had access to anything). Luckily the developer left me a little cookie in his code:

#ifndef CONNECTION_POOL_TEST_FRIENDS

#define CONNECTION_POOL_TEST_FRIENDS()

#endif

class CDdlConnectionPool

{

...

CONNECTION_POOL_TEST_FRIENDS();

};

By providing a blank macro inside the class, he gave me the opportunity to modify the class without having to change the header file. I just have to define the CONNECTION_POOL_TEST_FRIENDS macro from my test code before including the above header file, and I can inject my own friendliness:

// enable the API test to be a "friend" of the connection pool

#define CONNECTION_POOL_TEST_FRIENDS \

friend HRESULT Pool_InitializeTest(void); \

friend HRESULT Pool_AddEntryTest(void);

#include <dashdl.h>

I like cookies.

Upcoming Testing Conferences (2nd half of 2008)

joshpoley — Wed, 30 Jul 2008 23:25:00 GMT

I was talking with a test manager yesterday about recruiting opportunities, and he was wondering about what testing conferences were coming up in the (fairly) near future. So here is a listing of some of them:

STANZ	August 11 - August 12	Wellington, New Zealand
STANZ	August 14 - August 15	Sydney, Australia
softec	August 27 - August 28	Kuala Lumpur, Malaysia
Practical Software Quality and Testing	September 8 - September 12	Minneapolis, Minnesota, USA
World Congress for Software Quality	September 15 - September 18	Bethesda, Maryland, USA
QUEST	September 22 - September 26	Toronto, Ontario, Canada
Software Test & Performance Conference	September 24 - September 26	Boston, Massachusetts, USA
CONQUEST	September 24 - September 26	Potsdam, Germany
StarWest	September 29 - October 3	Anaheim, California, USA
Pacific NW Software Quality Conference	October 13 - October 15	Portland, Oregon, USA
Test	October 13 - October 16	New Delhi, India
QA&TEST	October 29 - October 31	Bilbao, Spain
Great Lakes Software Excellence Conference	November 4 - November 5	Grand Rapids, Michigan, USA
EuroStar	November 10 - November 13	The Hauge, Netherlands
expo:QA	November 27 - November 28	Madrid, Spain

Update 7/31/2008: Added more conferences.

Know Thine Product's Language

joshpoley — Wed, 25 Jun 2008 18:36:00 GMT

As a technical tester (SDET), you are expected to help find bugs that are written in the programming language used by your product team (SDETs commonly contribute by writing test code and tools). To this end, you truly need to understand and master that language (especially if working with inexperienced developers). As such, your test code and tools should be written in the same language as the product code. In fact, let me Noguchi Rage that:

Your test code and tools should

be written in the same language

as the product code.

Some of the very best testers I've ever worked with are awesome developers and know many of the languages' vulgarities that often plague others. Unfortunately, I've also seen a lot of testers spend their time writing tools in unrelated languages. Now, of course, there is some balance that needs to be maintained around picking the right tool for the problem, but if multiple tools fit just fine, choose wisely. If you are splitting your efforts across multiple technologies it will be harder to become an expert in the language that matters the most for your product's success.

This also comes into play while debugging the application code. Without continual exposure, it will be much harder to diagnose and root cause bugs in your areas of responsibility. And in a very timely fashion, Dmitry made a similar comment about this on his crash dump analysis blog: "The lack of sufficient unmanaged [re: native] code programming experience might partly explain many analysis mistakes".

To be exceptional at your job, you will want to condition yourself appropriately. A sports metaphor fits nicely here; there is a significantly small percentage of professional athletes that can successfully compete in multiple disciplines. The technology profession is no different here - professionals need to train to be the best they can be, and so do you. Choosing the correct (daily) training regimen is import to you and your team's success.

Colossal Handle Leak

joshpoley — Mon, 05 May 2008 20:07:00 GMT

So I was un-installing an application (which I won't name) and just happened to notice a rather large handle count in Task Manager. Now, most applications which leak handles will consume on the order of several thousand handles, but this program (which was a custom setup application) had opened hundreds of thousands of handles.

Oops, that was a missed test case.

I took the screenshot just before heading off to a meeting, but since my curiosity was piqued, I turned on a memory tracking script and started to take a snapshot of the handle and memory usage every 5 minutes. After getting back into the office, I checked the log and saw that the setup app had maxed at 430,097 handles.

Stress and leak testing is an important task, even on "run once" (twice?) applications like a setup program. Also don't forget that there are other types of leaks than memory: any resource can be leaked.

Selecting Random Seeds

joshpoley — Tue, 04 Mar 2008 20:25:00 GMT

A few months ago I ran across some test code which was using GetTickCount() + timeGetTime() as the seed input to a random number generator. Unfortunately this code was bad for two reasons. First, the systems that the test code was running on would be frequently rebooted, so GetTickCount() would be returning similar results across a large set of test runs. Secondly, timeGetTime() is based on the exact same counter that GetTickCount() is, so adding the two together really doesn't buy you very much.

This of course raises the question of "how do you pick a good seed?" And if we take a step back, it is also good to ask "how random/secure/performant do you need your data?" For the sake of this post, we will assume that rand() is good enough, but picking a truly random seed is desired. I should also make the distinction that this is for test code, not code which needs to be cryptographically secure, so while we want to reduce collisions, it isn't catastrophic if they do happen.

The goal of the original code was to inject a certain amount of randomness into the testing such that over several months of automated testing on daily builds, there would be an increased level of coverage (think fuzz testing). Unfortunately, there just wasn't enough variance built into the seed, which will result in the tests reusing the same values (over an extended period of time).

So, in answer to the first question, it is a fairly common practice to take a handful of diverse data sources and munge them together using a hashing function (MD5, SHA-1, etc). This hash value then becomes the seed. The following list is provided to give you some examples as to possible inputs to the hash. It is obviously not required or practical to use all of them, so pick the ones that make the most sense for your application.

CryptGenRandom()	If you have access to this API, then it is a great way to get some random data on Windows and CE systems, and may be all you need for the seed (without having to bother with hashing).
XNetRandom()	This is the Xbox's version of CryptGenRandom()
Registry: HKLM \ SOFTWARE \ Microsoft \ Cryptography \ RNG	The "seed" value in this registry node will periodically be updated with a new value.
GetTickCount() or QueryPerformanceCounter()	Throwing in the current CPU counter can be a good way to perturb values across a single test run if you don’t have access to something like CryptGenRandom().
Date	Assuming the current date/time is "real" (i.e. not being reset), then this will work to change things on a daily basis.
MAC Address	Providing a unique value about the computer/system you are running on will help prevent box A from running the same cases as box B.
Build Number	The build number will help prevent today's tests from being the same as tomorrow's.
GetCurrentProcessId() or GetCurrentThreadId()	Provides a small chance to get different number.
Network Latency	If you have a network based application to begin with, then it should be fairly simple to capture some transaction timings.

Now, if you want to get really fancy, it wouldn't be all that hard to setup a web service which just dishes out hashes or values from CryptGenRandom (or XNetRandom if you happen to have a spare Xbox devkit lying around). Your tests could just grab a new seed value each time the test is run, ensuring a good starting point from which to crank through those test cases.

API Testing Best Practices

joshpoley — Fri, 29 Feb 2008 18:58:00 GMT

MSDN just published my article on API testing:

"If you ask several testers how to write good API tests you will get several different answers. Not only are peoples' styles and approaches different, but often API sets will lend themselves to various methodologies. That being said, my goal here is to describe some of the practices that have worked well for me while testing the Xbox Operating System. Hopefully the information detailed here will be beneficial and useful in your own tests."

If you haven't swung by recently, you should check out the Tester Center which hosts a whole bunch of testing related content.

Tester Center: White-Box Fuzzing

joshpoley — Wed, 13 Feb 2008 03:32:00 GMT

The Tester Center recently added some more content. One of the new articles is a pretty good piece on Fuzzing, so check it out when you get a chance:

This article presents a case study of fuzzing during development of Microsoft Internet Security and Acceleration (ISA) Server 2006, and discusses efforts, bug density, and ROI. During this release, the internal testing team found over 30 bugs that were either Important or Critical according to Microsoft Security Response Center (MSRC) rankings in over 500 KLOC parsing code

Testing Your PDB Files

joshpoley — Thu, 10 Jan 2008 20:52:00 GMT

Very little attention is often applied to PDB files, but if you do ship them to customers, then you will want to ensure they don't contain more information than is necessary. The debugging information stored here can be used to reverse engineer or attack your code, so it is worthwhile to take a precautionary step and verify (i.e. test) that there is no private data being exposed via your public symbol files.

You can use the Debug Interface functions to crack open and query your .pdb files to see exactly what information is being provided. Then it is just a matter of creating a simple tool which can be inserted into your release process to validate the files. As a straightforward sample, we will show how to use the DIA APIs to query for keywords which could be used by an attacker.

1 void KeywordList(IDiaSymbol *globalSym, LPOLESTR search)

2 {

3 IDiaEnumSymbols *searchEnum = NULL;

4 globalSym->findChildren(SymTagNull, search, nsCaseInRegularExpression, &searchEnum);

5 if(searchEnum)

6 {

7 LONG numItems = 0;

8 searchEnum->get_Count(&numItems);

10 for(LONG i=0; i<numItems; i++)

11 {

12 IDiaSymbol *sym = NULL;

13 searchEnum->Item(i, &sym);

15 if(sym)

16 {

17 BSTR symName = NULL;

18 sym->get_name(&symName);

20 printf(" %ws\n", symName);

22 SysFreeString(symName);

23 sym->Release();

24 sym = NULL;

25 }

26 }

28 searchEnum->Release();

29 searchEnum = NULL;

30 }

31 }

The "globalSym" pointer is the PDB files' main collection of symbols; to get this pointer, the typical call flow looks like:

· Create an IDiaDataSource object

· Call IDiaDataSource::loadDataFromPdb to open a specific PDB file

· Call IDiaDataSource::openSession

· Call IDiaSession::get_globalScope to get the global symbol container

On line 4 we do a case-insensitive wild-card search through the global scope (* and ? are the only supported wild-cards). If any matches are found, the result will be a collection stored in (and accessed by) the IDiaEnumSymbol interface.

The "Count" property (line 8) will tell us how many matches were found. From there we pull out each symbol using the "Item" method (line 13) and then query the symbol's name (line 18). Even though the documentation for get_name doesn’t mention anything about freeing up the returned BSTR, you will want to clean up the memory (line 22) using SysFreeString, otherwise you will end up with a memory leak.

We can then call this helper function, for example, to dump out all instances of the word "password", which is probably not something we would want to leave around for others to gather information about.

KeywordList(pGlobalSymbol, L"*password*");