Update: this blog is no longer active. For new posts and RSS subscriptions, please go to http://saintgimp.org.

Before I get into technical subjects, I should briefly outline the context in which I do my work.

Microsoft Game Studios publishes games written by internal MGS studios and by third-party game developers, for both the PC and Xbox platforms.  MGS has a shared testing organization called the GTO that’s responsible for testing most of the games that MGS publishes.

The GTO has a testing philosophy that’s somewhat unique in the game industry; rather than just throw hordes of bodies at an almost-finished game to find all the bugs, we apply the engineering-oriented testing approach that’s found in the rest of Microsoft.  That means we have a strong emphasis on driving quality early in the development cycle, on preventing bugs from being created rather than just finding them after the fact, and on using tools and technologies to reduce the amount of tedious manual testing that we have to do.  We want to automate as much of the testing process as possible, and make the rest of it as efficient as possible.

Of course, trying to do these things as a game publisher presents some big obstacles.  As I noted in my intro post, I came to MGS nine years ago thinking it would be a pretty easy task and it turned out to be no such thing.  It’s an interesting, multi-faceted topic that I may go into detail on later, but for now I’ll just outline the highlights:

• Most games (of the type MGS publishes, anyway) are real-time, non-deterministic systems.  They’re not simple state machines where input A always leads to result B.
• Game engines are almost never built to be testable.  In fact, they’re usually built to be as opaque as possible in order to reduce the possibility of cheating and exploits.
• In games, we don’t have the luxury of operating system facilities to help us decipher and manipulate program state.  For example, we can’t just ask the OS which window is in the foreground and what text is in the window.
• Individual games don’t have long lifespans that allow you to easily earn back the investment you made in testability.  Once the game ships and you maybe do an update or two, then you flush the whole thing and start over.  Compare this to Windows or Office where they’re still running automated test that were written many years.  MMOs are one exception to this rule.
• The game industry as a whole has a pizza-and-caffeine, cowboy-coder legacy that doesn’t encourage a careful, disciplined approach to engineering.  This is changing, thankfully, and our development partners are the best in the business, but that legacy is often still part of the culture.
• The developer-publisher relationship can make it difficult to influence real engineering changes because there are two different organizations with well-established cultures.  We and our partners have the same overall goals but sometimes we have different optimal paths for getting there.

In the face of all of those challenges, how do we focus on engineering-oriented testing?  First, we hire really smart engineers who have the skills to solve those challenges.  A position in the GTO isn’t merely entry-level or dead-end; it’s a serious and rewarding career and lots of us have been around for a long time because we love what we do and we’re good at it.

Second, we have a team of programmers, of which I am a member, who are dedicated full-time to making our games testable and to building testing tools.  This team takes on a wide variety of jobs depending on the needs of each game title but the major themes include:

• Building test hooks into game engines that don’t have them
• Building technologies to make automated testing easier and cheaper
• Building automated testing tools.
• Building tools to optimize the collection and reporting of bug information
• Building tools to analyze the contents of games so our test teams can better plan their
• test coverage.

From a software development standpoint, whenever we work with our development partner’s source code (like when adding test hooks to a game engine) we strictly hold to their coding standards and ways of doing things.  That work is usually in C++.  However, when we work on standalone tools that don’t get compiled into the game engine, we work in C# where possible and we’re adopting more and more XP and Agile practices to improve the way we design and write our code so we can deliver more business value.

I should note that while the GTO has had several full-time tools programmers for a number of years, we came together as a united tools team within the GTO less than a year ago and we’re still working on meshing our individual ways of doing things into a cohesive whole.  Obviously, I’ll write my blog posts from my own perspective which doesn’t necessarily represent the position of anyone else in the GTO tools team, the GTO, or MGS.