JW on Test : Future of testing

more about test case reuse

James Whittaker — Thu, 22 Jan 2009 22:33:00 GMT

We mostly write test cases that are specifically tied to a single application. This shouldn’t come as any big surprise given that we’ve never expected test cases to have any value outside our immediate team. But if we want to complete the picture of reusable test cases that I painted in my last post we need to write test cases that can be applied to any number of different apps.

Instead of writing a test case for an application, we could move down a level and write them for features instead. There are any number of web applications, for example, that implement a shopping cart so test cases written for such a feature should be applicable to all such apps. The same can be said of many common features like connecting to a network, making SQL queries to a database, username and password authentication, and so forth. Feature-level test cases are far more reusable and transferable than application-specific test cases.

The more focused we make the scope of the test cases we write, the more general they become. Features are more focused than applications, functions and objects are more focused than features, controls and data types are more focused than functions and so forth. At a low enough level, we have what I like to call “atomic” test cases. A test atom is a test case that exists at the lowest possible level of abstraction. Perhaps you’d write a set of test cases that simply submits alphanumeric input into a text box control. It does one thing only and doesn’t try to be anything more. You may then replicate this test atom and modify it for different purposes. For example, if the alphanumeric string in question is intended to be a username, then a new test atom that encoded the structure of valid usernames would be refined from an existing atom. Over time thousands (and hopefully orders of magnitude more) of such test atoms would be collected.

Test atoms can be combined into test molecules. Two alphanumeric string atoms might be combined into a test molecule that tests a username and password dialog box. I can see cases where many independent test authors would build such molecules and then over time the best such molecule would win out and yet the alternatives would still be available. With the proper incentives, test case authors would build any number of molecules that could then be leased or purchased for reuse by application vendors that implement similar functionality.

At some point, enough test atoms and molecules would exist that the need to write new, custom tests would be minimal. I think that Wikipedia, a site with user supplied, policed and maintained content, would be what the industry would need to store all these tests. Perhaps such a community Testipedia can be constructed or companies can build their own internal Testipedias for sensitive applications. But a library of environment-carrying (see my last post) test atoms and molecules would have incredible value.

A valuable extension of this idea is to write atoms and molecules in such a way that they will understand whether or not they apply to an application. Imagine highlighting and then dragging a series of ten thousands tests onto an application and having the tests themselves figure out whether they apply to the application and then running themselves over and over within different environments and configurations.

Ah, but now I am just dreaming.

test case reuse (in the future)

James Whittaker — Fri, 16 Jan 2009 22:47:00 GMT

I’ve given my ‘future of testing’ talk four times (!) this week and by far the part that generates the most questions is when I prophesize about test case reuse. Given that I answered it differently all four times (sigh), I want to use this space to clarify my own thinking and to add some specifics.

Here’s the scenario: one tester writes a set of test cases and automates them so that she can run them over and over again. They are good test cases so you decide to run them as well. However, when you do run them, you find they won’t work on your machine. Your tester friend used automation APIs that you don’t have installed on your computer and scripting libraries that you don’t have either. The problem with porting test cases across machine boundaries is that they are too specific to their environment.

In the future we will solve this problem with a concept I call environment-carrying tests (nod to Brent Jensen). Test cases of the future will be written in such a way that they will encapsulate their environment needs within the test case using virtualization. Test cases will be written within virtual capsules that embed all the necessary environmental dependencies so that the test case can run on whatever machine you need it to run on.

The scope of technological advances we need for this to happen are fairly modest. However, the Achilles heel of reuse has never been technological so much as economic. The real work required to reuse software artifacts has always been on the consumer of the reused artifact and not on its producer. What we need is an incentive for testers to write reusable test cases. So, what if we create a “Testipedia” that stored test cases and paid the contributing tester, or their organization, for contributions? What is a test case worth? A dollar? Ten dollars? More? Clearly they have value and a database full of them would have enough value that a business could be created to host the database and resell or lease test cases on an as-needed basis. The more worthy a test case, the higher its value and testers would be incentivized to contribute.

Reusable test cases will have enough intrinsic value that a market for test case converters would likely emerge so that entire libraries of tests could be provided as a service or licensed as a product.

But this is only part of the solution. Having test cases that can be run in any environment is helpful, but we still need test cases that apply to the application we want to test. As it turns out, I have an opinion on this and I’ll blog about it next.

manual v. automated testing again

James Whittaker — Wed, 29 Oct 2008 23:51:00 GMT

In my Future series I was accused of supporting both sides of the manual v. automated debate and flip-flopping like an American politician who can’t decide whether to kiss the babies or their moms. Clearly this is not an either-or proposition. But I wanted to supply some clarity in how I think about this.

This is a debate about when to choose one over the other and which scenarios one can expect manual testing to outperform automated testing and vice versa. I think the simplistic view that automation is better at regression and API testing and manual is better for acceptance and GUI testing diverts us from the real issues.

I think the reality of the problem has nothing to do with APIs or GUIs, regression or functional. We have to start thinking about our code in terms of business logic code or infrastructure code. Because that is the same divide that separates manual and automated testing.

Business logic code is the code that produces the results that stakeholders/users buy the product for. It’s the code that gets the job done. Infrastructure code is the code that makes the business logic work in its intended environment. Infrastructure code makes the business logic multiuser, secure, localized and so forth. It’s the platform goo that makes the business logic into a real application.

Obviously, both types of code need to be tested. Intuitively, manual testing should be better at testing business logic because the business logic rules are easier for a human to learn than they are to teach to a piece of automation. I think intuition is bang-on correct in this situation.

Manual testers excel at becoming domain experts and they can store very complex business logic in the most powerful testing tool around, their brains. Because manual testing is slow, testers have the time to watch for and analyze subtle business logic errors. Low speed but also low drag.

Automation, on the other hand, excels at low-level details. Automation can detect crashes, hangs, incorrect return values, error codes, tripped exceptions, memory usage and so forth. High speed but also high drag. Tuning automation to test business logic is very difficult and risky. In my humble hindsight I think that Vista got bit by this exact issue, depending so much on automation whereas a few more good manual testers would have been worth their weight in gold.

So whether you have an API or a GUI, regress or test fresh, the type of testing you choose depends on what type of bug you want to find. There may be special cases, but the majority of the time manual testing beats automated testing in finding business logic bugs and automated testing beats manual testing in finding infrastructure bugs.

There I go again, fishing in both sides of the pond.

the future of software testing (part 8)

James Whittaker — Mon, 13 Oct 2008 19:06:00 GMT

Testing Beyond Release

This is the final part of my series on the future of testing. I hope you’ve enjoyed it. For this post I’ve saved what might be one of the more controversial of my predictions: namely that in the future we will ship test code with our products and be able to exercise that code remotely. I can see the hackers’ grins and hear the privacy advocates’ indignation already, but I’ll respond to those concerns in a minute.

I was in the Windows org when Vista shipped and I recall demonstrating it to my then 8 year old son at home one evening. He plays (and works if you’ll believe that) on computers a great deal and he really liked the Aero interface, the cool sidebar gadgets and the speed at which his favorite games (which at that time were line rider and zoo tycoon) ran really impressed him. I recall thinking ‘too bad he’s not an industry blogger,’ but I digress.

At the end of the demo, he hit me with the question every tester dreads: ‘Daddy, which part did you do?’

I stopped speaking, which is rare for me, and stammered something unintelligible. How do you tell an 8 year old that you worked for months (I had just started at Microsoft and only got in on Vista toward the end of its cycle) on something and didn’t actually create any of it? I tried my canned answers to this dreaded question (exclamation points required … they help me convince myself that what I am saying has some truth to it):

“I worked on making it better!”

“The fact that it works as well as it does … well that’s me!”

“If it wasn’t for us testers, this thing would be a menace to society!”

I am especially fond of that last one. However, all of them ring hollow. How is it that I can work on a product for so long and not be able to point to more than the absence of some of the bugs as my contribution?

I think that’s where this idea came from: that test code should ship with the binary and it should survive release and continue doing its job without the testers being present. This isn’t a lame attempt to give me and my compatriots something to point to for bragging rights, but to provide ongoing testing and diagnostics. Let’s face it, we’re not done testing when the product releases, so why should we stop?

We already do some of this. The Watson technology (the famous “send/don’t send” error reporting for Windows apps) that ships in-process allows us to capture faults when they occur in the field. The next logical step is to be able to do something about them.

Currently, Watson captures a fault and snaps an image of relevant debug info. Then some poor sap at the other end of the pipe gets to wade through all that data and figure out a way to fix it via Windows update. This was revolutionary in 2004, still is actually. In 2-5 years it will be old school.

What if that poor sap could run additional tests and take advantage of the testing infrastructure that existed before the software was released? What if that poor sap could deploy a fix and run a regression suite in the actual environment in which the failure occurred? What if that poor sap could deploy a production fix and tell the application to regress itself?

He’d no longer be a poor sap, that’s for sure. But since testing and test artifacts are lost when the final product is built and released this is all impossible.

In order to achieve these capabilities, it will be necessary for an application to remember its prior testing and carry along that memory wherever it goes. And that means that the ability to test itself will be a fundamental feature of software of the future. Our job will be to figure out how to take our testing magic and embed it into the application itself. Our reward will be the pleasure of seeing that sparkle in our kids’ eyes when they see that the coolest feature of all is the one we designed!

Oh, and to the hackers and privacy folks: never fear! Hugh Thompson and I warned about including test code in shipping binaries (see Attack 10 in How to Break Software Security) long ago. Since we know how to break it, we’ll be in a great position to get it right.

the future of software testing (part 7)

James Whittaker — Tue, 07 Oct 2008 19:07:00 GMT

Testers as Designers

Modern testers play largely a role of late cycle heroics that often goes unappreciated come review and bonus time. When we find the big bug it is because we were supposed to … that’s the expectation. When we miss the big bug, people ask questions. It’s often a case of ignored-if-you-do and damned-if-you-don’t.

This is going to change and it is going to change soon because it must. My friend Roger Sherman (Microsoft’s first companywide Director of Test) describes this change as the testing caterpillar becoming a butterfly. According to Roger: testing’s butterfly is design.

I couldn’t agree more. As testing and test techniques move earlier in the process testers will do work more similar to software design than software verification. We will place more emphasis on designing quality strategy for all software artifacts and not just the binary. We will spend more time recognizing the need for testing rather than actually executing test cases. We will oversee and measure automation rather than building and debugging it. We will spend more time reviewing the status of pre-existing tests than building new ones. We will become designers and our work will be performed at a higher level of abstraction and earlier in the lifecycle.

At Microsoft this role is that of Test Architect and I think most testing jobs are moving in this direction. If you’ve read the first six posts on this Future of Testing thread then you’ll appreciate the changes that are making this design centric role possible in the first place.

Now this sounds like a nice future but there is a decidedly black lining to this silver cloud. The blackness comes from the types of bugs and the types of testing we are currently good at in 2008. It is no great stretch to say that we are better at finding structural bugs (crashes, hangs and bugs having to do with the software and its plumbing rather than its functionality) than we are at finding business logic bugs. But the future I’ve painted in this series has any number of technological solutions to structural bugs. That will leave the software tester to deal with business logic bugs and that is a category of issues that I do not think our entire industry deals with in any organized or intentional fashion.

Finding business logic bugs means that we have to understand the business logic itself. Understanding business logic means far more interaction with customers and competitors; it means steeping ourselves in whatever industry our software operates; it means not only working earlier in the software lifecycle but involving ourselves with prototypes, requirements, usability and so forth like we have never done before.

There’s hard work early in the software lifecycle that testers aren’t experienced in doing. Performing well up font will mean facing these challenges and being willing to learn new ways of thinking about customers and thinking about quality.

Things are decidedly different at the front end of the assembly line and it’s a place more and more testers will find themselves as the future makes way for the present.

the future of software testing (part 6)

James Whittaker — Wed, 24 Sep 2008 19:41:00 GMT

Testing Culture

A couple of months ago I attended a lecture given by one of the Empire’s cache of Technical Fellows (maybe he was a Distinguished Engineer, I am not sure as they look so much alike). Like all our TFs the guy was wicked smart and as he presented a design for some new product he and his team were building I had an epiphany.

Evidently epiphanies cause me to display facial expressions akin to one who is passing a kidney stone. The TF noticed (so did the gal sitting next to me, but I don’t want to talk about that) and approached me after the talk. Here’s how that conversation went:

“James,” (he knew my name!) “you seem to have some issue with my design or with the product. I’d love to get your feedback.”

“No, I have no problem with either your product or with your design. My problem is with you.”

“Excuse me?”

“People like you scare me,” I told him. “You spend all your time dreaming about features and enabling scenarios and designing interfaces and protocols. You are in a position of importance and people listen to you and build the stuff you dream about. And you do all this without knowing squat about testing.”

And this was the moment he sought to do the right thing … reach out to test. He invited me review the design get involved. It’s exactly what you’d expect him to do.

But it is exactly the wrong response.

Having a tester involved in design is better than not having test represented at all. But not much better. Testers will be looking for testability issues. Developers will be looking for implementation issues. Who will be looking at both? Who will be able to decide on the right tradeoff? Neither. Getting testers involved in design is only incremental improvement; getting designers (and every other role) involved in test is the future.

Seriously, how is it that the people who build software understand so little about testing? And why have we not tried to fix this before? Are we, as testers, so vested in our current role that we are jealously guarding the keys to our intellectual kingdom? Is testing so arcane and obscure that developers can’t find the answers they seek? Have developers grown so accustomed to handing off this ‘less interesting’ aspect of the process to us that they now take it for granted?

Adding testers to the mix hasn’t worked. Getting them involved earlier hasn’t worked. We have products that have a 1:1 ratio of developers to testers and yet those products are not seen as highly reliable. We also have products that have far ‘worse’ ratio that are clearly better products. I think in the future we will come to see that the separation of roles isn’t working. The separation of roles might even guarantee that testing comes late to the dance and fails to fully leverage its intellectual potential on the product.

The current testing culture and separation of roles is broken and the way to fix it is by merging roles. Quality needs to be everyone’s job. Think of it in Tolkiensian terms: one role to rule them all!

Imagine a world where testing knowledge is contained in each and every contributor’s head. The architects know testing, the designers know testing, the developers know testing and they apply that knowledge constantly and consistently in everything they do. This doesn’t wipe out the separate testing role, there is something to be said for some amount of test independence, it enables better testing. If each decision made throughout product development asks the right testing questions, then the final system test can reach a level of thoroughness we can only dream about now. If everyone on the project understood testing, imagine what a few dedicated testers could accomplish!

Getting to this testing utopia is going to require a massive cultural change. Testing must reach into academia and the other places where programming is taught. As developers progress in their careers, this education must continue and become more advanced and powerful. We need to get to the point that all project stakeholders understand testing and can’t help but to apply its principles in everything they do. Tools will one day support this as well. One day we will be to the point that untestable software just never gets written, not because some strong tester made it happen, but because everyone on the project made it happen.

Testing is too important to be the ‘bit at the end’ of the process. It is early in the process where design decisions impact testing and it is there that the solutions lay. It's also too important to leave it in the hands of a single role dedicated to quality assurance. Instead we need a fundamental cultural shift that makes quality everyone’s job and embeds its principles in everything we do.

the future of software testing (part 5)

James Whittaker — Fri, 19 Sep 2008 22:23:00 GMT

Visualization

What does software look like? Wouldn’t it be helpful if we had a visualization of software that we could use while the software was being constructed or tested? With a single glance we could see that parts of it remain unfinished. Dependencies, interfaces and data would be easy to see and, one would hope, easier to test. At the very least we could watch the software grow and evolve as it was being built and watch it consume input and interact with its environment as it was being tested.

Other engineering disciplines have such visuals. Consider the folks who make automobiles. Everyone involved in the assembly process can see the car. They can see that it has yet to have bumpers or a steering wheel installed. They can watch it progress down the mechanized line from an empty shell to a fully functional product ready to be driven to a dealer. How much longer until it is complete? Well, its forty feet from the end of the line!

The fact that everyone involved in making the car has this shared vision of the product is extremely helpful. They speak in terms they can all understand because every part, every connection, every interface is where it is supposed to be when it is supposed to be there.

Unfortunately, that is not our world. Questions or the sort asked above how long until it is complete? or what tasks remain undone? vex us. This is a problem that 21^st century testers will solve.

Architects and developers are already solving it. Visual Studio is replete with diagrams and visualizations from sequence charts to dependency graphs. Testers are solving it too. Visualization solutions exist within the empire’s walls for seeing code changes in an Xbox title (objects whose code has churned glow green when rendered and then revert to normal after they have been tested) to identifying untested complexity within the Windows code base (heat maps of code coverage versus code complexity can be viewed in three dimensional space leading testers right to the problem areas). The visualizations are stunning, beautiful and allow testers to determine what needs testing simply by glancing at the visual.

We need more of this but we need to approach the problem carefully. We can’t simply accept the diagrams provided to us by the UML and modeling crowds. Those visuals are meant to solve other problems that may or may not overlap with the problems we face. Many of the existing visuals were created to serve architects or developers whose needs are different. We need to think this through as testers. We need visuals that map requirements to code, tests to interfaces, code churn to the GUI, and code coverage to controls. Wouldn’t it be nice to launch the app under test and be able to see controls glow with an intensity that reflects the amount of coverage or the number of tests that have touched them? Wouldn’t it be nice to be able to see a graphic animating network utilization or real time database communication? Why shouldn’t we be able to see the network traffic and the SQL queries as they happen? There is much that is going on unseen beneath the covers of our application and it's time we surfaced it and leveraged it to improve code quality.

This is an imminently solvable problem and one that many smart people are working on.

This is software testing in living color.

the future of software testing (part 4)

James Whittaker — Tue, 09 Sep 2008 18:24:00 GMT

Moving Testing Forward

There is a gap that exists in testing that is eating away at quality, productivity, and the general manageability of the entire development lifecycle. It is the gap between when a bug is created and when that same bug is detected. The larger the gap, the more time a bug stays in the system. Clearly that’s bad, but pointing out that the longer bugs stay in the system the more expensive they are to remove is what we’ve done in the past.

What we’re going to do in the future is close the gap.

But closing the gap means a fundamental change in the way we do testing. In 2008 a developer can introduce a bug, quite by accident mind you – our development environments do little to discourage that, and few concerted attempts are made to find the bug until the binary is built. We insert bugs and then simply allow them free reign until far too late in the process where we depend on late cycle bug finding heroics to bail us out.

As software testers we provide a valuable set of bug finding and analysis techniques; what we have to do in the future is apply these techniques earlier in the process, far sooner than we do now. There are two main things I foresee that will help us accomplish this. One is simply not waiting for the binary and applying our tests on early development artifacts. The second is building the binary earlier so we can test it earlier.

Let’s take these in order beginning with ‘testing on early development artifacts.’ During late-cycle heroics we apply any number of bug finding strategies on the binary through its published interfaces. We take the compiled binary or collection of assemblies, byte code etc, hook them to our test harnesses and pummel them with inputs and data until we ferret out enough bugs to have some confidence that quality is good enough (perhaps I’ll cover measurement and release criteria in a future blog entry). But why wait until the binary is ready? Why can’t we apply these test techniques on architecture artifacts? … On requirements and user stories? … On specifications and designs? Can it be possible that all the technology, techniques and testing wisdom collected over the past half century applies only to an artifact that executes? Why aren’t architectures testable in the same way? Why can’t we apply what we know to designs and storyboards? Well the answer is that there is no good reason we don’t. I actually think that many progressive groups at Microsoft do apply testing techniques early and that in the future we’ll figure out how to do this collectively. Testing will begin, not when something becomes testable as is the case now, but instead testing will begin the moment there exists something that needs testing. It’s a subtle but important distinction.

‘Building the binary earlier’ is the second part of this but doing so represents a technological hurdle that needs jumping. In 2008 we write software component by component and we can’t build the whole without each of the parts being ready. This means that testing must wait until all the components achieve some level of completion. Bugs are allowed to sit for days and weeks before testing can be brought to bear on their discovery. Can we substitute partially completed components with virtual ones? Or with stubs that mimic external behavior? Can we build general purpose chameleon components that change their behavior to match the system into which they are (temporarily) inserted? I predict we will … because we must. Virtual and chameleon components will allow testers to apply their detection craft soon after a bug is created. Bugs will have little chance to survive beyond their first breath.

Testing is too important to wait until the end of the development cycle to start it. Yes iterative development and agile create testable code earlier (albeit smaller, incomplete functionality) but we still have far too many bugs appearing after release. What we are doing now is not enough. The future must bring the power of testing to bear on early development artifacts and allow us to scaffold together a workable, testable environment long before the code is entirely build-able.

the future of software testing (part 3)

James Whittaker — Fri, 05 Sep 2008 17:35:00 GMT

So we are now at my third prediction which deals with information and how testers will use information to improve their testing in the future.

Prediction 1: Testsourcing

Prediction 2: Virtualization

Prediction 3: Information

What information do you use to help you test your software? Specs? User manuals? Prior (or competing) versions? Source code? Protocol analyzers? Process monitors? Does the information help? Is it straightforward to use?

Information is at the core of everything we do as software testers. The better our information about what the software is supposed to be doing and how it is doing it, the better our testing can actually be. I find it unacceptable that testers get so little information and none of it is specifically designed to make it easier to do our jobs. I am happy to say that this is changing … rapidly … and that in the near term we will certainly be gifted with the right information at the right time.

I take my inspiration for testing information from video games. In video games, we have the surfacing and use of information darn near perfected. The more information about the game, the players, the opposition, the environment, the better you play and the higher score you achieve. In video games this information is displayed in something called a HUD or heads up display. All a players’ abilities, weapons, health info are displayed and clickable for immediate use. Likewise, your location in the world is displayed in a small minimap and information about opponents is readily available (my son used to play Pokémon in which he had access to a Pokédex which kept information about all the various species of Pokémon he might encounter in the game … I’d like a Bug-é-dex that did the same for bugs I might encounter). The idea is simple: the more information you have and can use, the better your chances of success in the game.

I'd very much like to replicate this for software testers: give them more information to increase their success. But most of the testing world is mired in black box testing without such a rich information infrastructure. Where’s is our minimap that tells us which screen we are testing and how that screen is connected with the rest of the system? Why can’t I hover over a GUI control and see source code or even a list of properties the controls implements (and that I can test)? If I am testing an API, why can’t I see the list of parameter combinations that I and all my fellow testers have already tried? I need all of this information quickly and in a concise and readily consumable format that assists my testing rather than having to shuffle through some SharePoint site or database full of disconnected project artifacts. All this does is distract me. I want a heads up display!

My colleague at Microsoft Joe Allan Muharsky calls the collection of information that I want so badly a THUD – the Tester’s Heads Up Display – putting the information a tester needs to find bugs and verify functionality in a readily-consumable format for software testers. Think of a THUD as a skin that wraps around the application under test and surfaces information and tools that are useful in context of the application. Few THUDs are in use today or even contain the right information. In the future, no tester would think of testing without one, just like no gamer could imagine traversing an unpredictable and dangerous world without their HUD.

If this sounds a little like cheating then so be it. Gamers who add cheats to their HUD have an even bigger advantage over gamers who don’t. And as in-house testers who have access to the source, the protocols, the back-end, front-end and middleware we can indeed “cheat.” We can leverage these cheats to gain a massive bug-finding advantage over ordinary black box testers and users. This is exactly the situation we want: to be in a position to find our own bugs faster and more efficiently than anyone else. This is cheating I approve of wholeheartedly but today we’re not taking advantage of the information required for the cheats.

In the future, we will. That future will be fundamentally different than the information-starved present in which we are current working.

the future of software testing (part 2)

James Whittaker — Tue, 26 Aug 2008 18:36:00 GMT

In order for testsourcing to take hold of the future of testing, two key technological barriers must be broken: the reusability of test artifacts and the accessibility of user environments. Let me explain:

Reusability: The reusability of software development artifacts, thanks to the popularization of OO and its derivative technologies in the 1990s, is a given. Much of the software we develop today is comprised of preexisting libraries cobbled together into a cohesive whole. Unfortunately, testing is not there yet. The idea that I can write a test case and simply pass it off to another tester for reuse is rare in practice. Test cases are too dependent on my test platform: they are specific to a single application under test; they depend on some tool that other testers don’t have; they require an automation harness, library, network config (and so forth) that cannot be easily replicated by a would-be re-user.

Environment: The sheer number of customer environments needed to perform comprehensive testing is daunting. Suppose I write an application intended to be run on a wide variety of mobile phones. Where do I get all these phones in order to test my application on them? How do I configure all these phones so they are representative of my intended customers’ phones? And the same thing goes for any other type of application. If I write a web app, how do I account for all the different OS, browsers, browser settings, plug-ins, registry configurations, security settings, machine-specific settings, and potentially conflicting application types?

The answer that is emerging for both of these needs is virtualization which is steadily becoming cheaper, faster and more powerful and is being applied to application domains that run the gamut from lab management to IT infrastructure deployment.

Virtualization has great potential to empower the ‘crowd’ for crowdsourcing. Specialized test suites, test harnesses, test tools can be one-clicked into virtual machines that can be used by anyone, anywhere. Just as software developers of today can reuse the code of their colleagues and forebears, so too will the testers in the crowd be able to reuse test suites and test tools. And just as that reuse has increased the range of applications that a given developer can reliably build, it will increase the types of applications that a tester can test. Virtualization enables the immediate reuse of complicated and sophisticated test infrastructures.

Conveniently, virtualization does the same favor for testers with respect to user environments. A user can simply one-click their entire computer into a virtual machine and make it available to testers via the cloud. If we can store all the videos in the world for instant viewing by anyone, anywhere then why can’t we do the same with virtual user environments? Virtualization technology is already there (in the case of PCs) or nearly there (in the case of mobile or other specialized environments). We simply need to apply it to the testing problem.

The end result will be the general availability of a wide variety of reusable, automated test harnesses and user environments that can be employed by any tester anywhere. This serves to empower the crowd for crowdsourcing, putting them on more than even footing with specialized outsourcers from a technology standpoint and since they far outnumber the outsourcers (at least in theory if not yet in practice) the advantage is clearly in favor of this new paradigm.

Market forces will also favor a crowdsourcing model powered by virtualization. User environments will have a cash value as crowd testers will covet them to gain a competitive advantage. Users will be incentivized to click that button to virtualize and share their environment (yes there are privacy implications to this model, but they are solvable). And since problematic environments will be even more valuable than those that work well, there will be an upside for users who experience intermittent driver and application errors: the test VMs they create will be more valuable … there’s gold in those lemons! Likewise, testers will be incentivized to share out testing assets and make them as reusable as possible. Market forces favor a future with reusable test artifacts and virtualization makes it possible.

So what does this virtualization-powered future mean to the individual tester? Well, fast forward 2-5 (or longer if you are skeptical) years in which time millions (?) of user environments will have been captured, cloned, stored and made available. I can envision open libraries of such environments that testers can browse for free or proprietary libraries available by subscription only. Test cases and test suites will enjoy the same treatment and will be licensed for fees commensurate with their value and applicability.

Perhaps, there will come a time when there are very few human testers at all, only a few niche and specialized products (or products of extreme complexity like operating systems) will actually require them. For the large majority of development, a single test designer can be hired to pick and choose from the massive number of available test virtual environments and execute them in parallel: millions of person-years of testing wrapped up in a matter of hours because all the automation and end-user configurations are available and ready to use. This is the world of testsourcing.

It’s the end of testing as we currently know it, but it is the beginning of a whole new set of interesting challenges and problems for the test community. Everything we currently know about testing applies to this new world, it's just executed in a completely new fashion.

And it’s a viable future that doesn’t require more than virtualization technology that either already exists or is on the near term horizon. It also implies a higher-order effort by testers as we move into a design role (in the case of actually performing testing) or a development role (in the case of building and maintaining reusable test artifacts). No more late cycle heroics, testers are first class citizens in this virtualized future.

the future of software testing (part 1)

James Whittaker — Thu, 21 Aug 2008 00:57:00 GMT

This is the first post of a multipart series on my predictions about the future of software testing and how testers of the future will do their job.

Outsourcing. It’s a familiar term and the way a lot of testing gets done here in 2008. However, it wasn’t always so and it’s not liable to be that way in the future either. In this post I will talk about how I think testing will get done in the future and how outsourcing might fundamentally change as a business model for software testing.

In the beginning, very little testing was outsourced. Testing was performed by insourcers, people employed within the same organization that wrote the software. Developers and testers (often the same people performing both tasks) worked side by side to get the software written, tested and out the door.

The vendors’ role in the insourcing days was to provide tools that supported this self service testing. But the vendors’ role soon changed as demand for more than just tools surfaced. Instead of just providing tools to insourcers, vendors emerged that provided testing itself. We call this outsourcing and it is still the basic model for the way many development shops approach testing: hire it out.

So the first two generations of testing look like this:

Generation Role of Vendors

(1^st) Insourcing Provide tools

(2^nd) Outsourcing Provide testing (which subsumes the tools)

The next logical step in the evolution of testing is for vendors to provide testers and this is exactly the era we’ve entered with crowdsourcing. Yesterday’s announcement by Utest marks the beginning of this era and it is going to be very interesting to see it unfold. Will crowdsourcers outperform outsourcers and win this market for the future? Clearly market economics and the crowds’ ability to execute will determine that but my personal view is that the odds are stacked in favor of the crowd. This is not really an either-or situation but the evolution of the field. The older model will, over time, make way for the newer model. This will be a case Darwinian natural selection played out in the matter of only a few short years. The fittest will survive with the timeframe determined by economics and quality of execution. Crowdsourcing has much going for it including the sheer number of tests and test environments that can be brought to bear by the size and expertise of the crowd.

That gives us the third generation:

(3^rd) Crowdsourcing Provide testers (which subsumes the testing and tools)

And what about the future? Is there an aggressive gene buried deep in the DNA of our discipline that will evolve crowdsourcing into something even better? I think so, though it may be years and a few technological leaps away. I’ll coin a new term for now just to put a name on this concept: testsourcing.

(4^th) Testsourcing Provide test artifacts (which subsumes the testers, testing and tools)

Testsourcing cannot be explained however without one key technological leap that has yet to occur. This technological leap is virtualization will be described in part two of this series.