I am filled with solutions : software

Go slow now to go fast later.

SaintD — Wed, 27 Feb 2008 20:45:00 GMT

Again I shall recomend you read Jason Gorman's blog. Speed Comes Later. Start Slow. Focus On Getting It Right. I would go one further than him and say that the same thing applies to performance goals in the software. Make it go faster after you make it go right and good.

I have come to sunny Denver Colorado this week to catch up with my MDM team mates who work in the Denver Tech center. My flight was quite delayed. The first airplane displeased the captian and he sent it back for another. Who am I to argue? The second plane has some issues. We got to the runway four times and aborted three times due to a gauge that went dark whenever the engines were throttled up. I don't know if anyone actually fixed it the third time, we flew without it or it kept working. The other passengers were a mixture of annoyance at bewing 3 hours late and alarmed that two of the planes had problems.

I can't help wondering how good the "Sustained Engineering" effort for the airlines are. I know in software it often gets the short end of the stick.

Screencast: 8 minutes with Visual Studio 2008. Easy unit tests and code coverage.

SaintD — Sat, 16 Feb 2008 09:05:00 GMT

This week I spent my blogging allowance on creating a screencast. This video shows you how easy it is to create unit tests and measure code coverage in VSTS 2008.

Video: Creating unit tests and measuring code coverage in VSTS2008

Learn to “get traction” as a tester on your team.

SaintD — Sat, 15 Dec 2007 03:34:00 GMT

Learn to “get traction” in your team. A guide for professional testers.

Software development is a fast paced world. We are constantly trying to work to the rhythms of “web time.” Professional software testers have valuable input for the team. The problem is that it can be hard to break free of the daily concerns to make that big impact.

Ideally testers are empowered to make positive changes in the organization and the product. The term empowerment is lampooned in Dilbert for good reasons. It’s a great management buzzword but companies rarely do a good job of putting it into practice. The reason is simple. No one can empower you like you.

You can empower yourself when you know how to get traction. Empowerment can trickle weakly from the top down, but it doesn’t have to. When you know the tricks you can empower yourself.

Make sure your peers are your allies.

A tester’s job is to look at things and perceive where they might be deficient. A good tester may have a habit of breaking everything they get their hands on. Be careful that you don’t break the people you are working the closest with. Turn that critical eye inward and ask yourself how you can strengthen your work relationships with the people you work with. The best way to do that is by having a strong vision of what your ideal world looks like. Hold on to that and “show” it to your peers when the going is rough.

Leaf nodes do all the work.

In the software world all of the power is really in the hands of the worker bees. Think about it. The developer you work with writes 100% of the code for his or her features. The developer’s boss doesn’t write a line of code. The product designer’s boss doesn’t write the specification or make the day to day design decisions that come up. Your boss doesn’t write the test specs, the test automation or run more than a handful of test cases at best. The further you go up the chain, the less direct impact a manager has on the actual code. The Product Unit Manager has almost zero influence on the actual technical work that happens in the product. Wow. You and your “feature crew” have all the power. Even in a non-feature crew model. All you have to do to change the product is convince one developer to do something different.

Energize your co-workers.

If you want to make a change in your area, product or organization you must start by getting the people doing the real work excited about it. If you convince one developer to do less and do it better there is nothing his boss can really do about. If you convince the product designer to make a feature a better fit for customer goals, the VP of your product can’t stop them. Start your great ideas right in your own feature. Get your peers excited about the cool ideas you have. Sell them to the people you work with every day. It doesn’t get easier than that.

Listen to Yoda.

“Always with you it cannot be done.” - Yoda chastising Luke on Dagoba. Don’t be a whiner like Luke. Stop thinking about what you can’t accomplish. It’s probably true you can’t get your entire company to change to a better process tomorrow. That’s not important. Find something you can make a little bit better today and do that. Go talk to a developer and ask them to help you think of ways to make a part of the program easier to test via automation. Have a passionate discussion about the benefits doing less and doing it better. Passion and enthusiasm are contagious. Concentrate on the things you are enthused about. Save the gripe sessions for later or never. Always frame your criticisms as “we can do better and here is my idea how.” It’s easy to complain about things that are wrong, but not usually effective. It’s surprising how often people will adopt your plan if you have one.

Re-factor your process the same way you re-factor code.

If you have some code and you want to factor out all the common stuff you don’t do it all in the same instant. It’s a process of iteration. Move a line here, a function there, a helper object over here. Then test each time you make a change. Think of your process the same way. Don’t try to overhaul it overnight. Pick one thing and move it to a better home. See how that goes. Then move on.

Iterate more, plan less.

Test specifications are a perfect example of where over planning happens a lot. A tester reads the specs and other materials then spends days locked in their office creating a test spec. Then in the review certain sections get ripped to shreds. The tester fixes those and soon the test spec is closed. Later in the product cycle you realize you left out something basic and important. A better pattern is to start with the outline and short explanations. Run this by a peer, your boss, the developer and/or the pm informally. See what feedback stems out of this embryonic document. People will be less constrained by a deep level of detail and give you good big picture feedback. Go back and add some details. Get more feedback. Rinse and repeat. Both techniques will take you a week or two of calendar time. The iterative model will create better documents in same number of person hours and less “work” for you. Look for other places where you can replace planning with iteration. Iteration is very powerful.

Close feedback loops.

Look for places where there is feedback in the software process. A prime one is logging bugs. How long is it on average between the time a developer writes code and you log the majority of the bugs? If you measure this time in weeks you have a feedback loop that needs closing badly. If the developer isn’t getting timely feedback, how can they improve their skills? What if the build process pointed out common coding mistakes? Some development environments have build in code analysis tools. Using them closes the code review feedback loop from days to minutes. Those kinds of tools have improve my coding substantially more and faster than traditional code reviews. This is a good example of doing less and doing it better.

Look for these loops and be ruthless about making them shorter and more meaningful.

Don’t worry about The Process.

Every software development effort has an official document set with The Process. I don’t know a single place that is following that exact process. Don’t worry about that. Concentrate on the actual traditions and workflow that is happening. You can get work done in any framework. Thinking too much about the big picture process will blind you to the major improvements you can make locally today. Just find one thing we can do better now. Can you find a way to do less and do it better today? Go do that. Then tell people how you did it and get them excited about it. The Process will mutate on its own for good or ill. Be an agent for positive change.

Management wants to change for the better.

There isn’t a manager at any level in any company that wants to do worse today than they did yesterday. If you can show a clear path to improvement, management will embrace it. The trick can be showing the path in a way that management is compatible with. If you are reading this article you probably believe that one or more managers you deal with is obstructing progress. I am sure they don’t see it that way. You can still open their eyes if they won’t listen. Here are some tricks you can use to show your managers how to do things better.

Repeat your good ideas, over and over.

The first time you say something like “I read on the web we should do less and do it better,” you may get a lot of resistance. That’s fine. That’s human nature. We aren’t entirely rational creatures. Keep at it. If you are genuinely passionate about doing less and doing it better it should be easy to keep discussing it. Don’t be shy about speaking up. Keep talking about it with anyone who will converse with you on the subject. Bring it up in the hallways and at lunch. It may take months, but your message will sink in. Pretty soon you will overhear people say things like “We need to do less and do it better. How can we accomplish that today?”

Take baby steps.

There are many clichés that come to mind about how to do something big. The journey of a thousand miles starts with a single step, and so on. It’s a true cliché. Try not to get frustrated because you didn’t scale Everest today. If you make one thing a tiny bit better every day, at the end of the year your impact will be huge. Break your tasks down, then break them down again. Just make sure you are going the right direction a little each day. Other people may seem to want to sabotage your efforts. Don’t worry about that either. Their distractions and setbacks are likely to be random. If you are committed to making just one thing better and work on it all the time you will win.

Build momentum from the grass roots.

Remember that all the “real” work happens in the roots of the organization. If every worker bee in your organization is talking about doing less and doing it better, management will naturally fall in line. They won’t have a choice because the real work force will already be doing it. When you have an “aha moment” and realize how things could be better shop it around. Encourage the people you work with to think about it. Ask them if they think it’s a good idea. Ask them how they would go about doing it. Ask them what’s wrong with it. Infect them with your enthusiasm and be willing to be infected with theirs. Try to reach out to people who are your peers but you don’t talk to very often. Have lunch with someone from a different part of your company and discuss your good ideas with them. Get your meme out there. You will know you have won when it mutates, takes on a life of its own and runs away. Time to get started on your next great plan.

When you see how much influence you have you will be shocked.

Testers feel like everything is out of their power. It can really feel like we are at the bottom of the hill and all the crap is rolling down to us. Don’t just whine about it. You can empower yourself when you know how to get traction. If you can do that you move from just running tests to improving the company. That’s a really big change in role, for the better. Don’t forget to put that on your evaluation this year. You can make any project you work on better.

Getting traction can be a subtle art. Think about ways to get traction and go make your product, your company and the world better.

Escape the BVT automation trap.

SaintD — Sat, 17 Nov 2007 00:13:00 GMT

Escape the BVT automation trap.

Create outstanding BVTs on time and on budget.

The build verification tests (BVTs) for your product are important to get right. Automated BVTs are a very powerful tool in your test arsenal. They allow you to know if a build is worth testing in a short amount of time with little or no user intervention. If you tie them to your official build system you can usually have results waiting for you when you arrive to work in the morning.

You need a clear plan for moving from manual BVTs to automated ones. The methods that make good manual BVTs aren’t a good fit for automated BVTs. In fact they are a trap! You need another set of principles to work from.

Don’t fall into the clunky, slow BVT automation trap. Bad automation doesn’t justify the cost to create it. Testers will idled for hours every day while someone tracks down the root causes of failures. If you are in the trap you can lose days of productivity when it matters the most, crunch time. The bait in the trap is complex, customer focused scenarios. It’s yummy and can be irresistible to testers and managers. A good idea at the wrong time turns bad.

Imagine BVT’s that are fast, reliable, trustworthy and useful. This is the ideal that you should be shooting for. This gets you out of the trap where BVT’s try to be all things to all people.

Create your BVTs with the simplicity (also known as K.I.S.S.) principle. Simpler is better when it comes to BVTs. Resist the urge to use complex tools to solve simple problems. Instead use simplicity itself to create outstanding automated BVTs. Simplicity means optimizing for automation goals and leaving other testing goals for other tests. BVT automation goals include things like being fast, maintainable and diagnostic. BVT automation non-goals include core customer scenarios and ensuring features are correct.

BVT automation is a tool.

Just as a hammer is a great tool for nail, BVT automation is a great tool for a specific purpose. It’s not a catch all place to dump all the automation you have ever created for your product. Be vigilant and keep things out that don’t belong. Good BVTs systems have certain qualities in common. Keep your tests simple and keep your BVTs focused on their mission.

A good chance that testing can proceed is enough what the tool is for.

The purpose of good BVTs is a quick way to tell if a particular build is worth testing further. The “worth testing” threshold will be different for every product and change as the cycle progresses. If the BVT system can wave you off a bad build even half the time (50%) it’s probably paid for itself. You reach the point of diminishing returns quickly. In order to have a 80% chance of testability you need to do a ton more work just to get 30% better. Don’t waste time. Predicting possible build breaks is a trap. Observe your product over time and add tests where they will do the most good. Adding a BVT for each detected build break may be like closing the barn door after the cows are gone. Ask yourself if this test will catch future breaks. If the answer isn’t “probably” then don’t add it to the BVT suite.

Fast results are money in the bank.

When you build a new version of your product you want the green light as soon as possible. BVT systems that take hours to run are wasting valuable cycles. When a developer has a potential check in they should be able to run BVTs in a matter of minutes. If it running the tests requires a full build and human intervention in a customized environment you won’t be getting very good leverage on your BVTs. I have seen BVT systems that were so top heavy that a check in late in the game could cost an entire day just to build, get on the test bunch and run a “sanity” pass. Don’t fall into that trap.

An automated test in the hand is worth two in the bush.

You want to deliver BVT automation as early in the process as possible. You can solve a lot of problems with enough time. But when you are delivering BVTs time is the first problem you need to solve for. If you have a great idea for an object model for the product but it will take weeks to code, don’t wait on it to create your BVTs. Go for a few cheap (and yes, possibly dirty) tests you can deliver today.

BVT automation needs to trusted

If your team has no faith in the results they get from the BVTs you have wasted your time writing them. When a BVT fails everyone should believe that the results are valid. You won’t be right 100% of the time of course, but you can come close. If your BVTs are failing on a regular basis because of setup issues or other reasons outside actual product flaws they lose a lot of their punch. If you find yourself in this situation you may need to rethink your BVT design. Get away from complex brittle scenarios and move to something simpler. If BVT failures are catching more problems with the environment, the network of other “noise” you either need new BVTs or a more solid environment or both. Make this a priority or manual BVTs will be the only thing you can trust.

Create fast diagnostic BVTs

Fast BVTs make your automation more trustworthy. You can use the short feedback loop to iterate them until they are rock solid. Make sure BVTs can be run quickly. Then run lots of iterations and verify they are good. Create BVTs with diagnostic results. A BVT that points clearly to a method in code is superior to one that can’t isolate failures. When you detect a break you want to be able to zero in on the offending part of the build and fix it very quickly. Couple highly diagnostic with a quick execution time and you cut the time you need to fix a break to practically nothing.

Create adaptable BVTs

Adaptable BVTs are more trustworthy. They will spend less time being irrelevant. Your product will change out from under the tests as the cycle progresses. Adapt the BVT test suite to these changes quickly. It’s ok and even desirable for an individual test to be brittle. By making the BVT tests as simple as possible they are less likely to go wrong. If the product changes you can easily decide what tests are obsolete and need to go and what new tests need to be created. Complex tests that rely on intricate libraries to interact will idle testers while fixes come in. Fixing a giant test library that was made with assumptions about the product that are no longer true can be time very consuming. If the library owner is on vacation or has left the project you are in real trouble. Keep individualt tests simple to keep the overall suite adaptable.

Create maintainable BVTs

Simple code leads to maintainable code. Make your tests as maintainable as possible. The bugs are clear. The repro scenarios are clear. You can easily prove the automation is working in the debugger. Developers can easily suggest fixes for faulty tests that are easy to understand. When you are on vacation and one of your tests fail you won’t be getting a panicky call or email. If the time comes to hand the product off to another team or the customer to maintain, they can easily run and update your tests.

BVT automation needs to be manageable

An automated system of tests has to have a lot of moving parts. This can make the systems difficult to manage. Help to keep the complexity down by leveraging things you will be doing anyway. Running BVTs isn’t free. There is a cost in hardware, support and time. Make wise use of these resources.

Leverage unit tests

Unit tests are really good BVT tests. Typically all you need to do is get them to run in your BVT system and output reports. If you work with developers up front and support them they will often be happy to write the tests to in whatever harness the test team is using. If this isn’t possible for some reason, it’s still easy to adapt or port the tests. Another key way to leverage the unit tests is to simply create more of them. Once a tester sees one unit test for the code it’s easy to use it as a template to add other tests. Developers often only create a functional test and maybe a negative test. You can improve these by adding boundary tests and tests that randomize valid inputs. This is a simple and fast way to crank out a lot of simple, fast and dependable tests that will last the life of the product.

Leverage simple component tests

Subsets of component tests are really good unit tests. My favorite is a test that checks to see if the component installed correctly. That is a dead simple test to make and a fantastic BVT. Don’t worry about all the core scenarios for the component. Save that for other automation. Just look for quick high level tests that are simple. Checking for performance counters, log entries and configuration files are all good examples. Checking the correctness of those same things is more complexity than the BVT system should have.

Leverage core API functions

If your product has simple to use APIs they make good BVTs. Say the product has a web services XML interface listening for HTTPS connections. A good test would be to check if it’s actually listening on the port. Listening on the port is really core to the way the API works. Enumerating users in the user table is a compelling scenario but save it for your other automation suites.

Will you find the right balance for your BVTs?

Creating a set of BVTs that prevents wasted time is harder than it seems. Prevent customer scenarios from luring you into a bloated BVT system. Customer scenarios are a Good Thing™. You must do them. Just don’t do them at the wrong time when they will cost you more than they should. Create your BVTs with the simplicity principle. Simple BVT tests are the path to having a great BVT system. A great BVT system focuses your team on shipping a superior product.

Keep asking if your BVTs are simple enough. Then make your product awesome with the help of simple, streamlined BVTs.

Get good returns on your test automation dollar.

SaintD — Sat, 03 Nov 2007 02:42:00 GMT

Get good returns on your test automation dollar.

Building test automation is like building a house.

Test automation seems like such a good idea. Convert something time consuming and boring into an automated process. It really is a good idea but you have to watch out for some pitfalls. Just as you wouldn’t start building a house with the paint, you shouldn’t start your test automation effort without thinking it through. Don’t fall in love with scenario tests just because they mimic the customer experience so closely.

Your customers really want you to build on a solid foundation. As professional software testers our role is to channel the customer. We are the ones they rely on to ensure their point of view is heard in the software creation process. We have to think hard about the best ways we can fulfill this vital mission. We have to make sure the software they will be using is built well and that means putting our automation efforts on a solid foundation.

There never seems to be enough time for all the automation you want. If we spent all the time it would take to automate every possible test, the product would never ship. If we don’t do any automation or have a very small set of automated tests we may not ship the product we were hoping for.

A strong foundation of good test automation has a big impact. Setting a house on a good foundation can mean the house stands the test of time. You are passionate about software. Plan your automation strategically to use your passion for the customer experience the best way possible.

Automate the lowest level of tests first; just like you would start with a foundation for a house. Product designs flow down from user scenarios. It’s tempting to approach test automation the same way. Experience shows this isn’t the best approach. Start from the bottom up if you want a fast, maintainable and balanced test automation set.

Start with unit tests and work your way up.

The levels of testing from low to high are unit, component, API and then scenario tests. Scenario tests are also known as end to end tests (E2E) and often use the exact same UI the user will. When you are writing your test automation there are a lot of reasons to make sure you have a good set of unit tests first.

Low level automation has a high return on investment.

Low level tests don't require broad product knowledge. Therefore you can get started automating much faster. Your ramp up time is reduced as compared to scenario testing. You may not even have to understand what the product does. You just need to understand what functions, components and APIs are supposed to do at a granular level.

You can write a lot of it early. Because of the relative simplicity of low level tests you can create a lot of tests early in the product cycle. You can have a few component tests done the same day a developer drops you a working build. Because you can deliver automation so quickly you can close the feedback loop for developers in a timely fashion. They can get a good gauge of how their work is progressing and what the problem areas are. This is vital to a healthy product cycle.

You can run it lots. Because it’s so well suited to running in a fully automated environment you can run the low level automation with a very low “tax”. Developers can run it whenever they make code changes. You can run it every time you build. You can run it as part of every test pass. This is a big advantage when trying to gauge the quality of your product.

It can enable you to create scenario tests robustly and more cheaply. The final advantage of a good set of low level tests is that they will inform your creation of the scenario tests. You can sand the rough edges off the product before you even try to test scenarios. You can make sure that people on the test team have an in depth understanding of the way the product is put together and works internally. You will discover if your product has API level problems or hazy specifications that need to be cleared up much earlier which will in turn make your scenario tests more focused.

API testing can often be completed automatically. If you have a well defined API that’s spelled out in technical language you can write lots of tests very quickly. You can use several approaches to generate your tests and test data. You can often create thousands or tens of thousands of API test cases in a very short period of time using tools.

Unit, component and scenario tests are a good match for early automation goals so do them early.

Scenario test have one set of advantages. They are highly customer focused. There are a nearly infinite number of interesting tests. The bugs they find are important to customers. Low level testing has another set of advantages. These advantages make them better suited to automation runs than scenario tests. There are several reasons for this.

Low level tests are more deterministic than scenario tests. Have you ever run a scenario test only to find out that it doesn’t run the same every time? You can end up endlessly tweaking and fiddling with them. You do hacky things like adding hard sleeps and resets into the code. Low level tests almost never have these kinds of problems. They run the same way every time. You can run those tests thousands of times in a row and get the same result. When you are looking at reports of the health of your product, you want to have the most reliable tests be the most numerous in the automation runs.

Low level tests are simple enough to be trusted. Nothing is worse than having the automation fail and having to carry out lengthy manual tests to figure out where the problem is. Developers will often point the finger at your test automation. Lower level tests are better in this regard. It’s easy to prove that the test is correct. It’s also easy to realize when the test is wrong and you can spend your time making the test better rather than doing lots of manual work to diagnose the problem.

Low level tests can be fast enough to run many pre-check in. Ideally your developers can run your automation before they check in changes to component code. If all you have are complex scenario tests that require a lot of setup you won’t get many people to run them. If your tests are more or less self contained and can run in a couple of minutes you are in much better shape. Developers love running tests that find bugs quickly and save them making bad check-ins. They can focus on the subset of tests that really matter to them. Since the developer will have a personal relationship with their testers they will be more willing to trust the automation and they know who to go to for help.

Low level tests are highly maintainable. When you have to hand off your test suite to someone else, low level tests are superior to scenario tests. They are usually made up of fairly elemental code. They can be put in the debugger and understood easily. Scenario tests are seldom this maintainable. This is also a big advantage when product code changes. Picking which tests to keep, which tests to fix and what new tests need to be created is much easier when considering the low level tests. When you go back to them after weeks or months you won’t be struggling to understand what you were thinking when you wrote them.

Low level tests are highly diagnostic. Sooner or later your tests will fail. It’s important to understand why the failure occurred. Product bugs need to be understood by developers quickly and completely. Test automation bugs needs to be fixed correctly by testers. When the time comes to diagnose failures lower level tests are much easier to deal with.

Scenario tests should be automated last (but not least).

After all these points you may think that I don’t like scenario tests. This isn’t true at all. But the reasons they are important shouldn’t be used as an excuse to skip the lower level automation. If you have to make cuts in automation, cut scenario testing and do it by hand. The scenario tests deserve to be run but they aren’t the highest automation priority.

Scenario tests don’t require expertise to run manually. This is not true of the unit, component and API tests. Anyone with a scenario script can run theses tests. If you have to cut automation, it’s a lot less risky to cut tests you can run by hand. These tests can be run by testers who don’t have strong automation skills. The testers won’t need in depth black box knowledge of the product. They can often be outsourced with reliable results.

Scenario tests are not diagnostic. Even if you manage to automate a large number of scenario tests they aren’t great at isolating bugs. It’s true your product should contain logging and diagnostics to make this process better. In practice these self analysis systems often have holes and blind spots that are hard to predict. Analyzing scenario results can be a very complex process. In a product with no or poor self analysis tools you will find yourself very frustrated when the scenario tests fail. You want your highly diagnostic tests to be created as early as possible. Save the less diagnostic tests for later.

Scenario tests are not easy to maintain. This is because of the complexity you must build into them. Even the simplest scenario touches on wide swaths of the product and may rely on external dependencies that are hard to deal with. Because of this complexity these tests are expensive to maintain. You can mitigate a lot of this problem with a solid library set. However the library will require expertise to create and a budget to maintain as well. You can make the problem better and move it around a little but in the end you can’t escape the complex nature of these tests. Get the maintainable tests running first, they won’t consume your life later while you are working on the harder stuff.

Scenario tests are best built on a solid framework of lower level tests. The best scenario tests are ones that build on a library of knowledge and library of well tested working code. If you can encapsulate your features into lower level tests you can set yourself up for success with the scenario tests. Your understanding of the product will be more complete and this will lead to better tests. You can use more reliable automation to setup the parts of the test not core to the particular test case. By keeping the UI driving code to a minimum you make your test suite higher quality and easier to maintain. You need to have this foundation built up before you can leverage it.

Think ahead and prioritize high return investments to keep your automation house in order.

By concentrating on the "top level" customer scenarios you often end up with a small set of tests that are expensive to run and keep. This means you should prioritize automating the lowest level of tests first.

Learn to do this well so you can become an effective advocate for quality and ultimately for the customer.

Edit: Removed "Do Unit Testing First" sidebar. I think it's the culprit for messed up formatting some people are seeing.

Why the world needs you to be a Test Architect

SaintD — Sat, 27 Oct 2007 03:01:00 GMT

Why the world needs you to be a Test Architect

How testers can be the heroes of the product lifecycle.

A band of testers

The modern software process can be like a minefield. Products slip. Products ship with poor quality. Shipped products can be expensive and cumbersome to maintain. Running services can require armies to keep them in good order.

The professional software tester runs head long into these and other mines all the time. Product specs aren’t complete before code is done. Code complete dates slip but ship dates are cast in stone and test has to “suck it up.” You have been there. You know how the bombs that go off during production unerringly land on the test team.

What testers need is a way to clear the mines from the battlefield. When products are on track it’s a glorious thing. Testers report quality metrics with authority and certainty. Big cuts can be made with confidence. Ship dates aren’t just ghost ships hanging out on the horizon.

Focusing on testability is the way to remove the majority of the mines. A product designed from the ground up to be testable is superior in every part of the lifecycle to one that is grown organically.

How focusing on testability helps win the war

Enforced Quality gates in the design phase win the first battle by saving time and money across the release

The way win this battle is to institute and stick by quality gates for every milestone. Especially the M0 (design) milestone. It’s tempting for teams to skimp on the gates early and make them solid later. Don’t make this strategic mistake. Ensure your designs complete with well defined interfaces that are treated as contracts.

You first gate should be to insist on good loosely coupled designs. Loosely coupled software is easy to test. If you speak about loosely coupled designs to developers and designers they all nod in agreement. The battle isn’t over though. All too often the designs are incomplete, incoherent and lead to hard to test code. Leaving interfaces to be cooked up organically on the fly by developers is a sure way to cut test out of the loop and build bugs into the product. Halt the process until the designs and interfaces are fully cooked to win this engagement.

The second gate is to insist that the test automation plan should flow naturally from the design. You should be able to read the product designs for a component and be able to imagine how you would easily and cheaply create thousands of automated tests. One example of a good design is a locked down XML API. You can use XSLT to automatically generate tests very efficiently. An example of a bad design is a complex product that can only be automated from the UI level. You have to create a lot of complex and brittle automation to test even a fraction of the product.

The cheapest bugs to fix are the ones you find early. Many sources including the excellent tactical manual Code Complete by Steve McConnell point out just this. Not only that but design bugs realized late in the process are the most expensive to fix. The first bugs you should find and fix are designs that are hard to test.

Cost Effective Automation wins the battle against exploding test matrixes

A well designed product doesn’t need extensive UI automation. Time after time I find testers intent on automating the end to end customer scenarios. They are convinced that running the computer programs in exactly the way customers do will find most of the bugs. Since the bugs we find after we ship are usually due to the customer doing something wacky this is natural. If your products are designed in a loosely coupled way with automated testing in mind, you will be able to automate thousands (maybe even millions) of test cases and work out all the corner cases before the UI is even applied to the product.

You want your BVT automation to be simple, fast and reliable. An extensive suite of tests is no good if nobody trusts them. BVT automation is held to a higher standard still. A highly testable product enables BVT automation that is dead simple. Even a developer can understand (and therefore trust) it. A product that’s difficult to test will be impossible to create lightweight BVTs for. You will blow your test automation budget on a few simple BVT tests and nobody will pay any attention to them.

A product with a well defined API set can be tested completely. On the other hand a product without defined APIs and organic connections between components has an infinite test surface. You would like to tell your boss “We tested 90% of the testable surface of the product and 99.7% of tests are passing.” You don’t want to have to say “We ran out of test automation time. There aren’t many tests and they don’t run reliably. They mostly pass most of the time.” If your product is designed with good testability in mind, you can assert product quality with high confidence.

Testable software wins the debugging battle.

With a testable design your test automation can be highly diagnostic. You not only know what when wrong. You know where. You can plug tests in at every level of the product and quickly diagnose failures. Without a testable design you could spend hours or days pouring over logs and network sniffs with the product in the debugger.

It is easy to take testable software apart when you find a bug. Integration testing will find bugs the lower level testing didn’t. When this happens you want to be able to isolate the variables and quickly determine the faulty component. This is easy if the software is designed with this problem in mind. Software that isn’t designed to be testable often can’t be taken apart at all. You can’t apply the scientific method to bugs. It can be a nightmare to reproduce problems and a guessing game about how to fix them.

Components should be easy to replace with emulators. A loosely coupled design with complete and locked interfaces is easy to emulate at various levels of the product. These emulators are a cheap way for test to isolate components and developers can use them to verify changes in interface touching code. You can deliver the emulators to third parties to allow them to get started coding against your API long before you have a running version.

How the battle ends

The professional software tester must evolve to compete in the battle for the global marketplace. The ones that will win are the ones who can make the leap from finding bugs to measuring quality. In order to measure quality the first battle you have to fight and win is the battle for testability. Once that battle is won the war isn’t over. But there is one less mine field to negotiate at least.

Winning the war

Getting testers involved too late the software process turns products into dangerous and costly minefields. Focusing on testability from day zero keeps testers from turning into cannon fodder. If you can push testability back into your product from day zero, you will be a hero to all the testers on your team and to the entire product. The difference between a tester and a Test Architect is a subtle one. Start thinking like an architect now, it’s never too soon.

Stay tuned for specifics on measuring design testabily, where to concentrate your BVT automation and more.