I am filled with solutions : Testability

Screencast: 8 minutes with Visual Studio 2008. Easy unit tests and code coverage.

SaintD — Sat, 16 Feb 2008 09:05:00 GMT

This week I spent my blogging allowance on creating a screencast. This video shows you how easy it is to create unit tests and measure code coverage in VSTS 2008.

Video: Creating unit tests and measuring code coverage in VSTS2008

Get good returns on your test automation dollar.

SaintD — Sat, 03 Nov 2007 02:42:00 GMT

Get good returns on your test automation dollar.

Building test automation is like building a house.

Test automation seems like such a good idea. Convert something time consuming and boring into an automated process. It really is a good idea but you have to watch out for some pitfalls. Just as you wouldn’t start building a house with the paint, you shouldn’t start your test automation effort without thinking it through. Don’t fall in love with scenario tests just because they mimic the customer experience so closely.

Your customers really want you to build on a solid foundation. As professional software testers our role is to channel the customer. We are the ones they rely on to ensure their point of view is heard in the software creation process. We have to think hard about the best ways we can fulfill this vital mission. We have to make sure the software they will be using is built well and that means putting our automation efforts on a solid foundation.

There never seems to be enough time for all the automation you want. If we spent all the time it would take to automate every possible test, the product would never ship. If we don’t do any automation or have a very small set of automated tests we may not ship the product we were hoping for.

A strong foundation of good test automation has a big impact. Setting a house on a good foundation can mean the house stands the test of time. You are passionate about software. Plan your automation strategically to use your passion for the customer experience the best way possible.

Automate the lowest level of tests first; just like you would start with a foundation for a house. Product designs flow down from user scenarios. It’s tempting to approach test automation the same way. Experience shows this isn’t the best approach. Start from the bottom up if you want a fast, maintainable and balanced test automation set.

Start with unit tests and work your way up.

The levels of testing from low to high are unit, component, API and then scenario tests. Scenario tests are also known as end to end tests (E2E) and often use the exact same UI the user will. When you are writing your test automation there are a lot of reasons to make sure you have a good set of unit tests first.

Low level automation has a high return on investment.

Low level tests don't require broad product knowledge. Therefore you can get started automating much faster. Your ramp up time is reduced as compared to scenario testing. You may not even have to understand what the product does. You just need to understand what functions, components and APIs are supposed to do at a granular level.

You can write a lot of it early. Because of the relative simplicity of low level tests you can create a lot of tests early in the product cycle. You can have a few component tests done the same day a developer drops you a working build. Because you can deliver automation so quickly you can close the feedback loop for developers in a timely fashion. They can get a good gauge of how their work is progressing and what the problem areas are. This is vital to a healthy product cycle.

You can run it lots. Because it’s so well suited to running in a fully automated environment you can run the low level automation with a very low “tax”. Developers can run it whenever they make code changes. You can run it every time you build. You can run it as part of every test pass. This is a big advantage when trying to gauge the quality of your product.

It can enable you to create scenario tests robustly and more cheaply. The final advantage of a good set of low level tests is that they will inform your creation of the scenario tests. You can sand the rough edges off the product before you even try to test scenarios. You can make sure that people on the test team have an in depth understanding of the way the product is put together and works internally. You will discover if your product has API level problems or hazy specifications that need to be cleared up much earlier which will in turn make your scenario tests more focused.

API testing can often be completed automatically. If you have a well defined API that’s spelled out in technical language you can write lots of tests very quickly. You can use several approaches to generate your tests and test data. You can often create thousands or tens of thousands of API test cases in a very short period of time using tools.

Unit, component and scenario tests are a good match for early automation goals so do them early.

Scenario test have one set of advantages. They are highly customer focused. There are a nearly infinite number of interesting tests. The bugs they find are important to customers. Low level testing has another set of advantages. These advantages make them better suited to automation runs than scenario tests. There are several reasons for this.

Low level tests are more deterministic than scenario tests. Have you ever run a scenario test only to find out that it doesn’t run the same every time? You can end up endlessly tweaking and fiddling with them. You do hacky things like adding hard sleeps and resets into the code. Low level tests almost never have these kinds of problems. They run the same way every time. You can run those tests thousands of times in a row and get the same result. When you are looking at reports of the health of your product, you want to have the most reliable tests be the most numerous in the automation runs.

Low level tests are simple enough to be trusted. Nothing is worse than having the automation fail and having to carry out lengthy manual tests to figure out where the problem is. Developers will often point the finger at your test automation. Lower level tests are better in this regard. It’s easy to prove that the test is correct. It’s also easy to realize when the test is wrong and you can spend your time making the test better rather than doing lots of manual work to diagnose the problem.

Low level tests can be fast enough to run many pre-check in. Ideally your developers can run your automation before they check in changes to component code. If all you have are complex scenario tests that require a lot of setup you won’t get many people to run them. If your tests are more or less self contained and can run in a couple of minutes you are in much better shape. Developers love running tests that find bugs quickly and save them making bad check-ins. They can focus on the subset of tests that really matter to them. Since the developer will have a personal relationship with their testers they will be more willing to trust the automation and they know who to go to for help.

Low level tests are highly maintainable. When you have to hand off your test suite to someone else, low level tests are superior to scenario tests. They are usually made up of fairly elemental code. They can be put in the debugger and understood easily. Scenario tests are seldom this maintainable. This is also a big advantage when product code changes. Picking which tests to keep, which tests to fix and what new tests need to be created is much easier when considering the low level tests. When you go back to them after weeks or months you won’t be struggling to understand what you were thinking when you wrote them.

Low level tests are highly diagnostic. Sooner or later your tests will fail. It’s important to understand why the failure occurred. Product bugs need to be understood by developers quickly and completely. Test automation bugs needs to be fixed correctly by testers. When the time comes to diagnose failures lower level tests are much easier to deal with.

Scenario tests should be automated last (but not least).

After all these points you may think that I don’t like scenario tests. This isn’t true at all. But the reasons they are important shouldn’t be used as an excuse to skip the lower level automation. If you have to make cuts in automation, cut scenario testing and do it by hand. The scenario tests deserve to be run but they aren’t the highest automation priority.

Scenario tests don’t require expertise to run manually. This is not true of the unit, component and API tests. Anyone with a scenario script can run theses tests. If you have to cut automation, it’s a lot less risky to cut tests you can run by hand. These tests can be run by testers who don’t have strong automation skills. The testers won’t need in depth black box knowledge of the product. They can often be outsourced with reliable results.

Scenario tests are not diagnostic. Even if you manage to automate a large number of scenario tests they aren’t great at isolating bugs. It’s true your product should contain logging and diagnostics to make this process better. In practice these self analysis systems often have holes and blind spots that are hard to predict. Analyzing scenario results can be a very complex process. In a product with no or poor self analysis tools you will find yourself very frustrated when the scenario tests fail. You want your highly diagnostic tests to be created as early as possible. Save the less diagnostic tests for later.

Scenario tests are not easy to maintain. This is because of the complexity you must build into them. Even the simplest scenario touches on wide swaths of the product and may rely on external dependencies that are hard to deal with. Because of this complexity these tests are expensive to maintain. You can mitigate a lot of this problem with a solid library set. However the library will require expertise to create and a budget to maintain as well. You can make the problem better and move it around a little but in the end you can’t escape the complex nature of these tests. Get the maintainable tests running first, they won’t consume your life later while you are working on the harder stuff.

Scenario tests are best built on a solid framework of lower level tests. The best scenario tests are ones that build on a library of knowledge and library of well tested working code. If you can encapsulate your features into lower level tests you can set yourself up for success with the scenario tests. Your understanding of the product will be more complete and this will lead to better tests. You can use more reliable automation to setup the parts of the test not core to the particular test case. By keeping the UI driving code to a minimum you make your test suite higher quality and easier to maintain. You need to have this foundation built up before you can leverage it.

Think ahead and prioritize high return investments to keep your automation house in order.

By concentrating on the "top level" customer scenarios you often end up with a small set of tests that are expensive to run and keep. This means you should prioritize automating the lowest level of tests first.

Learn to do this well so you can become an effective advocate for quality and ultimately for the customer.

Edit: Removed "Do Unit Testing First" sidebar. I think it's the culprit for messed up formatting some people are seeing.

Why the world needs you to be a Test Architect

SaintD — Sat, 27 Oct 2007 03:01:00 GMT

Why the world needs you to be a Test Architect

How testers can be the heroes of the product lifecycle.

A band of testers

The modern software process can be like a minefield. Products slip. Products ship with poor quality. Shipped products can be expensive and cumbersome to maintain. Running services can require armies to keep them in good order.

The professional software tester runs head long into these and other mines all the time. Product specs aren’t complete before code is done. Code complete dates slip but ship dates are cast in stone and test has to “suck it up.” You have been there. You know how the bombs that go off during production unerringly land on the test team.

What testers need is a way to clear the mines from the battlefield. When products are on track it’s a glorious thing. Testers report quality metrics with authority and certainty. Big cuts can be made with confidence. Ship dates aren’t just ghost ships hanging out on the horizon.

Focusing on testability is the way to remove the majority of the mines. A product designed from the ground up to be testable is superior in every part of the lifecycle to one that is grown organically.

How focusing on testability helps win the war

Enforced Quality gates in the design phase win the first battle by saving time and money across the release

The way win this battle is to institute and stick by quality gates for every milestone. Especially the M0 (design) milestone. It’s tempting for teams to skimp on the gates early and make them solid later. Don’t make this strategic mistake. Ensure your designs complete with well defined interfaces that are treated as contracts.

You first gate should be to insist on good loosely coupled designs. Loosely coupled software is easy to test. If you speak about loosely coupled designs to developers and designers they all nod in agreement. The battle isn’t over though. All too often the designs are incomplete, incoherent and lead to hard to test code. Leaving interfaces to be cooked up organically on the fly by developers is a sure way to cut test out of the loop and build bugs into the product. Halt the process until the designs and interfaces are fully cooked to win this engagement.

The second gate is to insist that the test automation plan should flow naturally from the design. You should be able to read the product designs for a component and be able to imagine how you would easily and cheaply create thousands of automated tests. One example of a good design is a locked down XML API. You can use XSLT to automatically generate tests very efficiently. An example of a bad design is a complex product that can only be automated from the UI level. You have to create a lot of complex and brittle automation to test even a fraction of the product.

The cheapest bugs to fix are the ones you find early. Many sources including the excellent tactical manual Code Complete by Steve McConnell point out just this. Not only that but design bugs realized late in the process are the most expensive to fix. The first bugs you should find and fix are designs that are hard to test.

Cost Effective Automation wins the battle against exploding test matrixes

A well designed product doesn’t need extensive UI automation. Time after time I find testers intent on automating the end to end customer scenarios. They are convinced that running the computer programs in exactly the way customers do will find most of the bugs. Since the bugs we find after we ship are usually due to the customer doing something wacky this is natural. If your products are designed in a loosely coupled way with automated testing in mind, you will be able to automate thousands (maybe even millions) of test cases and work out all the corner cases before the UI is even applied to the product.

You want your BVT automation to be simple, fast and reliable. An extensive suite of tests is no good if nobody trusts them. BVT automation is held to a higher standard still. A highly testable product enables BVT automation that is dead simple. Even a developer can understand (and therefore trust) it. A product that’s difficult to test will be impossible to create lightweight BVTs for. You will blow your test automation budget on a few simple BVT tests and nobody will pay any attention to them.

A product with a well defined API set can be tested completely. On the other hand a product without defined APIs and organic connections between components has an infinite test surface. You would like to tell your boss “We tested 90% of the testable surface of the product and 99.7% of tests are passing.” You don’t want to have to say “We ran out of test automation time. There aren’t many tests and they don’t run reliably. They mostly pass most of the time.” If your product is designed with good testability in mind, you can assert product quality with high confidence.

Testable software wins the debugging battle.

With a testable design your test automation can be highly diagnostic. You not only know what when wrong. You know where. You can plug tests in at every level of the product and quickly diagnose failures. Without a testable design you could spend hours or days pouring over logs and network sniffs with the product in the debugger.

It is easy to take testable software apart when you find a bug. Integration testing will find bugs the lower level testing didn’t. When this happens you want to be able to isolate the variables and quickly determine the faulty component. This is easy if the software is designed with this problem in mind. Software that isn’t designed to be testable often can’t be taken apart at all. You can’t apply the scientific method to bugs. It can be a nightmare to reproduce problems and a guessing game about how to fix them.

Components should be easy to replace with emulators. A loosely coupled design with complete and locked interfaces is easy to emulate at various levels of the product. These emulators are a cheap way for test to isolate components and developers can use them to verify changes in interface touching code. You can deliver the emulators to third parties to allow them to get started coding against your API long before you have a running version.

How the battle ends

The professional software tester must evolve to compete in the battle for the global marketplace. The ones that will win are the ones who can make the leap from finding bugs to measuring quality. In order to measure quality the first battle you have to fight and win is the battle for testability. Once that battle is won the war isn’t over. But there is one less mine field to negotiate at least.

Winning the war

Getting testers involved too late the software process turns products into dangerous and costly minefields. Focusing on testability from day zero keeps testers from turning into cannon fodder. If you can push testability back into your product from day zero, you will be a hero to all the testers on your team and to the entire product. The difference between a tester and a Test Architect is a subtle one. Start thinking like an architect now, it’s never too soon.

Stay tuned for specifics on measuring design testabily, where to concentrate your BVT automation and more.