I am filled with solutions : return on investment

Why doesn’t anybody ask me to code a unit test in an interview?

SaintD — Sat, 19 Jan 2008 00:52:00 GMT

Why doesn’t anybody ask me to code a unit test in an interview?

Interviewing people for your company is one of the most important jobs you will do. You need to make the best call you can in a limited amount of time. The job interview isn’t a 100% accurate tool for predicting success, but it’s the best one we have. Sadly across the industry our interviewing techniques are dated. Ideally we have an up to date toolbox of interviewing techniques. Improve your interviewing skills with these three concepts.

Pay attention to the candidate.

This may seem like simplistic advice. There are levels of attention, strive to be totally present in the interview.

Get out of your normal chair and take notes.

When you are in your work chair, your body and mind are ready to work. That means checking email, instant messages and the web. Even if you avoid these obvious mistakes while conducting and interview they will still be pulling at you.

Reprogram your attention by just moving. Sit in your own “guest chair”. I like to sit on the desk while candidates are working at the white board. I am not hovering over them, but neither am I lounging while they work.

Adjust your questions to challenge but not overwhelm the candidate.

As you gage the experience and expertise of a candidate, try to keep them working. You don’t want to totally stump them and you don’t want to bore them with your easy questions. It takes a little work, but good interviewers can make all their interviews challenging.

If you are sure it’s a no hire make sure the candidate leaves happy.

You are going to interview a lot of people and hire a few. That means you will cut loose more people than you hire. If you have a good sense the interview is going poorly switch gears. Ideally the candidate leaves you office feeling like you challenged them and they did a good job. This might mean giving some real softball questions to weak candidates.

There are some good reasons to do this. In a team interview you may be the only “no” in the loop. You don’t want a candidate that doesn’t know a specific technology to give up when they might be a star in other areas. For people who are just completely under-qualified you want them to feel like you treated them fairly. That way they don’t badmouth your company and recommend their friends interview with you.

Ask technical questions that are relevant to the job.

Software jobs tend to ask a lot of the same questions that were useful 20 years ago. I don’t know how many people are still asking candidates linked list questions. If a tester working in my organization is writing test code with linked lists I would be shocked. I expect testers to leverage high level code to get a lot of tests done efficiently.

Your interview should evolve with technology.

If your team is programming ASP and JavaScript then ask the candidate to solve a typical problem from your world using up-to-date tools. To some extent coding is coding, but asking twenty year old questions isn’t good. It makes you look like a dinosaur and it might make a strong candidate look weak because they learned a different language than you in college.

We have to solve messy problems at work, ask them messy technical questions.

Being able to write efficient, elegant C code is a great skill. But the kinds of “computer science” questions I see a lot of have many problems. First, people who studied them in college might have memorized the algorithms and seem good, but then fail to be able to solve real problems. Second, people can and do study up on common questions. You don’t want someone with such a limited toolbox. Third, they don’t map well to the problems you are solving every day.

Really good interview questions have many “correct” answers and a good candidate will be able to show off a variety of skills solving them. You will also be able to spot general areas of weakness better and delve into them.

Don’t rely on pass/fail questions.

There are a lot of questions that require a leap of intuition to solve. Just because someone didn’t make the leap in the limited amount of time for the interview doesn’t mean much. It’s ok to ask some very short questions that assess a general skill level. Just don’t rely on questions with a “clever” answer.

A good example of a skill level question might be “write a sql select to join these two tables.” It’s short, and it’s pass fail. A common example of a “leap of intuition” question is a question about testing light bulbs you can’t see with switches you can. I won’t give away the answer but there is no systematic approach to solving it. You just have to “think outside the box” and come up with the one and only answer the interviewer is hoping for.

Those questions are poor predictors of problem solving ability. Some people get them right away, others who are very smart never do. Make sure your questions have a way to keep making incremental progress with a systematic approach. After all, that’s what we do in our jobs every day.

Leverage diversity.

If you work at Microsoft then the formal interview process here helps a lot with this. If you interview for another company, make sure you are leveraging the diverse talents on your team for interviews. Even if your company has a good process, make sure you implement it well.

This means technical interviews should be a minimum of four hours for the candidate and they should see at least 4 people with diverse questions and ways of thinking.

Cover “soft skills” equally with technical skills.

Asking questions like “tell me about a time when you…” are really good predictors of future behavior. Make sure someone on your team is going to be asking these questions. All technical interviewers in an interview will get you people who are technically great, but probably lacking some core attribute your team prizes.

Make sure soft skill questions get equal time with technical ones.

Probe for weaknesses and hand those off to the next interviewer to expand on.

You will probably only spend an hour with a candidate. Try to make a note of what you perceive weaknesses are and work with the rest of your team to probe the candidate more deeply. Sometimes this flushes out a weak candidate. Sometimes the follow up questions are met strongly by the candidate and you can stop worrying about that area.

Communicating with your team during the process is important. Don’t hand a candidate off until the next person has feedback of some kind to go off.

Face your own biases.

If a candidate doesn’t seem to understand your questions you have to ask a tough question. Is it a weak candidate or just someone I am not in tune with? Differences in culture can magnify even the smallest communication problems. When you perceive a weakness in a candidate be honest with yourself and make sure the weakness isn’t a bias in yourself.

A candidate that doesn’t understand your questions may just be lacking cultural context to interpret your body language. A candidate that doesn’t seem forceful enough or who seems too forceful may just be triggering your biases. I work with some excellent people from various cultures who don’t conform to the norms I am used to.

Don’t rob yourself of excellent talent for stupid and possibly illegal reasons.

Create a winning team by paying attention to how you interview.

Take a hard look at your interviewing practices. It’s vital we get this right in order to build strong teams. Strong teams deliver great software and that’s what we all want.

Special Thanks to Patent Analyst David Andrews for collaborating with me on this essay.

Add Round Trip Testing to your toolset

SaintD — Fri, 04 Jan 2008 23:22:00 GMT

Add Round Trip Testing to your toolset

Test automation authors have to balance a lot of factors to deliver good automation. You want to create high value tests and deliver them early in the process. Ideally you have a short feedback loop with developers. One of my favorite tricks for closing the feedback loop is round trip testing.

Round trip testing is simply letting the product handle its own complexity. You concentrate on writing simple tests that use two or more calls into the product.

A simple calculator example

Addition is a simple way to explain. Here is some sample code to show how round trip tests work in practice. Imagine a calculator class with a static “DoOperation” method that takes two doubles and an enum indicating the operation to run. Here is a test class with 3 round trips. (In practice it is a good idea to create three tests, but I compact them here for brevity).

Code

[TestMethod()]

public void DoOperationRoundTripTest()

{

double a = getRandomDouble();

double b = getRandomDouble();

double accuracy = 0.00000001D;

double answer = Calculator.DoOperation(a, b, MathOperation.add);

Console.WriteLine("{0} {1} {2} = {3}", a, MathOperation.add, b, answer);

double roundTrip1 = Calculator.DoOperation(answer, a, MathOperation.subtract);

Console.WriteLine("{0} {1} {2} = {3}", answer, MathOperation.subtract, b, roundTrip1);

Assert.AreEqual(roundTrip1, b, accuracy);

double roundTrip2 = Calculator.DoOperation(answer, b, MathOperation.subtract);

Console.WriteLine("{0} {1} {2} = {3}", answer, MathOperation.subtract, b, roundTrip2);

Assert.AreEqual(roundTrip2, a, accuracy);

double roundtTrip3 = Calculator.DoOperation(roundTrip2, a, MathOperation.subtract);

Console.WriteLine("{0} {1} {2} = {3}", roundTrip2, MathOperation.subtract, a, roundtTrip3);

Assert.AreEqual(roundtTrip3, 0, accuracy);

}

Analysis

There are three round trips in the code. The first leg of each trip is adding two numbers. The first round trip is subtracting one of the numbers from the answer. You expect the result to be the other number you added due to the commutative property of addition. The second round trip is the same as the first one, but with the numbers in the other order. The last round trip is longer. Both numbers are subtracted in sequence from the answer and you expect to end up at 0.

Notice that the data are not hard coded. We call the random function to generate random data. You get a little extra free coverage by selecting random data. You might find some boundary bugs for conditions you didn’t realize existed. The second advantage to random data is that you don’t have to do anything complicated to create and store the data. You just make something up on the fly. Be sure to log the data and answers you are getting. When the random data fails you can reproduce the problem from the logs. If you can’t generate some kind of random data you may not really have a round trip.

The next thing to notice is that the verifications depend completely on the application to do the “heavy lifting”. This test code doesn’t need to know how to add or subtract numbers. It only needs to know the round trip should work. Addition may seem trivial but this technique works with lots product code. If you look closely at most applications you can send data on a round trip and verify it comes back correctly.

Lastly, due to overflows and rounding errors, the static method getRandomDouble() method needs some thought. Otherwise you will overflow the calculator when you aren’t planning too. Also due to computer rounding an accuracy threshold is set.

Other simple round trips

One very simple round trip is to just add a lot of data to an API and then count how many elements are in the store. One developer told me that he ran that test on a library he was dependant on and found that the library was dropping every 42^nd data entry. Testers had run five to ten inserts at a time and didn’t find the bug or if they did, they couldn’t reproduce it and it didn’t get fixed. A simple round trip test spotted the error and proved which module it belonged to.

For another example imagine a program that has a 3D globe. You click the screen and the program converts the location on the globe to latitude and longitude. The program then looks up the country or ocean at that coordinate. Testing this program can be tricky because of the translation of mouse clicks to latitude and longitude and then another lookup to the countries and oceans. There are two sets of round trips for this program.

The first one is the mouse click to lat-lon. This program had a function to put a marker at any coordinates (lat-lon) on the globe. The marker (for the tests) was converted to a small dot that way a special color not used to color the globe. The mouse test was like this:

1. Create a random latitude and longitude.

2. Put a marker on that spot.

3. Dump the 3D scene to a BMP. Check the BMP for the special color. (If you don’t find it, spin the globe a random amount and try again till you do).

4. Call the convertMouseClickToLatLon(x,y) function and verify it matches the random coordinates generated within tolerance.

The second test is to make sure we can ID the countries and oceans correctly.

1. Pick a random country or ocean.

2. Call the getRandomSpotInCountry method (the product had this built in for various reasons).

3. Call getCountryOrOceanNameFromLatLon(lat, lon) function and make sure the country matches.

In the actual program the mouse test passed 10,000 tests in just a few seconds. However the second test was failing about 20% of the time. The developer was able to quickly find the bug when I gave him the logs with the coordinates that didn’t match.

After some hand spot checking we were confident this part of the program was working well. One of the big benefits was the simplicity. The test code didn’t have to know a thing about converting mouse clicks to polar coordinates and then converting those to latitude and longitude. It was a lot like the calculator example. The code just cooked up some very simple random data and put it through a round trip in the system.

Fuzzier round trips

The above examples show pretty cut and dried cases of round trip testing. What about cases where an exact match is pretty much impossible for the test code to figure out without re-implementing the product code?

Say software that deals with postal addresses. For example http://www.usps.gov has a lot of address functions. You can search for zip codes by address, by city and search for cities by zip code. Getting random addresses from zip codes isn’t something the site offers (if you were the tester for this web site, you might ask for a test hook that does to help with testing.)

Still there are some tricks we can use to write simple, fast and diagnostic test automation for this API.

Reversing the round trip

Reversing the round trip means taking data out of the application and transforming it in test code. Then you give the data back to the application and expect it to handle it in some predictable way. The key to a reverse round trip is that you typically only make one call to the product. Not two or more.

You can put in an address on the web site and it will canonicalize the address into a standard form with the zip+four.

One approach that works here is to take an address that is already canonical and degrade it. For example we have a canonical address of

WHITE HOUSE

1600 PENNSYLVANIA AVE NW

WASHINGTON DC

20500-0003

There are several transformations we can do to it, and put it back in the system. For example, remove street descriptor words (AVE, NW) and feed it back into the system. Make sure you get the canonical address back, or an appropriate error. If your address engine supports spell checking (USPS doesn’t) you can randomly mutate the words and see if the system can guess what you mean most of time. Because we are doing a fuzzy match, we might not care if the spelling engine isn’t 100% robust for randomly mutated characters. We could create an advanced heuristic for mutations, but that sounds like complex and bug prone software.

In this example we have reversed the round trip since we started with known data from the system and then changed it in some way. It’s not simple as the calculator example. You have to use your imagination for ways transform the data and get useful, automatically verifiable results back from the system. In this case we expect the system to fix up the addresses and give them back exactly like what we started with.

Looking for other trips

Besides making addresses into canonical versions the site also offers the ability to get a list of zip codes in a city and a list of cities in a zip code. There are a finite number of US zip codes and we can discover the entire list with a little research. Once you have the list your test function could provide the zip codes in a loop and get the city or cities as a result. Then it would feed those cities back into the reverse function and verify it matches our input.

Other trips are less direct. There are 25 zip codes listed for Seattle, Washington today. You could record that number and check that it stays constant across builds of the software. You could also put the system under load and see if the number of results is always 25. Truncated data is certainly a possibility in a web service that’s strained. In fact these types of tests are really good ones to run while doing load testing. They are simple and fast to run so you can do a lot of them and use them to drive some of the load.

C.R.U.D.dy trips

Many data API’s follow the CRUD (Create Retrieve Update and Delete) model. These APIs follow some typical patterns that make round trip testing simple and effective.

· Create: create a new data item (create functions usually return an ID), query the ID and make sure your randomly generated data matches.

· Retrieve: Think about the various things you can retrieve. For example many API’s allow you to search for records and count the results. One way to test this is to create a GUID, input some random number of records including the GUID and they query for your GUID. Verify that the count matches.

· Update: Pick a random record, put some random data into the field(s) and then retrieve the record on every possible key and make sure the changes are sticky.

· Delete: Create a random number of records with some unique tag like a GUID. Delete one. Count how many are left and verify it’s N-1.

Notice that the sample tests above don’t care about the data in any specific way. You are just looking for round trips the product will allow you to exploit and verify. These kinds of tests find a lot of bugs the first time they are run and are valuable to run anytime you make changes to the product. You can also write them quickly and trust them when report failures.

With a little creativity you can create an automated suite of tests with tens of thousands of individual tests that run in a very short amount of time and utilize round trips to keep the tests very simple.

Dealing with complex product logic

Some round trips are so complex you can’t test the results simply. For example Excel has the ability to open CSV files and then save them as XML files. You can write a complex program that understands Excels XML format and can do 1:1 comparisons of the data. If you are testing this feature in Excel you probably should write that complex automation at some point. However, your developer will be more productive if you can deliver some fast, simple and effective automation beforehand.

Simple comparisons don’t always work

Here is the problem. You can’t just do the full round trip and simply diff the results. There are a lot of valid ways to write a CSV file. A couple of solutions come to mind. The simplest one you can deliver today is to just count the rows and columns. Sure, you might miss subtle bugs, but your developer will be able to run that test quickly every time they make code changes and know they didn’t create an off by one error in the transformations.

You could parse the CSV files item by item and compare them using the same rules Excel uses. That’s a reasonable approach, but you can do even better.

Don’t give up on making the product “do your work”

Another approach is to start with a CSV file (you might make it randomly) and have Excel save it as XML. Then load it back into Excel and save it as a new CSV file.

Now load both CSV files back into fresh instances of Excel and use the document object model to compare all the values in each cell. Your test code doesn’t need to know a thing about CSV files (except the part that makes random CSV files for the tests) and you can easily compare the round trips. This code is very easy to tweak to check all the round trips to every format Excel supports. Best of all, you delivered a lot of effective BVT automation without code having to understand any details of the file formats. Now your developer has an effective tool for smoke testing file conversions and you have time to delve into the arcana of each format one at a time.

When you can’t get there from here

You will run into situations where you just can’t create reasonable round trips. You will find that you put data in point A and can’t see it until it’s been seriously mangled and comes out way downstream at point B. My name for this type of software is “The soup factory”. The soup factory takes vegetables in one side and a pureed blend of steaming hot soup comes out the other. It’s great at lunch time, but you can’t reverse the transformation. A lot of information is lost in the blender.

The soup factory problem is that the end result isn’t reversible into the inputs. You put 10 carrots into the factory. How can you tell from the bowl of soup that the carrot content is within spec? You can (with a spectral analyzer and a PhD), but doing so is just not cost effective. The best you can quickly do is taste the soup and see if it’s any good. Sadly your feedback will be vague and difficult to take action on. Worse it won’t be very diagnostic.

A good approach here is to “get inside” and try to do round trips on the components. That carrot slicer should take 10 pounds of carrots and turn them into carrot slices and stems. Stems plus sliced carrots should be 10 pounds unless you have a carrot leak. If you can’t find a way to peer inside the factory you need to have a conversation with your developer. They might not want to change the production code to expose the carrot slicing machine, but maybe they can add some debug code for you.

Round trip testing is a valuable tool to put in your automation toolbox.

There are a lot of effective ways to create high value test automation. Concentrating on delivering the fastest, most diagnostic and robust automation first will help your team deliver high quality products on schedule. Using round trip testing will let the product handle its own complexity and allow you to deliver powerful test automation quickly.

Good luck with round trip testing. I hope it makes your test code faster, simpler and you find better bugs.

Learn to “get traction” as a tester on your team.

SaintD — Sat, 15 Dec 2007 03:34:00 GMT

Learn to “get traction” in your team. A guide for professional testers.

Software development is a fast paced world. We are constantly trying to work to the rhythms of “web time.” Professional software testers have valuable input for the team. The problem is that it can be hard to break free of the daily concerns to make that big impact.

Ideally testers are empowered to make positive changes in the organization and the product. The term empowerment is lampooned in Dilbert for good reasons. It’s a great management buzzword but companies rarely do a good job of putting it into practice. The reason is simple. No one can empower you like you.

You can empower yourself when you know how to get traction. Empowerment can trickle weakly from the top down, but it doesn’t have to. When you know the tricks you can empower yourself.

Make sure your peers are your allies.

A tester’s job is to look at things and perceive where they might be deficient. A good tester may have a habit of breaking everything they get their hands on. Be careful that you don’t break the people you are working the closest with. Turn that critical eye inward and ask yourself how you can strengthen your work relationships with the people you work with. The best way to do that is by having a strong vision of what your ideal world looks like. Hold on to that and “show” it to your peers when the going is rough.

Leaf nodes do all the work.

In the software world all of the power is really in the hands of the worker bees. Think about it. The developer you work with writes 100% of the code for his or her features. The developer’s boss doesn’t write a line of code. The product designer’s boss doesn’t write the specification or make the day to day design decisions that come up. Your boss doesn’t write the test specs, the test automation or run more than a handful of test cases at best. The further you go up the chain, the less direct impact a manager has on the actual code. The Product Unit Manager has almost zero influence on the actual technical work that happens in the product. Wow. You and your “feature crew” have all the power. Even in a non-feature crew model. All you have to do to change the product is convince one developer to do something different.

Energize your co-workers.

If you want to make a change in your area, product or organization you must start by getting the people doing the real work excited about it. If you convince one developer to do less and do it better there is nothing his boss can really do about. If you convince the product designer to make a feature a better fit for customer goals, the VP of your product can’t stop them. Start your great ideas right in your own feature. Get your peers excited about the cool ideas you have. Sell them to the people you work with every day. It doesn’t get easier than that.

Listen to Yoda.

“Always with you it cannot be done.” - Yoda chastising Luke on Dagoba. Don’t be a whiner like Luke. Stop thinking about what you can’t accomplish. It’s probably true you can’t get your entire company to change to a better process tomorrow. That’s not important. Find something you can make a little bit better today and do that. Go talk to a developer and ask them to help you think of ways to make a part of the program easier to test via automation. Have a passionate discussion about the benefits doing less and doing it better. Passion and enthusiasm are contagious. Concentrate on the things you are enthused about. Save the gripe sessions for later or never. Always frame your criticisms as “we can do better and here is my idea how.” It’s easy to complain about things that are wrong, but not usually effective. It’s surprising how often people will adopt your plan if you have one.

Re-factor your process the same way you re-factor code.

If you have some code and you want to factor out all the common stuff you don’t do it all in the same instant. It’s a process of iteration. Move a line here, a function there, a helper object over here. Then test each time you make a change. Think of your process the same way. Don’t try to overhaul it overnight. Pick one thing and move it to a better home. See how that goes. Then move on.

Iterate more, plan less.

Test specifications are a perfect example of where over planning happens a lot. A tester reads the specs and other materials then spends days locked in their office creating a test spec. Then in the review certain sections get ripped to shreds. The tester fixes those and soon the test spec is closed. Later in the product cycle you realize you left out something basic and important. A better pattern is to start with the outline and short explanations. Run this by a peer, your boss, the developer and/or the pm informally. See what feedback stems out of this embryonic document. People will be less constrained by a deep level of detail and give you good big picture feedback. Go back and add some details. Get more feedback. Rinse and repeat. Both techniques will take you a week or two of calendar time. The iterative model will create better documents in same number of person hours and less “work” for you. Look for other places where you can replace planning with iteration. Iteration is very powerful.

Close feedback loops.

Look for places where there is feedback in the software process. A prime one is logging bugs. How long is it on average between the time a developer writes code and you log the majority of the bugs? If you measure this time in weeks you have a feedback loop that needs closing badly. If the developer isn’t getting timely feedback, how can they improve their skills? What if the build process pointed out common coding mistakes? Some development environments have build in code analysis tools. Using them closes the code review feedback loop from days to minutes. Those kinds of tools have improve my coding substantially more and faster than traditional code reviews. This is a good example of doing less and doing it better.

Look for these loops and be ruthless about making them shorter and more meaningful.

Don’t worry about The Process.

Every software development effort has an official document set with The Process. I don’t know a single place that is following that exact process. Don’t worry about that. Concentrate on the actual traditions and workflow that is happening. You can get work done in any framework. Thinking too much about the big picture process will blind you to the major improvements you can make locally today. Just find one thing we can do better now. Can you find a way to do less and do it better today? Go do that. Then tell people how you did it and get them excited about it. The Process will mutate on its own for good or ill. Be an agent for positive change.

Management wants to change for the better.

There isn’t a manager at any level in any company that wants to do worse today than they did yesterday. If you can show a clear path to improvement, management will embrace it. The trick can be showing the path in a way that management is compatible with. If you are reading this article you probably believe that one or more managers you deal with is obstructing progress. I am sure they don’t see it that way. You can still open their eyes if they won’t listen. Here are some tricks you can use to show your managers how to do things better.

Repeat your good ideas, over and over.

The first time you say something like “I read on the web we should do less and do it better,” you may get a lot of resistance. That’s fine. That’s human nature. We aren’t entirely rational creatures. Keep at it. If you are genuinely passionate about doing less and doing it better it should be easy to keep discussing it. Don’t be shy about speaking up. Keep talking about it with anyone who will converse with you on the subject. Bring it up in the hallways and at lunch. It may take months, but your message will sink in. Pretty soon you will overhear people say things like “We need to do less and do it better. How can we accomplish that today?”

Take baby steps.

There are many clichés that come to mind about how to do something big. The journey of a thousand miles starts with a single step, and so on. It’s a true cliché. Try not to get frustrated because you didn’t scale Everest today. If you make one thing a tiny bit better every day, at the end of the year your impact will be huge. Break your tasks down, then break them down again. Just make sure you are going the right direction a little each day. Other people may seem to want to sabotage your efforts. Don’t worry about that either. Their distractions and setbacks are likely to be random. If you are committed to making just one thing better and work on it all the time you will win.

Build momentum from the grass roots.

Remember that all the “real” work happens in the roots of the organization. If every worker bee in your organization is talking about doing less and doing it better, management will naturally fall in line. They won’t have a choice because the real work force will already be doing it. When you have an “aha moment” and realize how things could be better shop it around. Encourage the people you work with to think about it. Ask them if they think it’s a good idea. Ask them how they would go about doing it. Ask them what’s wrong with it. Infect them with your enthusiasm and be willing to be infected with theirs. Try to reach out to people who are your peers but you don’t talk to very often. Have lunch with someone from a different part of your company and discuss your good ideas with them. Get your meme out there. You will know you have won when it mutates, takes on a life of its own and runs away. Time to get started on your next great plan.

When you see how much influence you have you will be shocked.

Testers feel like everything is out of their power. It can really feel like we are at the bottom of the hill and all the crap is rolling down to us. Don’t just whine about it. You can empower yourself when you know how to get traction. If you can do that you move from just running tests to improving the company. That’s a really big change in role, for the better. Don’t forget to put that on your evaluation this year. You can make any project you work on better.

Getting traction can be a subtle art. Think about ways to get traction and go make your product, your company and the world better.

Escape the BVT automation trap.

SaintD — Sat, 17 Nov 2007 00:13:00 GMT

Escape the BVT automation trap.

Create outstanding BVTs on time and on budget.

The build verification tests (BVTs) for your product are important to get right. Automated BVTs are a very powerful tool in your test arsenal. They allow you to know if a build is worth testing in a short amount of time with little or no user intervention. If you tie them to your official build system you can usually have results waiting for you when you arrive to work in the morning.

You need a clear plan for moving from manual BVTs to automated ones. The methods that make good manual BVTs aren’t a good fit for automated BVTs. In fact they are a trap! You need another set of principles to work from.

Don’t fall into the clunky, slow BVT automation trap. Bad automation doesn’t justify the cost to create it. Testers will idled for hours every day while someone tracks down the root causes of failures. If you are in the trap you can lose days of productivity when it matters the most, crunch time. The bait in the trap is complex, customer focused scenarios. It’s yummy and can be irresistible to testers and managers. A good idea at the wrong time turns bad.

Imagine BVT’s that are fast, reliable, trustworthy and useful. This is the ideal that you should be shooting for. This gets you out of the trap where BVT’s try to be all things to all people.

Create your BVTs with the simplicity (also known as K.I.S.S.) principle. Simpler is better when it comes to BVTs. Resist the urge to use complex tools to solve simple problems. Instead use simplicity itself to create outstanding automated BVTs. Simplicity means optimizing for automation goals and leaving other testing goals for other tests. BVT automation goals include things like being fast, maintainable and diagnostic. BVT automation non-goals include core customer scenarios and ensuring features are correct.

BVT automation is a tool.

Just as a hammer is a great tool for nail, BVT automation is a great tool for a specific purpose. It’s not a catch all place to dump all the automation you have ever created for your product. Be vigilant and keep things out that don’t belong. Good BVTs systems have certain qualities in common. Keep your tests simple and keep your BVTs focused on their mission.

A good chance that testing can proceed is enough what the tool is for.

The purpose of good BVTs is a quick way to tell if a particular build is worth testing further. The “worth testing” threshold will be different for every product and change as the cycle progresses. If the BVT system can wave you off a bad build even half the time (50%) it’s probably paid for itself. You reach the point of diminishing returns quickly. In order to have a 80% chance of testability you need to do a ton more work just to get 30% better. Don’t waste time. Predicting possible build breaks is a trap. Observe your product over time and add tests where they will do the most good. Adding a BVT for each detected build break may be like closing the barn door after the cows are gone. Ask yourself if this test will catch future breaks. If the answer isn’t “probably” then don’t add it to the BVT suite.

Fast results are money in the bank.

When you build a new version of your product you want the green light as soon as possible. BVT systems that take hours to run are wasting valuable cycles. When a developer has a potential check in they should be able to run BVTs in a matter of minutes. If it running the tests requires a full build and human intervention in a customized environment you won’t be getting very good leverage on your BVTs. I have seen BVT systems that were so top heavy that a check in late in the game could cost an entire day just to build, get on the test bunch and run a “sanity” pass. Don’t fall into that trap.

An automated test in the hand is worth two in the bush.

You want to deliver BVT automation as early in the process as possible. You can solve a lot of problems with enough time. But when you are delivering BVTs time is the first problem you need to solve for. If you have a great idea for an object model for the product but it will take weeks to code, don’t wait on it to create your BVTs. Go for a few cheap (and yes, possibly dirty) tests you can deliver today.

BVT automation needs to trusted

If your team has no faith in the results they get from the BVTs you have wasted your time writing them. When a BVT fails everyone should believe that the results are valid. You won’t be right 100% of the time of course, but you can come close. If your BVTs are failing on a regular basis because of setup issues or other reasons outside actual product flaws they lose a lot of their punch. If you find yourself in this situation you may need to rethink your BVT design. Get away from complex brittle scenarios and move to something simpler. If BVT failures are catching more problems with the environment, the network of other “noise” you either need new BVTs or a more solid environment or both. Make this a priority or manual BVTs will be the only thing you can trust.

Create fast diagnostic BVTs

Fast BVTs make your automation more trustworthy. You can use the short feedback loop to iterate them until they are rock solid. Make sure BVTs can be run quickly. Then run lots of iterations and verify they are good. Create BVTs with diagnostic results. A BVT that points clearly to a method in code is superior to one that can’t isolate failures. When you detect a break you want to be able to zero in on the offending part of the build and fix it very quickly. Couple highly diagnostic with a quick execution time and you cut the time you need to fix a break to practically nothing.

Create adaptable BVTs

Adaptable BVTs are more trustworthy. They will spend less time being irrelevant. Your product will change out from under the tests as the cycle progresses. Adapt the BVT test suite to these changes quickly. It’s ok and even desirable for an individual test to be brittle. By making the BVT tests as simple as possible they are less likely to go wrong. If the product changes you can easily decide what tests are obsolete and need to go and what new tests need to be created. Complex tests that rely on intricate libraries to interact will idle testers while fixes come in. Fixing a giant test library that was made with assumptions about the product that are no longer true can be time very consuming. If the library owner is on vacation or has left the project you are in real trouble. Keep individualt tests simple to keep the overall suite adaptable.

Create maintainable BVTs

Simple code leads to maintainable code. Make your tests as maintainable as possible. The bugs are clear. The repro scenarios are clear. You can easily prove the automation is working in the debugger. Developers can easily suggest fixes for faulty tests that are easy to understand. When you are on vacation and one of your tests fail you won’t be getting a panicky call or email. If the time comes to hand the product off to another team or the customer to maintain, they can easily run and update your tests.

BVT automation needs to be manageable

An automated system of tests has to have a lot of moving parts. This can make the systems difficult to manage. Help to keep the complexity down by leveraging things you will be doing anyway. Running BVTs isn’t free. There is a cost in hardware, support and time. Make wise use of these resources.

Leverage unit tests

Unit tests are really good BVT tests. Typically all you need to do is get them to run in your BVT system and output reports. If you work with developers up front and support them they will often be happy to write the tests to in whatever harness the test team is using. If this isn’t possible for some reason, it’s still easy to adapt or port the tests. Another key way to leverage the unit tests is to simply create more of them. Once a tester sees one unit test for the code it’s easy to use it as a template to add other tests. Developers often only create a functional test and maybe a negative test. You can improve these by adding boundary tests and tests that randomize valid inputs. This is a simple and fast way to crank out a lot of simple, fast and dependable tests that will last the life of the product.

Leverage simple component tests

Subsets of component tests are really good unit tests. My favorite is a test that checks to see if the component installed correctly. That is a dead simple test to make and a fantastic BVT. Don’t worry about all the core scenarios for the component. Save that for other automation. Just look for quick high level tests that are simple. Checking for performance counters, log entries and configuration files are all good examples. Checking the correctness of those same things is more complexity than the BVT system should have.

Leverage core API functions

If your product has simple to use APIs they make good BVTs. Say the product has a web services XML interface listening for HTTPS connections. A good test would be to check if it’s actually listening on the port. Listening on the port is really core to the way the API works. Enumerating users in the user table is a compelling scenario but save it for your other automation suites.

Will you find the right balance for your BVTs?

Creating a set of BVTs that prevents wasted time is harder than it seems. Prevent customer scenarios from luring you into a bloated BVT system. Customer scenarios are a Good Thing™. You must do them. Just don’t do them at the wrong time when they will cost you more than they should. Create your BVTs with the simplicity principle. Simple BVT tests are the path to having a great BVT system. A great BVT system focuses your team on shipping a superior product.

Keep asking if your BVTs are simple enough. Then make your product awesome with the help of simple, streamlined BVTs.

Get good returns on your test automation dollar.

SaintD — Sat, 03 Nov 2007 02:42:00 GMT

Get good returns on your test automation dollar.

Building test automation is like building a house.

Test automation seems like such a good idea. Convert something time consuming and boring into an automated process. It really is a good idea but you have to watch out for some pitfalls. Just as you wouldn’t start building a house with the paint, you shouldn’t start your test automation effort without thinking it through. Don’t fall in love with scenario tests just because they mimic the customer experience so closely.

Your customers really want you to build on a solid foundation. As professional software testers our role is to channel the customer. We are the ones they rely on to ensure their point of view is heard in the software creation process. We have to think hard about the best ways we can fulfill this vital mission. We have to make sure the software they will be using is built well and that means putting our automation efforts on a solid foundation.

There never seems to be enough time for all the automation you want. If we spent all the time it would take to automate every possible test, the product would never ship. If we don’t do any automation or have a very small set of automated tests we may not ship the product we were hoping for.

A strong foundation of good test automation has a big impact. Setting a house on a good foundation can mean the house stands the test of time. You are passionate about software. Plan your automation strategically to use your passion for the customer experience the best way possible.

Automate the lowest level of tests first; just like you would start with a foundation for a house. Product designs flow down from user scenarios. It’s tempting to approach test automation the same way. Experience shows this isn’t the best approach. Start from the bottom up if you want a fast, maintainable and balanced test automation set.

Start with unit tests and work your way up.

The levels of testing from low to high are unit, component, API and then scenario tests. Scenario tests are also known as end to end tests (E2E) and often use the exact same UI the user will. When you are writing your test automation there are a lot of reasons to make sure you have a good set of unit tests first.

Low level automation has a high return on investment.

Low level tests don't require broad product knowledge. Therefore you can get started automating much faster. Your ramp up time is reduced as compared to scenario testing. You may not even have to understand what the product does. You just need to understand what functions, components and APIs are supposed to do at a granular level.

You can write a lot of it early. Because of the relative simplicity of low level tests you can create a lot of tests early in the product cycle. You can have a few component tests done the same day a developer drops you a working build. Because you can deliver automation so quickly you can close the feedback loop for developers in a timely fashion. They can get a good gauge of how their work is progressing and what the problem areas are. This is vital to a healthy product cycle.

You can run it lots. Because it’s so well suited to running in a fully automated environment you can run the low level automation with a very low “tax”. Developers can run it whenever they make code changes. You can run it every time you build. You can run it as part of every test pass. This is a big advantage when trying to gauge the quality of your product.

It can enable you to create scenario tests robustly and more cheaply. The final advantage of a good set of low level tests is that they will inform your creation of the scenario tests. You can sand the rough edges off the product before you even try to test scenarios. You can make sure that people on the test team have an in depth understanding of the way the product is put together and works internally. You will discover if your product has API level problems or hazy specifications that need to be cleared up much earlier which will in turn make your scenario tests more focused.

API testing can often be completed automatically. If you have a well defined API that’s spelled out in technical language you can write lots of tests very quickly. You can use several approaches to generate your tests and test data. You can often create thousands or tens of thousands of API test cases in a very short period of time using tools.

Unit, component and scenario tests are a good match for early automation goals so do them early.

Scenario test have one set of advantages. They are highly customer focused. There are a nearly infinite number of interesting tests. The bugs they find are important to customers. Low level testing has another set of advantages. These advantages make them better suited to automation runs than scenario tests. There are several reasons for this.

Low level tests are more deterministic than scenario tests. Have you ever run a scenario test only to find out that it doesn’t run the same every time? You can end up endlessly tweaking and fiddling with them. You do hacky things like adding hard sleeps and resets into the code. Low level tests almost never have these kinds of problems. They run the same way every time. You can run those tests thousands of times in a row and get the same result. When you are looking at reports of the health of your product, you want to have the most reliable tests be the most numerous in the automation runs.

Low level tests are simple enough to be trusted. Nothing is worse than having the automation fail and having to carry out lengthy manual tests to figure out where the problem is. Developers will often point the finger at your test automation. Lower level tests are better in this regard. It’s easy to prove that the test is correct. It’s also easy to realize when the test is wrong and you can spend your time making the test better rather than doing lots of manual work to diagnose the problem.

Low level tests can be fast enough to run many pre-check in. Ideally your developers can run your automation before they check in changes to component code. If all you have are complex scenario tests that require a lot of setup you won’t get many people to run them. If your tests are more or less self contained and can run in a couple of minutes you are in much better shape. Developers love running tests that find bugs quickly and save them making bad check-ins. They can focus on the subset of tests that really matter to them. Since the developer will have a personal relationship with their testers they will be more willing to trust the automation and they know who to go to for help.

Low level tests are highly maintainable. When you have to hand off your test suite to someone else, low level tests are superior to scenario tests. They are usually made up of fairly elemental code. They can be put in the debugger and understood easily. Scenario tests are seldom this maintainable. This is also a big advantage when product code changes. Picking which tests to keep, which tests to fix and what new tests need to be created is much easier when considering the low level tests. When you go back to them after weeks or months you won’t be struggling to understand what you were thinking when you wrote them.

Low level tests are highly diagnostic. Sooner or later your tests will fail. It’s important to understand why the failure occurred. Product bugs need to be understood by developers quickly and completely. Test automation bugs needs to be fixed correctly by testers. When the time comes to diagnose failures lower level tests are much easier to deal with.

Scenario tests should be automated last (but not least).

After all these points you may think that I don’t like scenario tests. This isn’t true at all. But the reasons they are important shouldn’t be used as an excuse to skip the lower level automation. If you have to make cuts in automation, cut scenario testing and do it by hand. The scenario tests deserve to be run but they aren’t the highest automation priority.

Scenario tests don’t require expertise to run manually. This is not true of the unit, component and API tests. Anyone with a scenario script can run theses tests. If you have to cut automation, it’s a lot less risky to cut tests you can run by hand. These tests can be run by testers who don’t have strong automation skills. The testers won’t need in depth black box knowledge of the product. They can often be outsourced with reliable results.

Scenario tests are not diagnostic. Even if you manage to automate a large number of scenario tests they aren’t great at isolating bugs. It’s true your product should contain logging and diagnostics to make this process better. In practice these self analysis systems often have holes and blind spots that are hard to predict. Analyzing scenario results can be a very complex process. In a product with no or poor self analysis tools you will find yourself very frustrated when the scenario tests fail. You want your highly diagnostic tests to be created as early as possible. Save the less diagnostic tests for later.

Scenario tests are not easy to maintain. This is because of the complexity you must build into them. Even the simplest scenario touches on wide swaths of the product and may rely on external dependencies that are hard to deal with. Because of this complexity these tests are expensive to maintain. You can mitigate a lot of this problem with a solid library set. However the library will require expertise to create and a budget to maintain as well. You can make the problem better and move it around a little but in the end you can’t escape the complex nature of these tests. Get the maintainable tests running first, they won’t consume your life later while you are working on the harder stuff.

Scenario tests are best built on a solid framework of lower level tests. The best scenario tests are ones that build on a library of knowledge and library of well tested working code. If you can encapsulate your features into lower level tests you can set yourself up for success with the scenario tests. Your understanding of the product will be more complete and this will lead to better tests. You can use more reliable automation to setup the parts of the test not core to the particular test case. By keeping the UI driving code to a minimum you make your test suite higher quality and easier to maintain. You need to have this foundation built up before you can leverage it.

Think ahead and prioritize high return investments to keep your automation house in order.

By concentrating on the "top level" customer scenarios you often end up with a small set of tests that are expensive to run and keep. This means you should prioritize automating the lowest level of tests first.

Learn to do this well so you can become an effective advocate for quality and ultimately for the customer.

Edit: Removed "Do Unit Testing First" sidebar. I think it's the culprit for messed up formatting some people are seeing.