PART 1 in a series of posts on Load Test Planning
As part of my work inside Microsoft, I have to interface with several different groups of people, from sales to account management to the technical delivery people, and also to different product group teams. I have learned some very valuable lessons from these conversations. The biggest single lesson I have learned is to not only ensure that everyone is speaking the same language, but to ensure that we also all understand the reason we are even speaking. In this blog series, I will attempt to show (at a high level) the thought process I use and some of the steps I and my former testing team follow when we work on customer engagements.
Please note that you will see me refer to several different types of resources throughout the series. This is a critical part of successful performance load testing since it is impossible for only one or two people to know enough to pull this off. My old team has a very simple slogan: “We test WITH you, not FOR you.” That phrase sums up one of the most important reasons the engagements are so successful. It takes a team of skilled people to effectively design and execute performance load tests. I will also tell you that the people I have the privilege of working with all have several years of experience in their respective fields, which is also a key factor in success.
I find it critical to always ask the questions of the final goal(s) of a testing effort before doing any other work. If this question cannot be accurately answered, or if the answer does not make sense, then I start drilling into the specifics to get more clarification.
I recently bought a video series from O’Reilly books called “Software Architecture Fundamentals”, delivered by Neal Ford and Mark Richards. In one of their segments, they talked about understanding the requirements of a project and they gave a great example of how you can change the entire dynamic of any engagement by asking the right question (Sorry Mark, but as you said in the video, I would rather re-use your example than make a new one). Please note I am paraphrasing here:
“The design team of the F-16 fighter plane was asked by the Air Force to design a plane that would fly at Mach 2-2.5. At the time this was very difficult to achieve. The team struggled a lot. Finally they went back to the Air Force and asked ‘Why does it need to go so fast?’ The Air Force said ‘Because we need to be able to escape when we get into bad combat situations.’ So the design team took that knowledge and offered an alternative. They designed a plane that had high acceleration and high maneuverability. The plane would not go Mach 2, but was still very successful because they realized that the acceleration and maneuverability were better than overall speed in those situations.”
Here is the list of things I start with for every engagement I do. Sometimes it can take a few days to answer just these questions. Other times the customer can hand me the answers in their “ready to Use” state:
What are the GOALS/ OBJECTIVES of this testing effort?
A goal or objective is a “desired result or outcome” of an activity or endeavor. Defining this up front sets the stage for every other piece of work you will do. You need to make sure the goal(s) are realistic, and that they possible to obtain. They should also align with the team’s skill sets, the business partner’s needs, and the overall project. Here are a couple of goals to consider:
These goals all seem pretty good at first. However, I bet some of you are thinking that they are either too vague or that the first two are the same, or that there is some other problem.
The first one wants to figure out what load will cause the system to break. The second one wants to understand how the system will behave at a pre-defined load. The third one wants to find out how fast the client is. While they might all be easier to understand if we changed them a bit, you’ll see that I will clarify them in the next step.
What are the SUCCESS CRITERIA for this testing effort?
Now that we have goals, we want to know “What determines if we have successfully achieved the goal or not.” These are the success criteria. Continuing with the list of goals above, we can add the following:
What are the METRICS used to measure the success criteria?
This is the meat of the data collected. Metrics are concrete values obtained through specific tooling as part of the testing. The values are usually from the load test rig and/or from Performance Monitor collections or other monitoring tools. However, we still define them in the test plan in a generic fashion. Please note that I am only doing metrics for the FIRST goal above.
Looking at the above examples, you may notice that there is a big problem with one of my data points. Remember that I said I was going to remove the first Success Criteria for the first goal above? Here’s why. When I am trying to find the breaking point of a system, one of the things that I am always trying to find is “What throughput can the system handle?” Throughput usually contains metrics like requests/second or transactions/second, etc. If this is the number I am trying to determine, it should not be one of my success criteria. The success criteria in this case should be listed as “Obtain the maximum TPS for CompleteTheSale without the other criteria reaching a fail point.”
Think of it this way. The first goal is to define the acceptable behavior of the system and see how much load (throughput) it can handle. The second goal is to define the load (throughput) and see what the behavior is.
The bottom line for all of these is that they should be well defined, they need to make sense, they need to be measurable and the efforts and testing needs to be repeatable.
I will visit the considerations for designing a load pattern and how to model that load.
Thanks Geoff for this article. Well written, practical and clearly coming from field experience.