David Kline : Testing

Microsoft Tester Center open for business

DavidKlineMS — Mon, 22 Oct 2007 23:31:02 GMT

Alan Page announced today that the new Microsoft Tester Center site was live. This is very cool. What I love about this site is summed up in three words... it's about testing.

I encourage anyone interested in learning more about testing and what testers do to check out the new site -- especially the whiteboard videos.

Enjoy!
-- DK

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.

Knowing when to Ship: Part 2 - Interpreting Quality Metrics

DavidKlineMS — Mon, 27 Aug 2007 23:39:40 GMT

In the first installment of this series, I detailed a list of key metrics used to determine when a product is ready to ship. Today, I'm going to talk about how I interpret these metrics.

Avoid out of context metrics
The first thing to remember when looking at quality metrics is to look at the data in groups (at least pairs). Individual metrics can be misleading if taken out of context. Let's consider pass rate, for example. Deciding to ship a product, based on pass rate only is a scary proposition. Without understanding how much of the product was covered by the tests, it is easy to have a false sense of security about the product's quality and ship it too soon. The same is true for code coverage. Just because code was called does not mean it behaved correctly. Always consider the pass rate as well.

Warning sign(s):

High pass rate (~100%) and low code coverage (<50%)
High code coverage (ex: 90%) and low pass rate

Beware of large amounts of code volatility / churn
To me, the exception that proves the rule of "avoid out of context metrics" is code volatility / churn. Let me take that back... there is still contextual data to consider when looking at code volatility: time. When looking at code volatility, be mindful of where the product is in it's schedule. Is it in the initial development phase? In stabilization? The middle of the final test pass? Code changes. It's not a bad thing, its how bugs get fixed. The important thing to remember is that the more change in the code, the more testing is required to ensure the product meets the quality requirements.

Warning sign(s):

Large amounts of code change close to release (including betas)

Beware of low code coverage percentage
Though I spoke briefly about code coverage in the "avoid out of context metrics" section, it deserves some additional (pardon the pun) coverage here. In my earlier post, I mentioned that I consider code coverage to be a measurement of risk. It is this reason that I advise careful tracking of the code coverage of your product.

At MEDC, I said that I code that has not been covered cannot have it's quality quantified. I personally consider uncovered code to likely be buggy. Even if the code has been reviewed carefully by a group of very talented developers, there may be bugs hiding in the code. Race conditions are a prime example of a bug that arises in code that "looks right", but does not actually behave correctly.

Additionally, as Ryan Chapman has stated in his MEDC performance sessions, the larger a binary is, the longer it takes to load. If functionality is not necessary for the application, as possibly evidenced by the code not being covered, it acts as "excess baggage" and can lead to longer load times.

Performance data
Interpreting performance data is, to me, an interesting topic. When I approach performance, I have the following questions in mind:

On what device was this data collected
How long did the test run / how many iterations of the scenario
What reporting technique was used (fastest or "gymnastics" style)

In my mind, the most important of these questions is the first one (what device). When comparing performance data, it is very important to compare results on the same device. For comparing performance, I define "device" to be the pairing of hardware plus operating system. The same hardware running different versions of the operating system (ex: Windows Mobile 5 and Windows Mobile 6) can display noticeable performance differences.

I talk in more detail about performance testing, including reporting techniques, here.

It is also useful to couple code coverage data with performance results. By running the tests a second time, under code coverage, it helps to verify that the intended scenario is actually being measured.

Warning sign(s):

Code coverage data does not match intended scenario
Measuring micro-benchmarks using too few (< 10000) iterations or too short (< 1 sec) total run time

Take care,
-- DK

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.

Knowing when to Ship: Part 1 - Quality Metrics

DavidKlineMS — Mon, 30 Jul 2007 23:22:19 GMT

One of the most common questions I hear towards the end of a product cycle is "are we ready to ship?". The answer to the question depends on many factors, most importantly the status of the quality metrics.

Today, I would like to spend some time talking about some of the key quality metrics. In later parts of this series (I expect a couple more), I will go into some more detail including: interpretation of quality metrics and how to identify warning signs.

Pass Rate
Pass rate is one of the most straight forward metrics. This metric measures the percentage of tests which passed during the test run.

Bug / Defect Counts
When looking at bug counts as a quality metric, it is important to include resolved as well as active defects. Active bugs (the number of known defects in the product) are a well understood metric.

The more interesting number, for me, is the count of resolved (not yet closed) bugs -- especially those resolved as "fixed". For each of the bugs that have been fixed, a change has been made to the product. Every change introduces a risk of some part of the product being broken. It is very important to keep a close eye on the resolved bug and to make sure each one of them is re-tested to ensure that the bug was indeed fixed and that no other issues arose due to the change(s).

One related metric, which I consider to be part of the Bug Counts is the percentage of reactivated bugs. These are the ones that, once resolved, were reactivated because the original issue was not completely fixed. As a rule, I only reactivate a bug if the issue described within was not fixed. Any additional issues related to the fix, such as a new bug being introduced, should be tracked in a separate entry in the bug database.

Mean Time To Failure (MTTF) / Mean Time Between Failures (MTBF)
One of the most important aspects of testing a product is placing it under stress. A while back, I described my two classifications of stress tests (long haul and short haul) and mentioned that the results of long haul stress testing tend toward the product's MTTF/MTBF metric. These results should be tracked between stress test passes (looking for changes that cause robustness to decrease) and measured against the product specification.

Performance
As with stress results, it is important to track performance data between test passes and measure against the product specification. I discuss measuring performance here.

The above metrics directly measure the quality of a product: how well it works, how much work is left to do, how long it will run and how fast. The next two I will talk about are often discussed in quality meetings and status reports. While not direct measurements of quality, they are still very important as they are measurements of risk.

Code Coverage
Code coverage is one of the best measurements of the testing being performed on a product. If the code coverage data is very low (below 50%) the product is not being adequately tested. Any portion of a product that is not being covered by testing is a risk. Blocks of code that are not being tested cannot have their quality properly assessed. If the code has not been exercised, there is no way to be sure of the quality. Even when code is reviewed carefully (and appears correct), it does not always do what the developer intended.

Code Volatility / Code Churn
By measuring the amount of change in the product from one build to the next, the amount of testing required to validate quality can be assessed. The larger the amount of change, the more testing is required to ensure the quality of the product. As a product gets closer to the target ship date, the changes in the code should decrease, leading to a stable product. I like to think of the weeks leading up to releasing a product like watching an airplane land. A smooth, gradual descent leads to a very comfortable landing. A steep descent is significantly less comforting.

Take care!
-- DK

Disclaimer(s):
This posting is provided "AS IS" with no warranties, and confers no rights.

Stress Testing for Fun and Profit

DavidKlineMS — Tue, 10 Jul 2007 21:20:46 GMT

For me, one of the most enjoyable parts of being a test developer is writing stress tests. Stress testing can be a lot of fun and, contrary to it's name, a good stress reliever (for the test developer). During the most recent MEDC, I spoke briefly about stress testing and defined it as coming in two forms: "Long Haul" and "Short Haul".

Long Haul Stress
In long haul stress, the test is designed to emulate customer scenarios. If a scenario is expected to be executed three times per second under normal operation, a long haul stress test will perform that scenario on multiple threads for a long time (days/weeks). By using multiple threads and long run times, the Mean-Time-Between-Failures (MTBF) / Mean-Time-To-Failure (MTTF) for the component can be approximated.

In addition to simultaneous threads repeating the target scenario, long haul stress test often up the ante by manipulating other aspects of the system. Available storage (hard disk / flash) and RAM are common targets for increasing the stress level of a long haul test.

Short Haul Stress
Short haul stress, or "load testing" is my personal favorite type of stress testing. Short haul tests are designed to be extremely stressful and to run for a relatively brief amount of time (24 hours). If a long haul test exercises a scenario three times per second on, say, 10 threads, a short haul version of that same test may perform the scenario 30000 times per second on more than 1000 threads. The same system resource tweaking options used in long haul testing are often used in short haul testing as well.

To me, short haul stress is not really scenario based, however. I tend to treat short haul testing as an opportunity to be as hard on the component I am testing as humanly possible. I have been known to throw "everything but the kitchen sink" into my stress tests; every functional area of the component, multiple copies of data files, dozens of threads performing similar tasks, forced errors, etc.

At the start of this post, I asserted that stress testing can be fun. While writing short haul stress tests, I have occasionally noticed some odd glances into my office (maybe I should close the door more often) when I watch the app run and let that maniacal laugh creep out. Of course, the best reward of stress testing is to bring a component's developer into the office to show off the latest bug and hear a sound rivaling the most beautiful music... "Your app does WHAT?!?!". YES! I got one again. :)

Take care!
-- DK

Disclaimers:
This posting is provided "AS IS" with no warranties, and confers no rights.

Testing performance

DavidKlineMS — Fri, 08 Jun 2007 22:41:43 GMT

A couple of weeks ago, I described the test development priorities for the .NET Compact Framework team. As part of that discussion, I stated that performance should be tested in parallel with the other forms of testing (unit, customer scenarios, etc). Today, I would like to spend some time talking about performance testing.

General considerations
When testing performance, there are a few key things to keep in mind.

First, it is important to measure the intended scenario. In the past, I have seen a number of tests that were intended to time a data processing operation (ex: sort algorithm) that inadvertently included the time to load the data from the file system.  As a result, the performance data being reported was quite inaccurate.

Second, since today's operating systems are multi-tasking, it is recommended to test performance on as clean of a system (fewest running processes) as possible and to run the performance test multiple times to ensure accurate data. Depending on the scenario being timed, the jitter (variance in results) can be significant and important to take into account when reporting results. I will talk a bit more about performance reporting shortly.

Lastly, I have found that most meaningful performance measurements for applications I have written are when the application is in sustained operation. What I mean by this is that any Just-In-Time compilation has already occurred and I am measuring the typical run time performance of my scenario. Most often, code is Just-In-Time compiled once per instance of the application, so the performance impact is felt only the first time the method is called. There are some exceptions to this rule of thumb, and I have found that in those cases, even when pre-JIT compiling the scenario, the results are reasonably accurate. What are these exceptions? Memory pressure and moving the application to the background. In each of these cases, the .NET Compact Framework's Garbage Collector will run and may discard JIT compiled code and be forced to recompile when the situation is resolved (memory has been freed, the application is brought to the foreground). In both of these cases, the timing loop will reflect the time spent freeing memory and re-JIT compiling.

Macro-benchmarks
Macro-benchmarks are performance tests for long duration scenarios. By "long", I mean that the scenario takes a second or more to complete. When testing macro-benchmarks, I typically run the scenario a handful (10-50) times in a loop. Because the scenario takes a significant amount of time to run, the time spent looping does not significantly impact the measurements. When the timing loop is complete, dividing the time spent (number of elapsed ticks) by the number of loop iterations results in the scenario's performance measurement, as shown in the following example.
// run the scenario once (to avoid timing the initial Just-In-Time compile) Scenario(data); // timing loop Int32 iterations = 10; Int32 start_ms = Environment.TickCount; for(Int32 i = 0; i < iterations; i++) {     Scenario(data); } Int32 finish_ms = Environment.TickCount; // calculate the performance Double average_ms= (Double)(finish_ms - start_ms) / (Double)(iterations);
Macro-benchmarks are often based on customer scenarios and improving their performance can often lead to significant gains in customer satisfaction. At times, making improvements to macro-benchmarks requires changes to the scenario implementation. To address this, I highly recommend measuring your scenarios as early as possible in the development of the product. That way, if the performance does not meet the requirements, there is time to write and test the new implementation.

Micro-benchmarks
Micro-benchmarks are performance tests on the small scale, measuring very short duration operations. To get an accurate measurement, performance tests need to run the scenario a large number of times (I typically run my micro-benchmark scenarios 10,000 or more times). Also, it is important to keep in mind that, unlike macro-benchmarks, the time spent looping can significantly impact the performance measurements. To minimize these impacts, I recommend what I call "playing optimizing compiler" and partially unrolling your timing loop. The example below updates the macro-benchmark example for micro-benchmark testing.
// run the scenario once (to avoid timing the initial Just-In-Time compile) Scenario(data); Int32 iterations = 1000; Int32 callsInLoop = 10; // timing loop Int32 start_ms = Environment.TickCount; for(Int32 i = 0; i < iterations; i++) {     Scenario(data);     Scenario(data);     Scenario(data);     Scenario(data);     Scenario(data);     Scenario(data);     Scenario(data);     Scenario(data);     Scenario(data);     Scenario(data); } Int32 finish_ms = Environment.TickCount; // calculate the performance Double average_ms= (Double)(finish_ms - start_ms) / (Double)(iterations * callsInLoop);
Since the scenario is very short, this test runs the code 10,000 times and partially unrolls the loop (1,000 iterations of 10 calls). This significantly minimizes the impact of the time spent looping. In one example scenario I tested, the time spent when in a tight loop (10,000 iterations of 1 call), was reported as 1 millisecond. When using the timing loop shown above, that time was reduced to 0.95 milliseconds - 5% faster than previously reported. With additional loop unrolling (ex: 500 iterations of 20 calls) we can further improve the accuracy of the measurement. Of course, there is a point of diminishing returns when continued unrolling becomes unrealistic and the improved measurement accuracy is no longer significant.

Reporting performance results
I mentioned results reporting earlier. When I report performance results, I use one of two methods: raw speed reporting and what I call "gymnastics" reporting.

In raw speed reporting, I run my performance test multiple times (the exact number depending on whether or not I am timing a micro- or macro-benchmark) and keeping only the fastest result. This approach helps to factor out the sometimes subtle differences in results when running on multi-tasking operating systems (ex: the scheduler runs a background task) and is closer to the maximum throughput of the code.

In "gymnastics" reporting, I again run my performance test multiple times, but this time, I discard the extreme results (fastest and slowest) and average the remaining data. The resulting data is closer to the typical performance that the customer will see during everyday use of the product.

Take care!
-- DK

Disclaimers:
This posting is provided "AS IS" with no warranties, and confers no rights.

Alan Page on Root Cause Analysis

DavidKlineMS — Mon, 04 Jun 2007 20:44:26 GMT

I have worked with Alan off and on for a very long time now. When I heard that he gave a Lightning Talk at STAR East, the first thing that came to mind was "I wish I could have seen that". Last week, he posted what he talked about (and more?). I very much recommend giving his "Bug prevention in five minutes" post a read. He presents a great technique for root cause analysis (RCA) -- including one of my favorite analyses (I don't want to be a spoiler here, but it involves lack of sleep).

Take care,
-- DK

Disclaimers:
This posting is provided "AS IS" with no warranties, and confers no rights.

Testing Priorities

DavidKlineMS — Fri, 25 May 2007 01:47:00 GMT

Last time, I defined testing as the 'art of mitigating pain'. What I did not talk about is how to prioritize your testing (pain mitigation) efforts.

On the .NET Compact Framework team, we use the following set of test development priorities.

Unit Tests
Build Verification Tests (BVTs)
Customer Scenarios / Integration Tests
Conformance Tests

Performance, stress and security testing should be performed in parallel with the testing listed in the list above. My basic rules are to start performance, stress and security testing as soon as there is something to test and continue testing throughout the product cycle. The phrase I keep in mind is "measure early, measure often". I will discuss performance and security testing in the near future. Security testing I will leave to the experts and would like to reiterate my recommendation to read Hunting Security Bugs, by Tom Gallagher, Bryan Jeffries and Lawrence Landauer (Microsoft Press, ISBN: 0-7356-2187).

Unit Tests
What are unit tests? Unit tests are the simple tests that developers perform to ensure, at the most basic level, that their code behaves correctly. These tests typically consist of executing some code once with valid data, once with invalid data and verifying the proper behavior. A simple example is to pass 12 and 4 into function which divides the first value by the second and verify that the result is 3, then calling the same method with 7 and 0 to verify that the proper exception (DivideByZero) is thrown.

Some developers perform unit testing using the debugger to step through their code and monitor the state of the system. While this is a great approach while developing, I much prefer formalizing the unit tests into scripts or executables that can be automated and run as part of the daily acceptance testing.

To help make this easier, Visual Studio 2005 (Visual Studio 'Orcas', for Smart Device projects) provides a very cool Unit Test development feature which examines your project and generates test methods in which developers add the appropriate test code.

Build Verification Tests (BVTs)
Build Verification Tests are designed to validate that a basic functionality of a new product build is functioning correctly. BVTs are not exhaustive tests, they are the front line test that must pass before additional testing can be performed.

Build Verification tests are more complex than Unit Tests and cover more of the code. Wherever possilble (always?), BVTs should be automated so that they run consistently each and every time.

Customer Scenarios and Integration Tests
If a product cannot be used by customers, the product will likely fail in the market. With that in mind, I assert that Customer Scenarios are the most important tests to write. Bugs found by customer scenario tests should be prioritized very high.

What are customer scenario tests? These are the tests that are written to behave as much like the target customer as possible. For example, if the product is a class library (like the .NET Compact Framework), customer scenario tests are applications that use the class library to solve a 'real world' problem. Most often, these tests are also integration tests bringing together two or more features (ex: web services and file system) and verify that they behave correctly when used together. It is entirely possible for each feature's tests to pass when run in isolation and fail when other features are added to the test application due to resource contention, memory pressure, etc.

Conformance Tests
Conformance tests are feature specific tests that expand upon the unit and build verification tests. Conformance tests come in three flavors: positive, negative and boundary.

Positive tests exercise the product using "acceptable" data. The result of a positive test is that the product correctly performs the requested task and returns the proper response (ex: 4 + 12 = 16).

Negative tests (also called error condition tests) exercise the product using "unacceptable" data. These tests are designed to ensure that the product correctly handles erronious situations (invalid data, server not responding, etc). To me, these are some of the most enjoyable tests to write.

Boundary tests exercise the boundaries of a product. If a product's specification states that values between 15 and 173 are allowed, boundary tests cover values such as 14, 15, 16 to ensure that behavior with data around the lower boundary is correct. Boundary tests are a mixture of positive and negative tests.

Take care,
-- DK

Disclaimers:
This posting is provided "AS IS" with no warranties, and confers no rights.
The information contained within this post is in relation to beta software. Any and all details are subject to change.

What is testing?

DavidKlineMS — Tue, 15 May 2007 02:02:58 GMT

At MEDC 2007, I had the pleasure of delivering a session entitled 'Real World Testing of Managed Smart Device Applications'. While developing the presentation, I spent some time thinking about a key theme of the talk -- what is testing.

I define testing as the art of mitigating pain. Whenever a bug is found after a product has shipped, it hurts. It hurts the customer in loss of productivity (while the bug is investigated / worked around) and it hurts the developer (cost of investigating and fixing in the field, delays work on new versions). It costs much much less to identify and fix bugs during development.

When I think of testing as pain mitigation, I think of the auto industry and their crash tests. Crash testing is designed to ensure that the least amount of harm comes to the car's passengers in the event of an accident. When I keep this image in mind, it drives home (no pun intended) the need for quality tests and high levels of product coverage during testing.

Take care,
-- DK

Disclaimers:
This posting is provided "AS IS" with no warranties, and confers no rights.