The questions are usually: "Your test is failing. Can you take a look?"; the questions are never: "Your test is passing. Can you take a look?"

There appears to be a misconception that "A good test is one that passes" or "Tests should pass". A few years ago I even had requests to send 100% passing test results to prove that the tests are in good quality. You can only scratch your head on such requests.

I spend most of my time trying to design the test to fail. It's what I'm paid for and what my products - tests - are used for. Obviously, it's easy to write tests that fails, but the challenge is in making sure the tests pass and fail under the right conditions, and that they do it consistently.

False passes are hard to find. They are usually hidden from the regular test passes. There are some mitigation to false passes though:

  1. My tests fail once a while due to product code regressions. Bugs in product code caught by my tests are great affirmation to a certain confidence level that the tests still work. I sometimes joke with my SDE counterparts that they are helping me with "fault injections".
  2. Test both the fail/pass case. This is more common in test driven development, and the technique is called red/green testing. But I found it useful outside of test-driven development too if you can create the environment for the failure cases easily. Since I test a lot of devices and they have varying qualities, they do provide great samples for the failure cases. Testability hooks that injects faults is also another great tool to make sure the tests can fail properly.
  3. I do get bugs on false passes when I'm lucky, but most of these bugs are filed based on code reviews or design reviews.
  4. When in doubt, fail. Test code design has an advantage over product code design that it doesn't have to handle a lot of unexpected behaviors gracefully. Failures are more eye catching and hence helps to expose bugs.

That's why I haven't lost slept over false passes. But I always wonder since we have regular test "passes", we don't have regular test "failures"?