So What Should A Test Case Look Like?

From Accountant To Scientist

The Braidy Tester — Wed, 03 Aug 2005 20:45:44 GMT

In many of my posts I have alluded to the automation stack my team is building, but I have not provided...

From Accountant To Scientist

The Braidy Tester — Wed, 20 Jul 2005 20:32:09 GMT

In many of my posts I have alluded to the automation stack my team is building, but I have not provided...

New Team System Stuff - 2005-05-25

Rob Caron's Blog — Thu, 26 May 2005 01:35:27 GMT

Visual Studio Team System

Yesterday marked the one-year anniversary of the public announcement of...

re: So What Should A Test Case Look Like?

humbugreality — Thu, 26 May 2005 00:38:04 GMT

Michael: It sounds like you are asking "What are you verifying?" One of my next posts will talk about how we've (mostly) decoupled verification from execution, allowing the test case to be completely ignorant about what is being verified and what that verification involves. This lets us reuse a single test in multiple contexts: it can be a Build Verification Test by turning off much of the verification, or it can be a comprehensive test by running it multiple times and having Execution Behavior Manager (http://blogs.msdn.com/micahel/archive/2005/05/25/OneMethodToRuleThemAll.aspx) run through the various implementations of each method with all verification enabled. Memory leaks can be caught or not by switching that part of the verification model on or off. Any number of microbehaviors (e.g., drawing a rectangle, then deleting it, then drawing a new one, then moving it to the proper location) could be injected along the way to fulfilling the semantics of an LFM method. What a test case is testing depends largely on its context.

Many other aspects I think are simply setup details - which is to say context in a different context. <g/> Running the test case at multiple resolutions could be as easy as looping through a series of resolutions, setting the screen to each resolution and then executing this test.

Separating the semantics of a test (what it does) from its implementation (how it does it) from its verification (what it is checking) can complicate matters when you want to know all of this at once. Maintenance of each piece, however, is much simpler. Also, we have found that once you get your head wrapped around dealing with each of these separately (which takes some doing, for sure!) it is pretty rare to actually need to smush everything back together again.

re: So What Should A Test Case Look Like?

humbugreality — Thu, 26 May 2005 00:12:55 GMT

Tester: We've decoupled (well, mostly anyway) verification from execution, so the test case doesn't have to worry about verification at all. I'll post more on that soon!

re: So What Should A Test Case Look Like?

Michael Bolton — Tue, 24 May 2005 08:51:10 GMT

The attributes that you mention are worthy, but shouldn't the list contain something about the risk that you're trying to test for, or the theory of error that might be at work?

I mean: this test could identify a memory leak (especially if run a sufficient number of times); it could run through big sets of values; it could trigger a crash. But suppose that the user that you're simulating tried to create a rectangle, and then changed her mind and deleted it. Could this test make sure that the SceneElements object can handle the deletion properly, free the memory correctly, display the data in an appropriate way on display resolutions of various sizes? Maybe not--and that would be okay. I see to some degree what the test is supposed to DO. I find it harder to see what the test is supposed to TEST.

Cheers,

---Michael B.

re: So What Should A Test Case Look Like?

Tester — Sat, 21 May 2005 22:24:59 GMT

How is this test confirming for expected results? What if the rectangle isn't drawn?
What if the app blows up, where do you test for unexpected failures?

re: So What Should A Test Case Look Like?

AndrewSeven — Fri, 20 May 2005 19:25:45 GMT

"Now let me explain the technology we are building to help us reach that vision."

The Friday cliff hanger! :D