Quantum Testing

re: Quantum Testing

Thayu — Sat, 11 Aug 2007 08:37:23 GMT

Michael: Thanks for the thoughts Michael. Yeah, I do understand perfection is not possible; and there have been bigger failures in the satellite domain which have been hushed up than the Ariane 5-501. But we don't hear of them too often, do we? You saw the flak they received! That was enough for them to get the subsequent models working just fine... ;) And yes, I realize that every field has bugs - what I meant was, they seem to push "bugs" or whatever of that nature under the convenient rug of "specifications". As you said, they do rate the ICs based on their limits. I understand perfection in the absolute sense is impossible, but hey, isn't everything relative? And I do know that some hardware we use is extremely buggy. My final year hardware project which seemed to have bugs everywhere we looked, is being used in the defence industry now! How? We just pushed them from the column of limitations to the column of specifications! ;) I say we should still strive for perfection, so that even when we do fall short (as we definitely will) we are at least way ahead of the pack! And yeah, the point is not having "near-zero defects", it is about having "near-zero complaints about defects"! Sorry if I seemed to have gotten carried away. Thanks michael for bringing me back to earth! ;)

Pete: Thanks Pete, what you say are issues for us to contend with that I hadn't given much thought to. But then again, they are issues only because of the durability of software. It is only because our software stays around for ages that it is subjected to these stresses due to change. So I prefer to look at this as our software being put in a totally new environment and being expected to survive or function properly - kinda like throwing an eagle in the water and expecting it to swim like a dolphin! The eagle remains the same - it is the water that kills it. There is nothing wrong with our software as such - it is just that they are not designed for all environments. And they shouldn't be! If they function perfectly in the environments they are intended for, that should be sufficient.

Also, after a little thought, I realize that by far, we face the almost unique problem of hackers. Almost anyone with access to the internet has access to data on how to hack/crack, whereas such technical know-how isn't exactly common in other fields like hardware. And to summarize, it is perfectly alright to have bugs as long as they aren't found - or, I quote "Do what you want son, so long as no one finds out. Problem is, someone usually finds out. So ensure it can be found out only when it can make no difference to you"... In short, no one cares as much about bugs in Windows 95 now, do they? It's lifetime is almost over, and all the changes Pete mentioned might not make much difference to it simply because no one uses it! ;) So we'll just have to put in bugs that nobody can find for ten-twenty years. Kinda like encryption - anything can be broken given time, but hey, what's the point of breaking a message (or software) when it's lifetime is over? Maybe that is what we have to focus on?

re: Quantum Testing

Pete. — Thu, 09 Aug 2007 01:08:56 GMT

Michael: Thanks for the link to AsmL. I've put it on my reading list.

Thayu: "The software domain is unique in that it is the only field that doesn’t face the inevitable problem of “wear and tear”. "

I respectfully disagree. Software quality naturally degrades over time in a number of ways:

1) Software often runs on newly created hardware that it cannot possibly have been tested with.

2) Software often depends on other software and/or services to operate, and those often change over time.

3) Customers often find new ways to use or abuse (e.g. hackers) software over time.

4) Protocols, rules, laws, needs, and expectations change over time.

5) Basically, the "things" that define the quality of software are in constant flux.

We would do better to think of software as a living entity that needs to be serviced regularly in order to maintain its integrity and quality.

Software is a service, and our thoughts around quality should reflect that.

Pete.

ENTROPY...

jib — Wed, 08 Aug 2007 21:22:22 GMT

Once I was also like Thayu :)

Nothing wrong with it, and some consideration points:

Spaceships (and derivatives :) also are buggy :). My favorite story: http://en.wikipedia.org/wiki/Ariane_5_Flight_501

I still believe, that future software should ALLOW bugs, but still MUST be functional. It's nature of nature (sorry:) - nothing is prefect, but still everything works and looks VERY GOOD. Another favorite on this topic VERY FUNDAMENTAL NATURE LAW, which changed a lot in ourlife): http://en.wikipedia.org/wiki/Quantum_indeterminacy

Making long story short - you can't simultaneously and precisely tell speed (energy) and position of a quantum particle. Precise can only one of those parameters not both.

As opposite to Thay I want to tell - we (and our software) shouldn't bet perfect. Nobody is. Don't get me wrong - our software should be perfect on the task, including UI, reliability, gotchas etc. But internally it may be buggy (let's call it "imperfect"). All industries producing physical items have defects. Especially electronics. To remind those "Yield rate" - all the chips are tested and rated: bad, 2Ghz, 2.2 GHz, 2.3 GHz. Difference is that chips are very reliable indeed, if they passed are - Only inThere is no industry with zero defects. Also a product can be labeled "near zero defects" only if it has quite limited on/of type functionality which ALL can be verified (like mentioned computer chips), any other product have "defect tolerance" - and still providing value.

Also I like this: http://en.wikipedia.org/wiki/Analog_computer#Timeline_of_analog_computers

It may bring "other industries is a grand idea" too when investigated deep enough :)

It's amazing how unstable things create very stable systems.

re: Quantum Testing

Thayu — Wed, 08 Aug 2007 13:45:57 GMT

Talking about the hardware industry, let us see how electronic design is different from software design. Let us take Ted, a Transistor level designer, Gary, a Gate level designer and Cody, a Component level designer. Ted puts transistors together and decides their arrangement on an IC to reduce space consumption, has to worry about the wiring to reduce capacitance and delays, and so many more factors, to design even a basic gate. So Ted takes his time and does whatever he has to do and gets a beautiful AND gate which functions perfectly and is quite small. Gary, who is a gate level designer, tries to put together logic gates in the best way possible to achieve basic functional blocks like a shift-register or adder and so on. Gary has to decide whether he is going to use just NAND gates throughout to facilitate manufacture, or whether he is going to use a mixture of gates to improve efficiency, and so on. Cody, the component level designer, will use shift-registers, counters, timers, adders and so on to realize some complex circuitry like the control for traffic signals. What is the pattern we see here? The efficiency that Ted can achieve in a complex circuit such as the traffic light control is much higher than what Gary can obtain, which is in turn higher than the efficiency Cody can achieve. Why then isn’t the industry infested with Teds? The answer is that Ted will take much longer than Gary, who will in turn take longer than Cody to finish the job. So it is the Cody’s of the industry who get the work. Where do Ted and Gary go? They go to where efficiency is imperative. Ted would be perfectly at home in Intel, where their processors have to be small and efficient. If that job was given to Cody, we’d be back to the days of computers taking up an entire building. Gary would go to consumer electronics companies which have to produce efficient products, but produce them quickly too. The Cody’s can be found in consumer electronics companies too, but ones that have to be nimble and worry more about being able to adapt to a rapidly-changing market which might not wait for them to come up with efficient products. So how do they individually ensure quality? I mean, Cody knows nothing about electromagnetic effects at the transistor-level, so he can’t even design a decent gate, let alone complex circuitry. Gary can design using gates, but he can’t design them. What happens? As you probably know, Gary doesn’t have to design gates because they are readily available. And Cody doesn’t have to design adders and so on, because they are readily available as well. And as they know that the gates or adders they buy are perfect, these guys can go ahead and design using them with confidence. As long as their design is right, they know that the end product will be so too. This is a case of abstraction at its best. How can we use this in the software field? How can we build on perfect code to write more code that is equally perfect? By employing the use of libraries? That is a good idea, as long as all entries to the libraries are perfect, i.e. bug-free. And we need to have multiple levels of libraries, so that the necessary level of abstraction is available to the coders. But this is more or less what we are already doing! Why then do we still have bugs? What do they do different in hardware? Let us take another look at what Ted, Gary and Cody are doing. Ted takes transistors and defines connections between them. Transistor design keeps changing due to constant research and Ted doesn’t have to worry about that because that is abstracted. Manufacture will handle their implementation. So Ted just decides which transistors to use and how they are connected. Since each transistor is a well-defined entity, he just handles the selection and more importantly, their arrangement and wiring. What about Gary? Again, he worries only about the selection of gates and wiring/interconnection. Cody? No difference there either – selection of functional components and wiring again. So now, what does wiring represent? The existence of the most basic interface. So in software terms, these guys just pick the functions and interfaces between them. And what do they test? The perfect functioning of the interfaces between two functions. And coupled with pre-existing data from the components, they can easily say how much current each circuit can take and so on by just taking the weakest component, i.e. if one of the gates will break down when it’s input voltage exceeds 3V, by just looking at the circuit, they can say what the maximum input can be. Thus, the limitations are decided by identifying the weakest component, or the weakest link. Why doesn’t something like that exist in software? We can see that we repeatedly use certain functions over and over again. And yes, they have been collected in libraries. Why then isn’t our code just a collection of function calls? Why aren’t we just writing automation to test all interfaces/interactions between various functions? Any answers? I’m just curious. That there would be too many interfaces to test can't be the answer, because it would just mean we haven't abstracted enough. We know how much wiring exists in even the most basic IC. If they can design despite that, I believe we can design software that is just as complex... And no, saying that even a declaration is like a function call and so, all our code is just a collection of function calls already isn't the answer. It would mean we should've been able to test them all then. And as I said, saying there would be too much to test just means we haven't abstracted enough - and that we aren't building enough on past successes (read "perfect code") to build more.

I think the reason is probably that we are more interested in coming up with more features and more capabilities than with perfecting what we already have. When JAVA was first introduced, it's amazing collection of built-in functions probably spurred people to explore more features than to write more perfect code with just slightly improved functionality. And that must have been only because the market responded to the features rather than to perfection. So maybe perfection isn't necessary as of now, and maybe it never will be. Looking at it this way makes me feel that when the time comes, the quantum leap might be simply a drastic decrease in the amount of "cool" new features being added, and the time saved being spent in improving our libraries or our languages, leading to better code. Maybe, and maybe not.

I am not wholly satisfied with this answer and think there must be something more. And I also think how ever many theories we come up with, we can never know for sure until we implement some of them and see what happens; but we need some theories first, and that was just one that I could think of. Let us look for more!

re: Quantum Testing

humbugreality — Wed, 08 Aug 2007 01:58:23 GMT

Thayu: I agree that one reason most software is rife with defects is that software makers do not have sufficient incentive to do otherwise. Thanks for the thoughts!

re: Quantum Testing

Thayu — Tue, 07 Aug 2007 08:13:18 GMT

Well, there are a couple of issues that have to be looked at first over here. The first is that software is unique, in that it is different from most fields. Second is that, maybe instead of using that uniqueness to spur us on to perfection, we are using it as slack to take it easy, or focus on other areas. What is this difference? Simply this:

The software domain is unique in that it is the only field that doesn’t face the inevitable problem of “wear and tear”. Software doesn’t fail simply because of aging. Data loss may occur in storage, but there is no degradation of software quality as such. Also, we don’t have to worry about damage during production. The CDs we ship our software on may be damaged, but that is not our problem either. We don’t have to worry about replacing damaged software in the sense that electronic goods have to be replaced on damage. We simply have to send another copy of the software. There is no extra cost in that, unless you count the cost of the CDs - a negligible cost.

I believe it is because we do not have enough of an incentive to strive for perfection. We can release software patches for bugs, and in the rare event of a recall, the losses aren’t too huge. Take the hardware industry on the other hand. If there was a fault in the circuit design, the entire product is useless since they can’t release “patches” to correct them. And as manufacturing is extremely expensive, the losses due to a small flaw could be in the billions. Which means they can’t afford even a small error. And that is the reason why hardware design has evolved around testing, and not the other way around. The same applies to almost all other fields. Take the satellite field for example. A small flaw means tens of billions of dollars down the drain. Almost nothing can be salvaged. And they have to make sure that if it crashes, it does so without harming humans or creating political troubles. This has led to testing being extremely important. And you wouldn’t believe the amount of time they spend in calculating the number of standbys to be used. Because if they use too few standbys, the entire operation might be a failure. Use too many, and they’re wasting cash in the form of fuel to launch it up in space. Stress testing especially is taken to extremes that the satellite might never face. And then when that is over, they can’t use the same piece since it would have been weakened by the tests. So they manufacture another matching piece. We wouldn’t have to even think of this in software - we can make infinite copies. An example to illustrate how rigorous testing is with satellites would be the case of one of NASA’s satellites. After a long time of testing, they finally certified a communication satellite to be good enough to stay in orbit for five years. After more than ten years, it was still functioning perfectly, and that is great, because component-aging and probability of failure increases exponentially with time! I'm not sure if the satellite is still up there, since this data is taken from a slightly old book. And this was for a communication satellite - just machinery. If humans were going to be in there, it would be a lot more rigorous I guess. We needn’t go to that extreme, but we can improve our testing, design and coding too. How, is the question. Aren’t we doing our best? Aren’t we the best brains in the industry? We are Microsoft, after all… So if the fault doesn’t lie with the people, it is the technique. What should we change? How can we design based on testing, and not test based on design? Or is there something else we should look at? I think the time is fast approaching when we ought to hold brainstorming sessions regularly to address this issue.

Why aren't we perfect? Is it just a lack of necessity? As evinced by bug bars and triages, we are allowing bugs to slip in. We only remove the ones that have to necessarily be removed. Maybe if it is a companywide necessity that there are fewer bugs than presently accepted, some of us will find some amazing solutions! ;)

Also, we needn't look at an industry that has near-zero defects. As just mentioned above, we can ignore wear and tear problems. We can ignore aging. What do we see when we remove those? We come face to face with the painful possibility that maybe we just don't have enough incentive to go the distance! There are differences in methodology of course, and we can analyze them, but till now most of us wouldn't have thought this could be simply because we weren't pushed far enough, hard enough! Painful thought - and maybe true.

P.S: Please do correct me if I'm wrong or am saying something stupid anywhere, because as I said, I've just finished college and haven't been in Microsoft for even a month!

re: Quantum Testing

humbugreality — Mon, 06 Aug 2007 17:12:31 GMT

Thayu: Learning from other industries is a grand idea. This is often a useful way to get past a block you are encountering. Do you know of an industry which has near-zero defects?

re: Quantum Testing

Thayu — Mon, 06 Aug 2007 14:34:33 GMT

Why don't we try to take some pointers from some other industry? From some industry where people find almost no defects. What helps them achieve that? I am an electronics engineer, and I have seen that almost all major electronics design revolves around testing. If you can't test it, you don't design it. As simple as that. Maybe we need a revolutionary change not just here in Microsoft, but in the field of software itself. I am fresh out of college and lack experience in both fields, but I can't help feeling we are missing something. Every field has to have quality control and assurance, and that is nothing but testing in some form or another... Why don't we learn from them? Any suggestions?

re: Quantum Testing

humbugreality — Sat, 04 Aug 2007 02:21:33 GMT

Michele: Aren't Michael and RST grand? Thanks for reminding me about defocusing. I will ponder on that for awhile.

re: Quantum Testing

Michele — Sat, 04 Aug 2007 01:28:38 GMT

Hi Michael: I was recently fortunate enough to attend the Rapid Software Testing course, delivered by Michael Bolton. One of the most helpful things (of the many that were given) was delivered in one word DE-FOCUS. Perhaps you are focusing too hard on the goal you want to achieve. Pull away from it and do something completely off the wall. Make up your mind that you will not look at the whole wheel, but just one of the spokes... maybe the biggest part of the answer to your quantum leap lies in the small, dark corner of the process.