Chris Pratley's Office Labs and OneNote Blog : Quality

More on Quality

Chris_Pratley — Thu, 05 Feb 2004 08:09:00 GMT

In my last post I discussed the subject of shipping quality products (which is really rather different from code quality or stability, or many other measures one could use). And let's not forget design quality - obviously, if a product does not meet the customer's needs or is too hard to use, it doesn’t matter how stable or well written it is. I got quite a few comments on my last post, so I guess this is an interesting topic. For the record, I am not writing a book here - so I am not necessarily exhaustive when I describe bugs or coding techniques. The remarkable thing about Watson that made me want to write about it is that it is a new way to make products better that tries to measure the real world. It is not an excuse to avoid good code architecture, or to not think about product design. One person pointed out that non-crashes can be more annoying than crashes - that's very true - although crashes and hangs are pretty nasty. In fact we're looking into ways to extend Watson beyond crashes and hangs. The Watson guys have a bunch of exciting ideas around that.

I mentioned last time that the goal is to ship a product with the highest known quality, not necessarily with fewer bugs. This is a counter-intuitive concept, so I'll explain a little more. The naïve way to think about bugs is that if you fix a bug, the product is better as a result. The truth is that it probably is better, but you cannot say for certain right after you check in your change that it absolutely is better. You may have inadvertently introduced a different problem by fixing the issue you were dealing with. That kind of bug is called a "regression" - the product quality has regressed (decreased) as a result of trying to improve it.

In the example I gave, fixing a redraw problem in a toolbar button may seem pretty harmless, but in fact something is now different, which might introduce a change somewhere else you did not anticipate. If you doubt this, then you haven’t developed enough software or had enough contact with the people who use your stuff to know that you sometimes make things worse, and you don’t always find out immediately. In the toolbar button example, if you do it by setting a flag on a call to the system which causes the video driver to be called in a different way, that'll probably be fine on your machine, but some fraction of people out there might be using a video driver that can’t handle that particular request, and they go down. Unless you or your test team happen to have that video card and driver, and happen to try this new code out, you will have no idea that your fix to a minor visual glitch has now caused 1.5% of your future users to have a horrible experience with bizarre crashes that seem to have no known cause since their actions are not causing the problem. It may take you months to discover what has happened - months of complaints from a seemingly random set of customers who have no repro steps for their crashes - truly a nightmare. If you fixed this bug right before you thought you were done, then you'll probably find out about the problem from your customers rather than your test team, which is not good. Once you do find out about this other bug and finally track it down to the video driver (if you ever do) you will of course fix it, but it is a little late - the damage is done. Some customer might have deployed your software on thousands of machines, and your fix is going to have to be deployed on all those machines as well. That little bug might cost your support lines and customers more money than they paid you for the software, or that you made from it.

So the goal is to get the product to a known state of quality before you put it out there. To maximize the known quality, the process we use in Office and the rest of Microsoft for the most part is a staged reduction in bug fixing, believe it or not. Naturally during the course of a project we fix just about every bug we find. But near the end of the project, as it gets harder to find bugs and our predicted ship date approaches, we need to make sure we can control the quality of the product in the end game. We begin a process of deciding which bugs to fix and which ones our customers can live with (because they won’t notice them, mainly). Some people will read this and say "I knew it! See, Microsoft guy admits shipping buggy software!". If you conclude that, you did not understand.

Bugs are not all equal - we use a measure called "severity" to gauge the impact of a bug in isolation of how common it is. This is a measure that a tester can apply without having to make a judgment on the importance of fixing a bug. It’s pretty simple: a "Sev 1" bug is a crash or hang or data loss. Sev 2 is serious impact on functionality - cannot use a feature. Sev 3 is minor impact on functionality - feature doesn't quite work right. Sev 4 is a minor glitch - visual annoyance, etc. The severity doesn’t control whether we fix a bug or not. We use another subjective measure - the opinion of a program manager, mainly. For example, if the splash screen for Word has "Word" misspelled, then that is a Sev 4, but clearly is a must-fix.

We know that statistically our testing process flushes out most regressions within about 3 months of the time they were introduced. Since there is such a long lead time, we start reducing the number of changes months before our expected ship date. Of course usually regressions are found within a day or two if not sooner because we test specifically for them, but some of them lurk, as I have explained. If a regression is not found by the special "regression testing" done specifically to try to catch these, then that area of the product may not be revisited for 2-3 months as the test team works on other areas, hence the long period.

Because of this long time that regression bugs can lurk, some months before we decide we can ship the product we begin a "triage" process. We start rejecting bugs (marking them as "won’t fix“, or “postpone”) that have little to no customer impact, and that only a very persistent customer would run into (like, typing 1,000,000,000,000,000 into the footnote number control causes the footnote format to look incorrect - who cares?) The goal is to reduce "code churn". Any sizeable software project is something you have to manage as an organic thing since as I discussed, it is no longer possible to know for certain what effect any particular code change will have. By simply reducing the amount of change, you can reduce the amount of random problems (regressions) being introduced.

As time goes by, we raise the "triage bar" that a bug must meet in order to be worth fixing. That is, bugs need to be a little more serious to get fixed. As a result of this process, fewer and fewer changes are happening to the code base. Time passes, and we can be more confident that new bugs caused by bug fixes are being introduced in much smaller numbers. Finally the number of bug fixes we are taking per week is down to single digits, and then finally no bugs remain that we would consider "must fix". We now have a "release candidate", and we will only take "recall class" bugs. Essentially we could now ship the product, but we leave it in "escrow", testing like mad to see if any truly heinous bug that would make us recall the product appears, and also whether those last few fixes have regressions. So now we have reached a state where by choosing to not fix bugs that we considered survivable, we have minimized the chance that even worse bugs are now in the code instead. We're at maximum known quality.

And of course the inexperienced among us are shocked, because they can't understand why this or that bug did not get fixed. But the wiser ones remember what happens when you buck the process and fix that last bug.

A last anecdote to leave you with. Even re-linking your code (not even recompile) can introduce a crashing bug. A few years ago, we were working on the release candidate for an Asian-language version of Word97. We thought we were done, and ran our last optimization on the build. We have some technology at Microsoft that profiles code usage and arranges the code modules so that they are in the executable in the optimal order to produce the best possible boot speed. After running this process which involves mainly the linker and is considered very safe, we put the code in escrow while the testers tried to break it. And they did - they found that it would crash on some of the machines they had when a certain feature was used. But the unoptimized build did not crash with the same steps and machine.

So we ran the "debug" build (a version of the build that has the same code as the "ship" build you all use, but includes extra information to allow debugging) and it also did not crash. We then tried to debug the ship build - but just running the ship build in the debugger caused the crash to go away. Some developers love this sort of mystery. One of the future inventors of Watson stopped by to get involved. No matter what they did, as soon as they tried to find out what was causing the crash, it went away. We went to get an ICE - a hardware debugger that might give us some clues.

Then we noticed that there was a pattern. The only machines that showed this crash had Pentium processors of 150MHz or less, and some of the 150MHz machines did not show the problem. We had a hunch, and searched the Intel web site for "errata" (their word for bugs - we prefer "issues"). Sure enough, there was a flaw in the Pentium chip which under certain obscure circumstances could cause a fault - there needed to be a JMP instruction followed exactly 33 bytes later by a branch, and the JMP instruction had to be aligned with a "page" (4096 byte block) boundary in memory. Talk about specific. The flaw had been fixed in later versions of the 150MHz generation, and all chips produced later.

Now, there are actually flaws in chips quite often (one or two are even famous). Those chips include software after all. But as they are discovered, chip manufacturers tell the people who write compilers about the bugs, and the compiler people modify their compilers to produce code that cannot hit the bug. So the flaws seem to just disappear (thank goodness for software!) It turned out we were using a slightly older version of the compiler which did not know about this flaw. An Intel rep at Microsoft confirmed the details of the problem, and rather than relinking or taking any chances whatsoever, we ran a check for this byte sequence, and manually moved three bytes in our 5MB executable to make sure those instructions were 34 bytes apart. Problem solved. So now when someone tells me a fix is "safe", I can tell them that no fix is truly safe - you really can never know.

Watson

Chris_Pratley — Wed, 04 Feb 2004 09:54:00 GMT

People who don’t build software for a living have a quite understandable attitude that you should write your program, fix all the bugs, then ship it. Why would you ever ship a product that has bugs in it? Surely that is a sign that you don’t care about quality, right?

Those of us who work on non-trivial software projects naturally see this a little differently. There are many factors that determine quality of software, and some of them are counter-intuitive. For example, the first thing to realize is that you are not trying to get rid of every bug. What you are trying to do is maximize the known quality of the product at the time you ship it. If the last bug you have in the product is that a toolbar button sometimes doesn’t display correctly, and you fix it, then you ship the product, how embarrassed will you be a little later when you find out that the code fix you took to get the toolbar to display causes your application to crash when used on machines with certain video cards? Was that the right decision? Fix every bug?

The discipline of software testing is one that is rarely taught in university, and when it is taught, it is taught in a highly theoretical way, not in the way that it needs to be done to successfully produce high quality products on a schedule demanded by commercial software.

I often interview university students applying for jobs at Microsoft. They're usually excellent people and very talented, although we have to turn away most of them. One thing that always intrigues me is when I am interviewing people who want to be a developer or tester. If I ask them to write a little code, most of them can. Then I ask them to prove to me that their code works. The responses I get to this vary so widely it is remarkable. Usually people run the example values I gave them through their algorithm and if it comes out correctly, they say it works. Then I tell them, "OK, let's say I told you this code is for use in a firewall product that is going to be shipped to millions of customers, and if any hackers get through, you get fired". Then they start to realize that they need to do a little more. So they try a few more numbers and some even try some special numbers called "boundary conditions" - the numbers on either side of a limit. They try a few more ideas. At some point they either say they're now sure, or they get stuck. So I then ask them if they think it is perfect now. Well, how do you know?

This is a central problem in large software projects. Once you get beyond about 100 lines of code, it becomes essentially impossible to prove that code is absolutely correct.

Another interesting detail here is that people constantly confuse the fact that they can’t find a problem with there being no more problems. Let's say you have a test team of one person, and a development team of 10 (at Microsoft, we have a 1:1 ratio of testers to devs, so we'd have 10 testers on that team). But what if our poor lone hypothetical tester goes on vacation for 2 weeks, and while he is away the developers fix the few bugs he has found. Is the program now bug free, just because there are no known bugs in it? It is "bug free" after all, by the definition many people would use. In fact a good friend of mine who runs his own one-man software business told me once his software is in fact bug free, since whenever any customer tells him about a bug, he fixes it. I suppose his reality is subjective. But you see the point - whether you know about the bugs or not, they're in there.

So how do you deal with them? Of course, you test test test. You use automated tests, unit tests that test code modules independently of others, integration tests to verify code works when modules are put together, genetic test algorithms that try to evolve tests that cause problems, code reviews, automated code checkers that look for known bad coding practices, etc. But all you can really say is that after awhile you are finding it hard to find bugs (or at least serious ones), which must mean you're getting it close to being acceptable to ship. Right?

One of my favorite developments in product quality that is sweeping Microsoft and now other companies as well is an effort started by a couple of colleagues of mine in Office, called simply "Watson". The idea is simple. A few hundred (or even thousands) of professional testers at Microsoft, even aided by their expert knowledge, automated tools and crack development and program management teams cannot hope to cover the stunningly diverse set of environments and activities that our actual customers have. And we hear a lot about people who have this or that weird behavior or crash on their machine that "happens all the time". Believe me, we don’t see this on our own machines. Or rather, we do at first, then we fix everything we see. In fact, the product always seems disturbingly rock solid when we ship - otherwise we wouldn’t ship it. But would every one of the 400 million Office users out there agree? Everybody has an anecdote about problems, but what are anecdotes worth? What is the true scale of the problem? Is everything random, or are there real problems shared by many people? Watson to the rescue.

As an aside, I suspect that some people will read this and say that open source magic solves this, since as the liturgy goes so many eyes are looking at the code that all the bugs are found and fixed. What you need to realize is that there are very few lawyers who can fix code bugs. And hardly any artists. Not so many high school teachers, and maybe just a handful of administrative assistants. Cathedral, bazaar, whatever. The fact is that the real user base is not part of the development loop in any significant way. With 10^8 diverse users, a base of people reporting bugs on the order of 10^4 or even 10^5, especially with a developer user profile cannot come close to discovering the issues the set of non-computer people have.

The Watson approach was simply to say: let's measure it. We'll match that 10^8 number 1 for 1. In fact, we'll go beyond measuring it, we'll categorize every crash our users have, and with their permission, collect details of the crash environment and upload those to our servers. You've probably seen that dialog that comes up asking you to report the details of your crash to Microsoft. When you report the crash, if that is a crash that someone else has already had, we increment the count on that "bucket". After a while, we'll start to get a "crash curve" histogram. On the left will be the bucket with the most "hits". On the far right will be a long list of "buckets" so rare that only one person in all those millions had that particular crash and cared to report it. This curve will then give you a "top N" for crashes. You can literally count what percentage of people would be happier if we fixed just the top 10 crashes.

When we started doing Watson, it was near the end of OfficeXP. We collected data internally for a few months, and started collecting externally for the last beta. We quickly learned that the crash curve was remarkably skewed. We saw a single bug that accounted for 27% of the crashes in one application. We also saw that many people install extra software on their machine that interferes with Office or other applications, and causes them to crash. We also saw that things we thought were fairly innocuous such as grammar checkers we licensed from vendors for some European languages were causing an unbelievable number of crashes in markets like Italy or the Netherlands. So we fixed what we could, and as a result OfficeXp was pretty stable at launch. For the first time, that was not a "gut feel" - we could measure the actual stability in the wild. For follow-on service packs for Office XP, we just attacked that crash curve and removed all the common or even mildly common crashes.

For Office2003 (and OneNote 2003), the Watson team kicked into high gear with more sophisticated debugging tools so we could more often turn those crash reports into fixes. They also started collecting more types of data (such as "hangs", when an application either goes into an infinite loop, or is taking so long to do something that it might as well be in one). We fixed so many crashing bugs and hangs that real people reported during the betas that we were way down "in the weeds", in the areas where crashes were reported only a handful of times - hard to say if that is real or just a tester trying to repro a bug they found. So again we know how stable the 2003 generation of Office applications is - we can measure it directly. What a difference from the guesswork that used to go on. In fact we know for a fact that Office 2003 at launch is already more stable than any previous release of Office ever, even after all service packs are applied, since Windows now collects Watson data for all applications.

Microsoft offers this data freely for anyone writing applications on Windows. You can find out if your shareware application is crashing or its process is being killed by your users often - sure signs of bugs you didn’t know you had. The Windows team will also give you crash dumps you can load into your debugger to see exactly what the call stack was that took you down, so you can easily fix your bugs.

This is an example of what BillG calls "Trustworthy Computing". You may think that's a bunch of hoopla, but in many ways it is quite real - things like Watson are used to directly improve the quality of our products.