I Love that New Syntax Smell : A day in the life

A Bug's Life

arich — Mon, 14 Jun 2004 21:32:00 GMT

No, not the Pixar movie. This is about the lifetime of a bug in our product. How they go from discovery, to fix, to maintainance.

Stage 1: Discovery. Perhaps what's most interesting are the various ways a bug can be discovered. We often find them through test suites, where a build of the compiler is run on a test suite by means of automation. The front-end compiler test suites are mostly a mix of specific feature tests, regression tests (more on that later), and third-party conformance tests. (For a lot more on how we test, you might check out some posts from FiveTesters.) Sometimes, a bug is found by a customer - these usually come to us via PSS, through newsgroup postings, or as beta feedback. Generally, simply the fact that a bug was discovered by a customer affects its impact.

Stage 2: Determination. Is it a bug? Just because you see odd behavior doesn't always indicate a bug. Determining if something is a bug usually means digging through the ISO C++ or C standards, or through other standards documentation. (Except for the obvious bugs, like an ICE or a rare compiler crash.) I like to get the confirmation of at least one other person, too, when it isn't completely obvious.

Stage 3: Raiding. I've mentioned before why we call it “Raiding” a bug. When you file a bug in our bug system, a lot of other pertinent information is included. Usually, you include the source code that reproduces the bug, along with a note to the developer (or other people looking at the bug) that justifies the bug, and explains why it is a bug. We also try to use very descriptive titles, so that when other people are searching for bugs, they have a high probability of finding the right one. When bugs are raided, their status is “Active.“

You give the bug a Severity and Priority - usually these two are quite similar. They range from 1 (most severe) to 4 (typo in the diagnostic?). There's a zero priority, in Microsoft lingo, we call that “Pri Zero.” Pri 0 bugs have an expected turnaround time of 24 hours. Like most Microsoft-isms, I hear people using them in other contexts. My former officemate referred to his son as “Pri 0.“

Stage 4: Triage. Representatives from the dev, QA, and product management team meet about twice a week, and go through Triage, where they inspect all newly-raided bugs and give them a quick once-over. They try to verify that it is a bug, that it isn't already raided, and that the priority and severity are correct. They then assign it over to a developer to investigate and fix. (Occasionally, the bug is kicked back to the QA person, either because it is duplicate of another bug or because it isn't a bug at all.)

Stage 5: Fixing. The developer performs his or her magic, and fixes the bug, linking in the pertinent information, and a link to the source diffs that represent their fix. They then “resolve“ the bug. A bug can have a number of resolutions, some of the most common are Fixed (the developer fixed the bug), Not Repro (the developer couldn't reproduce the behavior), and By Design (that behavior is not a bug). The Resolved bug usually ends up assigned back to the QA who opened the bug.

Stage 6: Verification. The QA who opened the bug gets a build of the compiler containing the fix, and verifies that the bug is fixed properly. If they don't feel the bug is fixed, or don't agree with the resolution of the developer, they “push back.” This is another Microsoft-ism (perhaps I should create a glossary...) - but at least this one is pretty self evident. Usually, this just means they re-assign the bug back to the developer with justification. Sometimes, they just go down the hall, knock on the developer's door, and have a discussion.

The QA person is also responsible for ensuring that this scenario is covered by our test suites. If there's no test to cover it, they check one in, and mark the location of the test case in the bug. We typically call these “regress tests.“ They're just there to make sure the bug gets fixed, and, if it somehow gets un-fixed, we catch it. When this is complete, the QA closes the bug.

Stage 7: Maintainance. Closed bugs don't disappear from our bug database. If a bug ever regresses, we can usually find the bug in our database, and reactivate it.

So, there's my quick explanation of a bug's life cycle.

QA Mode

arich — Wed, 09 Jun 2004 20:54:00 GMT

This will be the first post in a new category, “A day in the life.“ Through these posts, I hope to provide some background into what QA for the compiler team is like. Hopefully, you won't think I'm a bungling idiot. So, to get the ball rolling, I thought I might try and give a bit of perspective as to how we sometimes stumble across bugs.

So, a few weeks back, I was fooling around with a personal project (the details of which are really boring), and I was looking for a C# decompiler. I wanted to see how a method was accomplishing something. I did a bit of searching, and came up with Lutz Roeder's .NET Reflector. Now, if you haven't seen this tool, you should really go download it. It's like ildasm on steroids.

I was fooling around with it, and I discovered that it had C#, VB.NET, and even Delphi decompilation, but no C++! I emailed the author to see if there were any plans to develop one. There weren't any, but he pointed me at the GotDotNet workspace where people developed the snap-ins. From there, I was a few quick samples away from starting development on a C++ decompiler for Reflector. Being the good QA person that I am, I wanted to use my product to develop my snap-in (as opposed to C#). I worked for a while, and had a few things populating correctly, when I ran into a bit of a brick wall. Basically, the C++ compiler was balking when I tried to instance a particular public type from the assembly. I immediately switched into "QA Mode" to try and diagnose the problem.

Step one: Repro! “Repro” is like a holy word to a tester. It's short for “reproduction,” or maybe “reproducible,” but were I adding it to The Oxford Standard, I would probably define it as the smallest code sample that will demonstrate a bug. It's useful for giving the developers a bee-line to the problem, but not so useful as a test case, simply because its impact is too low.

(digression) And lately, I'm discovering that depending on the dev you give your repro to, there's such a thing as too small. Sometimes, if the repro is too small, the dev only fixes part of the bug. This is part of the difficulty of being a tester. Knowing what's too much, what's too little. Perhaps I'll get to this in a later posting.

Anyhow, in this case, I was able to boil down to about two lines of code. I'll preface this sample by saying that I didn't bother to instantiate the types necessary to pass to the TypeInformation constructor, but that wasn't important, as the bug reproed without it.

#using "reflector.exe"

int main(){
Reflector::TypeInformation^ ti = gcnew Reflector::TypeInformation();
}

When I compiled this, I received a few warnings (which wouldn't have stopped compilation) and one error:

t.cpp(4) : error C2373: 'Reflector::TypeInformation::_1' : redefinition; different type modifiers
t.cpp(1) : see declaration of 'Reflector::TypeInformation::_1'

I know this is a bug for two reasons. One: the author of this assembly intended for me to instantiate this type. Two: C# is able to instantiate this type without problem. (For us C++ guys, the .NET litany is sometimes “Can C# do it? Yes? Then so must we.“)

Step two: Workarounds! Workarounds are important in determining the impact of a bug. If a bug can be worked around in a simple and straightforward manner, then it's impact is far lower than a bug which completely prevents the user from doing something important. In this case, there was no workaround. I wanted that type, and I couldn't have it.

Step three: Investigation! Often, this is part-and-parcel with finding a repro. Usually, I like to have another person to concur with my investigation. Sometimes, part of investigation is determining if something is a bug. (For me, that means I grab a handy copy of the ISO C++ standard.) In this case, what interested me were the _1's. That's a weird name for a type. If it was labelled foo, I wouldn't have blinked, but _1 doesn't seem like even lazy identifier naming. So, I loaded up ildasm, and poked around inside the assembly for a while:

Now, I began to get suspicious. Why would someone name their types like this? All underscores and numbers, no indication of anything. It's like they were almost trying to make this difficult... to... read. Like they were obfuscating. Is this what output from a code obfuscator looks like? I hopped over to the Dotfuscator website and started reading up. Turns out, they do something called Enhanced Overload Induction, where they obfuscate types by overloading on return type, when possible, to further obfuscate. In fact, it appears they do this intentionally to prevent people from decompiling their code, since C# and C++ don't support overloading by return type, most of the time. Bingo.

So, I contacted Lutz, and he got back to me within an hour's time (good turnaround, considering it a Saturday afternoon). He said that he was running Demeanor on the assembly.

Step Four: RAID it! This is one of those fascinating Microsoft-isms. The bug-tracking system employed used to be called RAID. It isn't any longer, but people still refer to filing bugs as RAIDing them. What's interesting is, RAID was before my time here, but I still refer to filing bugs as RAIDing them. Anyhow, I filed the bug into our bug database, and it began its trek around our group. (I'll probably post another article detailing the life of a bug.)

Step Five: Verify! When the bug makes its way back to me, I verify that the fix has been implemented correctly, and that I agree with how the developer made it work. In this case, what the developer did was turn an error into a level-2 warning. This still lets the user know something is amiss, and lets them know what fields in the type are going to be a problem, should they try to use them.

Step Six: Regression! This is perhaps the most important step - making sure this feature does not regress. We typically do this by ensuring we have appropriate test cases in our suite that cover the recently-broken scenario. That way, if the test starts failing, we know that this bug has regressed. Sometimes, the simple fact that a bug is a regression is enough to raise its priority.

So, I hope this has given you some insight into how us QA people sometimes go from discovering a bug, to its eventual fix.