Hi, my name is Pat Brenner and I’m a software design engineer on the Visual C++ libraries team. I’d like to spend some time talking about our process for preventing regressions in our libraries code.
After I joined the libraries team about a year ago, I was told about the set of sniff tests that we needed to run before checking in new features or bug fixes. As it turned out, I worked on some of the new MFC code for Visual C++ 2008, and I didn’t need to run these tests for the first five or six months that I was on the team. When it came time for me to start running the tests, because I was fixing bugs for Visual C++ 2008, I found that some of the tests did not pass for me, and I was not getting consistent results. I (and the rest of the development team) got by with this up until Beta1 or thereabouts, because our QA team performs scheduled runs of all their tests, which included the tests included in the sniffs, and we were able to rely on their results. But then I found that, in addition to the sniff tests, we also had a large set of regression tests, written by the development team, which were largely being ignored.
So I decided that I wanted to get take some time early this summer and accomplish this goal: the sniff tests and the regression tests needed to run well enough that I could schedule my development machine to run a build and the sniff and regression tests overnight, and have results in the morning.
I first needed to get the sniff tests running and passing consistently on my machine. So I worked with QA to stabilize the broken tests, remove the tests that were obsolete, and clear out any other blocking problems so that the sniff tests would run start to end, and pass 100%. This took several weeks (as a background task) and we found several bugs in tests that needed to be fixed (mostly platform issues, where a test would pass on Windows XP but not on Windows Vista, for instance).
Then, with help from a member of our QA team, I got our regression tests ported over to a QA test harness similar to the one which runs our sniff tests. This meant that our regression tests now got run in many more configurations (static library vs. DLL, ANSI vs. Unicode, Debug vs. Release) than they had been before. This flushed out a number of test bugs, as well as several actual libraries code bugs, all of which we have fixed for Visual C++ 2008.
Our policy now is that for almost every bug we fix, we write a regression test which fails without the fix and succeeds with the fix. This helps us to keep our regression rate down when fixing bugs, especially those ones where a particular method has been changed a couple of different times. Using the test harness, it’s very easy to add a regression test.
The set of sniff tests (owned by our QA team) consists of 6542 tests, of which over 6000 run on the C++ runtimes. We have about 75 sniff tests on ATL and about 400 for MFC. The set of development regression tests (owned by the development team) consists of another 2536 tests, of which about 1500 test the C++ runtimes and 1000 test ATL and MFC. I have all of these 9000+ tests running on my development machine on a nightly basis, which is really convenient when it comes time to fix a bug. I can make the fix, check that it does indeed fix the bug, and then let my overnight process build it and fully test it. When I arrive in the morning, I can check my test results, and if all passed 100%, I know my fix has not caused any problems. I’ve also written up scripts that the rest of my team can use, so they can have the same productivity increase that I’ve been enjoying with the automation of these tests.
Thanks, and I welcome any questions you might have.
Visual C++ Libraries Team
Hey good on you. You're a bit behind in using the "before you fix, write a test that exposes the bug" idea but better late than never!
Its good to know that the quality of this excellent compiler will increase, not decrease.
Sohail: We of course have bug repros prior to fixing the bugs, Pat is more talking about harnessing an automated test so that it runs automatically and supports running on our many different supported configurations without failure. Surprisingly, it often takes a lot more work to get a good test working, automated and operating under every condition than it does to simply reproduce the problem reported in the bug.
Visual C++ Libraries QA Team
For your ANSI vs. Unicode tests, don't you need to run the sniff tests on around 10 machines overnight, instead of just 1 machine?
Even in my tiny employer, although we can't do 10 machines simultaneously, for some projects we did around 3 to 5 simultaneously and reinstalled Windows or reinstalled firmware as needed to test all language versions.
"Our policy now is that for almost every bug we fix, we write a regression test which fails without the fix and succeeds with the fix."
Absolutely. If you aren't doing this, you aren't doing regression testing, pure and simple.
Good post; thanks!
Your regression testing definitely sounds like an awesome productivity boost.
I was a bit confused by one thing, though: What is sniff testing?
Did this regression fix ever make it into VS 2005 SP1? I see this fix was made, but I don't see anywhere indicating if this fix was for VS 2005 SP1 or VS 2008.
jmm: That fix (for a "store forwarding stall" in the back-end, internal bug number DDB.12838) is in VC9 (as far back as Beta 1). It was never applied to VC8, and is not present in VC8 SP1.
Generally, fixes for Connect bugs go into the next full release. Service Packs consist of accumulated hotfixes, and hotfixes are made in response to support requests from corporate customers.
This regression is nasty enough that I wish the fix had been applied to VC8 SP1, but it sounds like it might have happened just too late for that to be accepted.
Stephan T. Lavavej
Visual C++ Libraries Developer
To answer Alex's question "What is sniff testing?": Sniff tests are a small sub-collection of available tests that are meant to execute quickly and cover the most crucial functionality/code paths in the product or feature being tested. As Pat described, high confidence that serious regressions are not introduced when checking in bug fixes or introducing new features into an existing product is achieved if all sniff tests pass before a code change is checked in.
Visual C++ Front End Compiler Test Team
"Generally, fixes for Connect bugs go into the next full release. Service Packs consist of accumulated hotfixes, and hotfixes are made in response to support requests from corporate customers."
Please tell me you are kidding. Are you actually saying that none of the bugs that we report on Connect will actually get fixed in the product that we buy unless a corporate customer pushes hard enough to get a QFE? I hope you realize this sounds REALLY bad.
Agreed - I'm disappointed with that approach. If I report a bug in a product, and some of these are productivity hits, I expect it to be fixed in a patch without having to push through some corporate channel where I've no idea how successful it's going to be.
Unfortunately the service pack record on VS has been really poor since VS6...
I agree with you guys. I too am very surprised that Connect bugs are only fixed in the next VS version. Call me old fashioned, but, paying customers should not be forced to buy a new version to get bugs fixed.
This sentiment must get really old but in the vs 6 days, fixing bugs appeared to be a major priority.
I am new to this blog and I haven't used VS in awhile (I am in a web phase). I have been reading these last few posts and I see alot of "tests" and "scripts". Can you give me an example of script or one of the tests that you use to make sure there are no bugs in VS.
Thanks and keep up the good work :)
Are you running larger C++ test suites besides your own tests? 9000 tests doesn't sound that much for a compiler that is in its 14th revision ... just wondering whether the compiler has to run through something like http://www.plumhall.com/suites.html before each release or whether you rely on in-house tests exclusivly.
The 9000 tests are only for the libraries owned by the VC team, not the compiler.