Visual C++ Team Blog

Hey good on you. You're a bit behind in using the "before you fix, write a test that exposes the bug" idea but better late than never!

Its good to know that the quality of this excellent compiler will increase, not decrease.

For your ANSI vs. Unicode tests, don't you need to run the sniff tests on around 10 machines overnight, instead of just 1 machine?

Even in my tiny employer, although we can't do 10 machines simultaneously, for some projects we did around 3 to 5 simultaneously and reinstalled Windows or reinstalled firmware as needed to test all language versions.

Your regression testing definitely sounds like an awesome productivity boost.

I was a bit confused by one thing, though: What is sniff testing?

jmm: That fix (for a "store forwarding stall" in the back-end, internal bug number DDB.12838) is in VC9 (as far back as Beta 1). It was never applied to VC8, and is not present in VC8 SP1.

Generally, fixes for Connect bugs go into the next full release. Service Packs consist of accumulated hotfixes, and hotfixes are made in response to support requests from corporate customers.

This regression is nasty enough that I wish the fix had been applied to VC8 SP1, but it sounds like it might have happened just too late for that to be accepted.

Thanks,

Stephan T. Lavavej

Visual C++ Libraries Developer

"Generally, fixes for Connect bugs go into the next full release. Service Packs consist of accumulated hotfixes, and hotfixes are made in response to support requests from corporate customers."

Please tell me you are kidding. Are you actually saying that none of the bugs that we report on Connect will actually get fixed in the product that we buy unless a corporate customer pushes hard enough to get a QFE? I hope you realize this sounds REALLY bad.

I agree with you guys.  I too am very surprised that Connect bugs are only fixed in the next VS version.  Call me old fashioned, but, paying customers should not be forced to buy a new version to get bugs fixed.  

This sentiment must get really old but in the vs 6 days, fixing bugs appeared to be a major priority.

Hey!

I am new to this blog and I haven't used VS in awhile (I am in a web phase). I have been reading these last few posts and I see alot of "tests" and "scripts". Can you give me an example of script or one of the tests that you use to make sure there are no bugs in VS.

Thanks and keep up the good work :)

The 9000 tests are only for the libraries owned by the VC team, not the compiler.

Pat,

I'd be interested in what kind of testing system you use. How do you collect and analyze the results of the nightly runs? What information does the system provide in case of a failing test - does it save a log, a minidump or something?

Thanks, Jo

[AntonioCS]

> Can you give me an example of script or one of the

> tests that you use to make sure there are no bugs in VS.

Sure, here's an example.

The Standard specifies that 9 iterators must derive from std::iterator. In VC8 SP1, only 6 did. This was further broken in early builds of VC9, where 0 did. In VC9 RTM, all 9 will derive from std::iterator. (More importantly, this fix makes these iterators smaller and faster.)

We wrote a regression test to prevent this from ever breaking again. It looks like this:

C:\Temp>type iterator.cpp

#include <cstddef>

#include <ios>

#include <iostream>

#include <iterator>

#include <memory>

#include <ostream>

#include <streambuf>

#include <string>

#include <vector>

using namespace std;

template <typename B, typename D> void test(const string& d) {

   test_helper<__is_base_of(B, D)>(d);

}

template <bool B> void test_helper(const string& d) {

   cout << "std::iterator is a base of " << d << "." << endl;

}

template <> void test_helper<false>(const string& d) {

   cout << "std::iterator is NOT a base of " << d << "." << endl;

}

int main() {

   typedef vector<int> vec_t;

   typedef iterator<output_iterator_tag, void, void, void, void> outit_t;

   typedef iterator<input_iterator_tag, int, ptrdiff_t, const int *, const int&> istrit_t;

   typedef iterator<input_iterator_tag, char, streamoff, char *, char&> istrbufit_t;

   typedef iterator<random_access_iterator_tag, int, ptrdiff_t, int *, int&> revit_t;

   test<    outit_t,  back_insert_iterator<vec_t>      >( "back_insert_iterator");

   test<    outit_t, front_insert_iterator<vec_t>      >("front_insert_iterator");

   test<    outit_t,       insert_iterator<vec_t>      >(      "insert_iterator");

   test<   istrit_t,      istream_iterator<int>        >(     "istream_iterator");

   test<    outit_t,      ostream_iterator<int>        >(     "ostream_iterator");

   test<    outit_t,  raw_storage_iterator<int *, int> >( "raw_storage_iterator");

   test<istrbufit_t,   istreambuf_iterator<char>       >(  "istreambuf_iterator");

   test<    outit_t,   ostreambuf_iterator<char>       >(  "ostreambuf_iterator");

   test<    revit_t,      reverse_iterator<int *>      >(     "reverse_iterator");

}

C:\Temp>cl /EHsc /nologo /W4 /MT iterator.cpp

iterator.cpp

With VC8 SP1:

C:\Temp>iterator

std::iterator is a base of back_insert_iterator.

std::iterator is a base of front_insert_iterator.

std::iterator is a base of insert_iterator.

std::iterator is a base of istream_iterator.

std::iterator is a base of ostream_iterator.

std::iterator is a base of raw_storage_iterator.

std::iterator is NOT a base of istreambuf_iterator.

std::iterator is NOT a base of ostreambuf_iterator.

std::iterator is NOT a base of reverse_iterator.

With VC9 20917:

C:\Temp>iterator

std::iterator is a base of back_insert_iterator.

std::iterator is a base of front_insert_iterator.

std::iterator is a base of insert_iterator.

std::iterator is a base of istream_iterator.

std::iterator is a base of ostream_iterator.

std::iterator is a base of raw_storage_iterator.

std::iterator is a base of istreambuf_iterator.

std::iterator is a base of ostreambuf_iterator.

std::iterator is a base of reverse_iterator.

(If you have VC9 Beta 2, that will say "NOT a base" for all 9 iterators.)

This is a slightly modified version (for human consumption) of the actual regression test we use. The actual test uses exit codes to report success or failure to our test harness, and upon failure, it prints the name of the offending iterator to be captured in our logs.

[Anonymous]

> Are you running larger C++ test suites besides your

> own tests?

As Andre said, Pat was describing only our developer-written library tests (which we call "regression tests", as we check them in alongside bugfixes) and our tester-written library tests (which we call "sniff tests").  We additionally have third-party library conformance tests from several companies.  And then the compiler (front-end, back-end) and IDE teams also have their own tests.

[Phaeron], [Andrew McDonald], [jmm]

That was my understanding of the Connect and hotfix processes.  I've been with VC for only 9 months, so my understanding is limited.  I'll talk to our PMs and see if I'm missing something.

Please understand this, though: Fixing bugs in the next full release is easier than creating hotfixes. To fix something in the next full release, we just need to write the fix, verify that it works (do builds, run tests, etc.), and check it in. There's plenty of time (and CTPs, and Betas, etc.) to be sure that we aren't horking anything. Also, we aren't restricted by binary compatibility, since the CRT gets totally renamed between each full release, and we don't support things like passing STL objects between DLLs compiled with different full releases.

The hotfix process is much more heavyweight (as it should be - you don't want us breaking stuff in hotfixes).  We must be careful to keep binary compatibility, and hotfixes get less testing than the months and years of testing that full releases endure.  Shipping a hotfix is also a big deal, since we have to package it and verify that the packaging works (from my perspective, this is an ordeal).

Service packs get additional testing, but the fixes in them generally start life as hotfixes.  (There can be exceptions to this, but that's the usual way it works.)

I've never personally observed hotfixes being produced in response to Connect bugs, nor have I seen Connect fixes backported to VC8. As I said, I'll check and see if we ever actually do that.

[Bob]

Your business can certainly request hotfixes (through the proper support channels, about which I know precisely nothing, sitting at the other end of them). My comments were for Microsoft Connect bugs submitted by individuals.

We ship hotfixes both for the latest service pack, and the previous service pack.  (Currently, that would be VC8 SP1 and VC8 RTM.)

Stephan T. Lavavej, Visual C++ Libraries Developer

Re including fixes in Service Packs, and Bob's request that we include fixes in SPs:

Actually, for VS2005 SP1 (before Stephan's time), we put in significant effort to fix almost every single customer reported bug up to a certain date (excluding those bugs which would break binary compatibility or which were erroneously reported).  I think we hit over 90% fix rate on these bugs.  We had a cutoff date of course to stabilize and test the service pack and if a bug was fixed but didn't make it, it was probably because it was fixed after that date.  I think the libraries team alone fixed roughly 500 bugs (don't hold me to that, I'm going by memory here), many of which were very minor, but it was nice to see things cleaned up.

Re: the anonymous posting about the small number of tests.  That 9000 tests was just the "sniff" tests and VC9 regression tests for the libraries.  To give you a rough comparison, the CRT sniff tests take about an hour to run while the entire set of CRT suites takes about two days to run on modern hardware.  (We have lots of tests).  As Marina said, our sniff tests are meant to be quick running while still hitting a lot of areas.  Doing a full test pass against VC takes a couple weeks and hundreds of machines.  

Regarding whether we run something like Plumhall - we actually run a number of licensed suites like Plumhall (not sure whether I'm allowed to say which ones).  Somewhere between 5-10 C++ licensed or conformance suites.  This is actually a very small part of our testing though.  We have a lot more internal tests to cover things in more depth.  

In fact, at this point, the sheer number of tests is one of our biggest challenges.  Because of the numbers of tests, machines and configurations we run under, both flakey tests and simply random hardware or network failures introduce a lot of noise into our results which take manual effort to isolate.  It's an especially troubling scenario since it can be very time consuming to determine the root cause of an isolated and non-reproducible failure when you need to determine whether it was due to an intermittent product issue, an intermittent test issue, or maybe just random EMI flipping a bit in memory (since we run probably millions of tests over the course of a product cycle on a lot of hardware, this is actually probably a more likely occurence than it would seem).

In other words, don't worry that we don't have too many tests :).  I don't, although it is legitimate to wonder whether we have all the right tests ;).  Unfortunately, we will probably never be complete, and we will probably always have redundancies.  I think the challenge in the coming years will be to remove redundancies while still maintaining adequate testing levels to avoid missing things.

> We're just building the tests using both the

> ANSI and Unicode versions of the libraries

> (ATL and MFC).  We're not running the tests

> on machines with different code pages set.

Up to a point, I understand this.  If you have no character strings and maybe no comments in your source code, then it doesn't matter what code page your development machine is running under when it's building.

When Visual Studio 2005 gives a warning about one of Microsoft's own files in an SDK, it might be because of characters in comments.  When the DDK's compiler gave errors about one of Microsoft's own files in the DDK, it was because of characters in a string.  Notice that this paragraph is abusing the word "character", since the reason for diagnostics in the first place was that certain entities didn't constitute valid characters where characters were needed.

When you have character strings and comments, I still think you need to repeat the builds under multiple code pages.

Just add a Visual Basic and Visual C++ Learning guide for free of cost. It'd help the weak programmers to expend their IT Knowledge.