[Ron]
Moving on from my previous post, I’d like to start talking about how we approach testing the C++ compiler. I’ll outline the three main approaches we use and in the next three posts I’ll drill down into each one in detail. A disclaimer: the compiler is tested by three teams inside the VC testing group; the front-end team, the C runtime (CRT) team (a lot of the functionality that the compiler relies on is implemented in the CRT), and the back-end team. Since I am a member of the back-end team, that perspective will be prevalent here.
Correctness Tests
The first way that we test the compiler is with what we call “correctness tests.” These are tests that exercise the compiler by giving it code to build and, in most cases, exercise the generated code and verify expected behavior. Cases where we would not build would include diagnostic tests. These are tests that give the compiler code that should not compile and verify that the correct error message is generated.
Our correctness tests are arranged into a number of suites. Some are implemented by us and some are implemented by 3rd parties. An example of test suites implemented by 3rd parties that we use would be some of the commercial 3rd party C++ conformance suites. Our correctness tests are implemented using a test harness written in Perl. The harness is pretty minimal but very flexible. It has served us well for many years now. For comparison, the IDE team is on their third test framework since I joined the team. The compiler team is still using the same framework they were using when I joined the team. There is a benefit to simplicity.
Real-World Code
The second way we test the compiler is with what we call “real-world code.” What that means is that we use our tools to build large code bases. This is in contrast to our correctness tests which are usually very small, focused tests. For this form of testing we build some of the larger code bases in the company. Real world code building has two big benefits: 1) we exercise our tools in large, real-world scenarios that are just not possible with our correctness tests, and 2) we demonstrate to ourselves and our internal customers that we can handle their code bases with our new tools. This helps convince those customers to adopt our newer toolsets which is a benefit to both them and us.
Benchmarks
The third approach that we use to test the compiler is benchmarks. These are scenarios that are used to measure the performance of generated code.
In my next post, I’ll drill down into the details of our correctness tests. In the mean time, feel free to comment on this post and ask any questions you may have related to testing the C++ compiler. I’ll do my best to answer them. Thanks for reading!
