Testing the C++ Compiler – a Three-Pronged Approach

FiveTestersFromVc — Tue, 10 Feb 2004 08:53:00 GMT

[Ron]

Moving on from my previous post, I’d like to start talking about how we approach testing the C++ compiler. I’ll outline the three main approaches we use and in the next three posts I’ll drill down into each one in detail. A disclaimer: the compiler is tested by three teams inside the VC testing group; the front-end team, the C runtime (CRT) team (a lot of the functionality that the compiler relies on is implemented in the CRT), and the back-end team. Since I am a member of the back-end team, that perspective will be prevalent here.

Correctness Tests

The first way that we test the compiler is with what we call “correctness tests.” These are tests that exercise the compiler by giving it code to build and, in most cases, exercise the generated code and verify expected behavior. Cases where we would not build would include diagnostic tests. These are tests that give the compiler code that should not compile and verify that the correct error message is generated.

Our correctness tests are arranged into a number of suites. Some are implemented by us and some are implemented by 3^rd parties. An example of test suites implemented by 3^rd parties that we use would be some of the commercial 3^rd party C++ conformance suites. Our correctness tests are implemented using a test harness written in Perl. The harness is pretty minimal but very flexible. It has served us well for many years now. For comparison, the IDE team is on their third test framework since I joined the team. The compiler team is still using the same framework they were using when I joined the team. There is a benefit to simplicity.

Real-World Code

The second way we test the compiler is with what we call “real-world code.” What that means is that we use our tools to build large code bases. This is in contrast to our correctness tests which are usually very small, focused tests. For this form of testing we build some of the larger code bases in the company. Real world code building has two big benefits: 1) we exercise our tools in large, real-world scenarios that are just not possible with our correctness tests, and 2) we demonstrate to ourselves and our internal customers that we can handle their code bases with our new tools. This helps convince those customers to adopt our newer toolsets which is a benefit to both them and us.

Benchmarks

The third approach that we use to test the compiler is benchmarks. These are scenarios that are used to measure the performance of generated code.

In my next post, I’ll drill down into the details of our correctness tests. In the mean time, feel free to comment on this post and ask any questions you may have related to testing the C++ compiler. I’ll do my best to answer them. Thanks for reading!

Testing the C++ Compiler - Introduction

FiveTestersFromVc — Sat, 31 Jan 2004 08:41:00 GMT

[Ron]

Inspired by Gus’ recent post on testing the C# compiler, I thought I’d talk a little bit about how we test the C++ compiler. Before I do that, though, I should talk about the organization of the VC test team to put things in context. I’ll do that in this post and focus on the compiler in future posts.

The VC test team (VCQA) is organized into three sub teams: IDE/Box, Programming Model, and Code Generation.

The IDE/Box QA team is responsible for the parts of VC that live in the IDE such as the VC project system, C++ intellisense, and the C++ wizards and designers. The “Box” part of this group is responsible for the QA parts of release management and other things such as setup, samples, and international aspects of the product. In general, box areas are areas of the product that don’t have developers on the VC team. Most of us FiveTestersFromVC are on the IDE/Box team.

The Programming Model QA team is responsible for the Visual C++ libraries (CRT, MFC, ATL) and the compiler front end. Andy is on this team.

The Code Generation QA team is responsible for the compiler back-end, the 64-bit .NET JIT compilers, and some of the tools (primarily the Linker). The Code Generation QA team is subdivided into 4 smaller teams: x86 back-end, IA64 back-end, AMD64 back-end, and 64-bit JIT compilers (the x86 JIT compiler lives on the CLR team). I am the test lead for the x86 back-end team.

Next time I’ll start focusing on how we test the C++ compiler.

Five Testers From VC : C++ Compiler Testing

Testing the C++ Compiler – a Three-Pronged Approach

Testing the C++ Compiler - Introduction