VC++ Compiler Front-End Testing

VC++ Compiler Front-End Testing

  • Comments 3

Hi, my name is Liu Xiong. I am the test lead for the Microsoft Visual C++ Front End QA team. Our passion is shipping the greatest C++ compiler to our customers. We are a very customer oriented team. The big challenge for us is protecting our C++ users from compiler breaking changes, while we also need ship the compiler with COOL new features and high conformance. People might be interested about how we test our compiler Front End. First we have very technical software engineers for directly testing the compiler. We understand every feature area; we create all types of evil test cases to break it; we focus on all platforms and configuration environments. We also build with real world code to detect new regressions, for instance we build NT for every live build. In our new product cycle, the C++ compiler front end team has a great opportunity to start a force multiplier project to build a new generation C++ compiler. Our test team also has a new opportunity to test a modern compiler with new testing methodologies. For instance, we might be able to directly write AST (Abstract Syntax Tree) checkers in the compiler source to verify the object model. We might be able to use new compiler extensions to add testing hook functions to verify the compiler in the API base. I am very excited by our new opportunities to ship a highly advanced compiler to our customers. I would be happy to discuss any compiler testing ideas and techniques.

 

Liu Xiong

Visual C++ Compiler

  • Hello, Something in your blog caught my eye - you mentined using AST for testing the front end. Are there any plans to make access to the AST programmatically accessible?
  • Hi Aleksandar,

    There has been some thinking of how best to expose ASTs programmatically. In the first phase of this project, we don't plan to provide a public API to access the AST. Though, we do see value in it and a later project could.
  • Well, probably the most important thing in a compiler testing is people who are involved in the process. Here in Unicals Group I wrote C89/C99 and EC++ front ends and I use VC++ (among other development systems) as a build platform as well as for sarching good test cases for our front ends. What I would conclude from this activity is that the VC++ compiler (the compiler itself, I mean) is very good and is very bad. No doubt you know what makes your compiler good, so let me explain only the latter point.

    Everybody knows that VC++ preprocessor is fairly fast. And a lot of developers know how hard that happens to make something more complex than trivial substitutions. The macro expansion process seems to be very strange, often letting on double scanning, etc. In fact, that would be pretty interesting to hear something from you regarding to whether you're really worrying about conformness and/or compatibility with other development systems. It's okay if not, but thus we would know that you wouldn't be interested in such kind of testing.

    Another matter everybody knows about VC++ is (often) terrible diagnostics. Just one example: while checking for assignment compatibility, the compiler first checking for type qualifiers matching and then for type kinds themselves. Another example is that the compiler declares unknown names with type int istead of using some special erroneous kind of type. Everybody see that that causes a lot of problems with determining what actually a diagnostics emitted mean.

    So my final conclusion is what I started with: perhaps, you don't have to look for more testing techniques. I know Bill pays a lot of attention to process automation, integration, etc. at Microsoft, but look at your public defect/feature base -- it is practically empty. Obviously people don't want to tell you much about how you would improve the compiler.

    Instead, you would have for people who can keep in their minds and direct the whole testing process. Of course you know that details are eveything in testing so you would need to have someone who could renew your compiler requirements and force to deliver the new level of quality to your users. Could you personally do something in this direction? :)

    Thanks!
Page 1 of 1 (3 items)