|
|
-
[Ron]
Moving on from my previous post, I’d like to start talking about how we approach testing the C++ compiler. I’ll outline the three main approaches we use and in the next three posts I’ll drill down into each one in detail. A disclaimer: the compiler is tested by three teams inside the VC testing group; the front-end team, the C runtime (CRT) team (a lot of the functionality that the compiler relies on is implemented in the CRT), and the back-end team. Since I am a member of the back-end team, that perspective will be prevalent here.
Correctness Tests
The first way that we test the compiler is with what we call “correctness tests.” These are tests that exercise the compiler by giving it code to build and, in most cases, exercise the generated code and verify expected behavior. Cases where we would not build would include diagnostic tests. These are tests that give the compiler code that should not compile and verify that the correct error message is generated.
Our correctness tests are arranged into a number of suites. Some are implemented by us and some are implemented by 3rd parties. An example of test suites implemented by 3rd parties that we use would be some of the commercial 3rd party C++ conformance suites. Our correctness tests are implemented using a test harness written in Perl. The harness is pretty minimal but very flexible. It has served us well for many years now. For comparison, the IDE team is on their third test framework since I joined the team. The compiler team is still using the same framework they were using when I joined the team. There is a benefit to simplicity.
Real-World Code
The second way we test the compiler is with what we call “real-world code.” What that means is that we use our tools to build large code bases. This is in contrast to our correctness tests which are usually very small, focused tests. For this form of testing we build some of the larger code bases in the company. Real world code building has two big benefits: 1) we exercise our tools in large, real-world scenarios that are just not possible with our correctness tests, and 2) we demonstrate to ourselves and our internal customers that we can handle their code bases with our new tools. This helps convince those customers to adopt our newer toolsets which is a benefit to both them and us.
Benchmarks
The third approach that we use to test the compiler is benchmarks. These are scenarios that are used to measure the performance of generated code.
In my next post, I’ll drill down into the details of our correctness tests. In the mean time, feel free to comment on this post and ask any questions you may have related to testing the C++ compiler. I’ll do my best to answer them. Thanks for reading!
|
-
[Ron]
Inspired by Gus’ recent post on testing the C# compiler, I thought I’d talk a little bit about how we test the C++ compiler. Before I do that, though, I should talk about the organization of the VC test team to put things in context. I’ll do that in this post and focus on the compiler in future posts.
The VC test team (VCQA) is organized into three sub teams: IDE/Box, Programming Model, and Code Generation.
The IDE/Box QA team is responsible for the parts of VC that live in the IDE such as the VC project system, C++ intellisense, and the C++ wizards and designers. The “Box” part of this group is responsible for the QA parts of release management and other things such as setup, samples, and international aspects of the product. In general, box areas are areas of the product that don’t have developers on the VC team. Most of us FiveTestersFromVC are on the IDE/Box team.
The Programming Model QA team is responsible for the Visual C++ libraries (CRT, MFC, ATL) and the compiler front end. Andy is on this team.
The Code Generation QA team is responsible for the compiler back-end, the 64-bit .NET JIT compilers, and some of the tools (primarily the Linker). The Code Generation QA team is subdivided into 4 smaller teams: x86 back-end, IA64 back-end, AMD64 back-end, and 64-bit JIT compilers (the x86 JIT compiler lives on the CLR team). I am the test lead for the x86 back-end team.
Next time I’ll start focusing on how we test the C++ compiler.
|
-
[Ron] I updated our VC bloggers links to point to their new http://blogs.msdn.com homes. (I didn't see a new address for Martyn. Did he stop blogging?) I added a link to Andy Rich's weblog. Andy's a tester working on the compiler front-end team. He's actually turned out to be the most prolific of the VC++ bloggers.
I also noticed that a couple of debugger developers have started weblogs: Andy Pennell and Steve Steiner. I'm really looking forward to their posts. Their links have been added to our growing list.
|
-
[rob]I've just stepped out of my office to go to another meeting, and I hear my phone ring...the double ring of an external call. I look down at the floor through the open doorway.
“I am not 'in the office'....”
I continue to the meeting that I am already late for.
So, I've been pretty quiet on the blog....it's been busy over here!
I have been attending a local users group for .NET developers. I had originally attended thinking it was generic.NET, but have since learned that most attendees are either VB or C# devs. I may very well be the only VC developer regularly sitting in on these meetings. It speaks to the benefit of .NET that it hasn't made much of a differnence (except when I can't help but point out that MFC is not VC.NET to some of the other developers who have no VC.NET experience).
I have found out that my perception is a bit skewed by my exposure to Whidbey. I was debating this week that VC.NET was just as fast as other languages. It turns out that I was not quite correct. In Everett (7.1), .NET from VC is, sadly, slower for some scenarios. For Whidbey, it is actually slightly faster! (but JIT compiling will be the near equalizer for all). I suppose if you average all these statements I was...correct? (or can this kind averaging only be done for election predictions?)
|
-
-
[Barton]
Yes, we are alive. We were all out of office for the holidays and now we are back and ready to start blogging. We apologize for being such poor bloggers, and plan to better this year (a new years resolution?).
Today we met and were discussing what we should be talking about in our blog. Some of the topics we considered were
- managing large projects
- finding bugs
- talking about VC features
- getting feedback on VC
- testing methodologies
- robotics - teaching kids programming
- what have I learned working at MS for the past 8 years (today is my 8 yr anniversary)
- testing in an agile development world
- test first development
- bugs to avoid in the product
- prioritizing and making tradeoffs when fixing bugs
Let us know if you are interested in any of these topics or any other topics that you would like to hear about.
We are investigating the question of sending bugs to MS. We found a process that exists but it looks more like a black hole than a mechanism to get feedback to us. As soon as we find the right solution we will let you know.
|
-
Hi, My name is Barton and I’m the Test manager for Visual C++. The VC Test Organization includes teams focused on the User Interface, Programming Model (C++ Front End and Libraries), and the Compiler Back End (code generators for x86, ia64, and amd64 architectures). I’ve worked at MS for the past eight years and have always worked on the VC Test Team. I started testing the SQL Debugger and moved up to lead the User Interface Test Team and then over to lead the Compiler Test Teams and am now responsible for all VC Test Teams.
Thanks, Barton…
|
-
[Ron] Matthew in his comment to Rob's introduction asks:
I'm curious. I'm sure most people on a daily basis find problems with the IDE (or improvements). But for mere mortal developers it's not transparent as to how to report these (or if they have been reported in the first place). Any suggestions on improving this process?
I think the standard answer to this question is the same as it's been for a while: call product support and report the bug. But I've also seen many complaints about this particular method of reporting bugs since, from what I understand, you have to give all the information needed to open up an incident (including credit card info if you're not using an incident that came with some Microsoft product, like MSDN) and it's up to the support engineer to remove the charge.
I do know that we are pretty good about responding to posts on the microsoft.public.dotnet.languages.vc newsgroup but depending on where we are in the product cycle, YMMV. Sometimes we need to focus on the private beta or alpha newsgroups, for example.
Certainly, if you post a problem here in a comment to one of our posts, we'll make sure to investigate it and take the appropriate action.
(BTW, don't limit your complaints to the IDE only. If you have issues with other areas of VC++ (compiler, etc.) we'd like to hear about those as well.)
I'll bring this up with the powers that be and post back if I get any better information.
|
-
|
[Rui] I am Rui. I am on the C++ project team responsible for the project management and build. If you have any questions, suggestions or grievances, bring them on. Let's see how we can make your project building experience more enjoyable.
|
-
[Rob] I'm Rob. Being one of two team leads for the VC IDE let's me get my fingers into the VC Wizards, .NET designers for VC, VC debugging experience, and the VC Resource Editors (and generally everything else as well). I've been testing at Microsoft since 1999. Before working in IT, I was doing research in molecular biology (and offended many millions of Tetrahymena thermophilia, I'm sure).
Welcome to our weblog!
-rob
|
-
[Michael] Hello, I'm Michael T. I'm a Tester for the Visual C++ portions of the Browsing and Navigation experience. This translates into the information displayed in classview, the navigation bar, parts of object browser, goto defintion, and find symbol for people writing C++ code in our editor. I've been in this position for a little over a year now. Prior to joining the Visual C++ team, I worked in MSN on a project to provide consumer web services, and before that I worked on a project to provide bug tracking and source control. I put in a brief stint on Visual SourceSafe 6.0 when I first joined the company. I've been with Microsoft for over five years.
I'll be talking about issues others have brought to my attention, and interesting issues that I run into. If you don't get the title, please refer to Transformers the movie.
Nice to meet you guys!
|
-
-
Hello out there. Welcome to our new weblog. Notice I said “our”. We are five folks in the Microsoft Visual C++ testing organization and this blog is an experiment in group weblogging.
Our group started as a book study group. We would get together for an hour a week to discuss a common book that we were reading. We recently finished going through How to Break Software and were looking for our next book when we got the idea to try something different and this weblog is the result.
So, who are we?
Barton - Our esteemed QA manager
Michael - A tester on the IDE team.
Rob - A test lead on the IDE team.
Ron - A test lead on the Compiler Back-End/Tools team.
Rui - A tester on the Project System team.
How will we keep things straight since there are five of us?
Our initial idea is to preface each of our posts with our name like so:
[Ron] I'm Ron and I'm writing this initial post. I'm a test lead for the x86 Back-End/Tools team. My team is responsible for testing the x86 version of the VC++ Compiler Back-end (a.k.a. c2.dll) and other tools like the Linker, Editbin, and Dumpbin. I've been in this position for about a year now. Prior to joining the back-end team I worked for about 6 years on the IDE side of VC++ QA and prior to that I spent a couple of years in Developer Support.
Other members of our group will be posting a little bit about themselves in future posts.
So, what will we do on our weblog? We plan to talk about issues that get raised in our work testing Visual C++. We don't have anything more specific in mind than that. Feel free also to ask us questions in the comments for our posts (start with this one if you like). We'll do our best to answer them.
Let's see what happens...
|
|
|
|
