Testing VC++ Compiler and Intellisense

Testing VC++ Compiler and Intellisense

  • Comments 25

Hello,

My name is Rashid Sarwar and I am on Visual C++ Compiler Front End team. I have been on this team for several months and have learned many things on how to test a complier. In this post I will be sharing with you how we test our Visual C++ compiler and it’s Intellisense.

Testing features in a compiler is not easy as each feature/construct can interact with the already existing features/constructs in many different ways. Coming up with all the possible cases in which the new construct can be used is a big task as the Visual C++ compiler has many features, starting from standard C++ language features to our own Microsoft extensions to compiler. Every time when we have to add a new feature to our compiler like the new C++0x features coming in Visual Studio 2010, we try to come up with the good number of possible cases in which the feature can be used as a standalone feature .We first try to exhaust the new grammar of the newly added language constructs and try to make sure that we are able to compile the newly added language constructs as a standalone construct.

For example:

Lambda expressions in new C++0x can be used in many different ways. A lambda can have an empty capture clause.

A simple compiler test to test empty capture clause would be something like this in which we will come up a source code related to empty lambda capture clause. The automated test will take these sources and compile them with Visual C++ compiler. If let's say any compilation on a valid usage of lambdas on empty capture clause fails than it is an indication that something is there which either compiler currently not supports or is broken.

 

//Source1: testing empty lambda capture clause

int main()

{

 bool even = [](int k){ return k%2 == 0; }(2);

 return 0;

}

 

The above test verifies that we are able to compile correctly the empty capture clause of lambda expressions. But how can we verify that we are generating the correct compiled output? How can we verify that code generated by compiler is correct? We take a further step in the test by looking at the output of the lambda function object. We do run time verification by changing the test in following manner:

//Source2 - testing empty lambda capture clause

int main()

{

 bool even = [](int k){ return k%2 == 0; }(2);

 if(even)

 {

   return 0;

 }

 else

  {

   return 1;

 }

}

Our testing automation framework will look for the return value of the main method. If the return value from main method is 0 than the test is assumed to pass and assumed that we are generating correct code from compilation. If the return value is non zero than the testing framework assumes that the test failed as the result of main method can only be non zero if let’s say we generated wrong machine code.

This is very simple source code to test a big feature like lambda expressions which can be used in many different ways. So we then try to test each feature from the perspective of how it can be used in combination with other different features. Example can be like testing lambda expressions in a class member functions.

Now, I will discuss testing Intellisense for Visual C++. For Intellisense testing we divide our automated tests into two categories. Engine/Component level tests and End to End User scenario tests.

In the End to End User scenario test our automated test will try to simulate user actions in IDE example test can be like:

1.       Creating a new Visual Studio instance.

2.       Creating a new Visual C++ project.

3.       Adding a new cpp file in the project and set the cursor focus in the new file.

4.       Writing some C++ code in the opened file in the editor.  Simple code can be like:

class person{

public :

char * name();

};

 

int main()

{

person p;

return 0;

}

5.       Trying to invoke some intellisense operations like QuickInfo on the code in the file by hovering the mouse cursor over the newly created instance of class person in the main method.

6.       Finally verifying the output of tooltip with the expected output the test expects.

7.       If tooltip output differs from the test expected output than QuickInfo operation is assumed to not working and thus making test fail. The test passes if both expected and actual outputs by hovering mouse cursor over the construct match.

 

The Engine\Component level tests directly talk to Visual Studio to invoke the intellisense operations on the code constructs. We let the test know that at which positions in the code it has to invoke the intellisense positions.

For example on sample code:

class person{

public :

char * name();

};

 

int main()

{

person p;

return 0;

}

Simple test scenario can be like testing QI functionality on variables of user defined types. We will let the test know to invoke QI operation on the position where class person p instance is declared in main function. We will define in the test the (row/col) position and the test when it runs it will look for the specified QI operation on the specified position. It will get the result of tooltip that appears and match it with the expected output.

Our Engine level tests are easy to write and they have less execution time, as the test itself does not need to do all the extra work. But they don’t exactly mimic user scenarios. We write IDE tests to get better coverage of user end to end scenarios .The drawback of IDE tests is that they take a long time to execute. So we try to strike a balance in engine level and end to end scenario testing when we have to test a new or existing feature in our Visual C++ intellisense.

Hopefully this post will give you an idea of how we test our Visual C++ compiler and also its intellisense.

 

 

 

 

 

  • Thanks a lot to everbody for putting down some interesting questions and also some interesting piece of code to look at.I will try to answer all of them.

    Jk & yuguang : Intellisense performance is really a good question.We have a lot of test specifically for Intellisense perfromance testing. In our performance tests we try to simulate user actions as i described earlier in the post.We try to perform some operation that is to be measured for some defined no of times continuously in an iteration and get the average amount of time for that operation. That is the most simple explanation. The peformance test try to take into account that they are measuring the time of only the opeartion that is to be measured.Some tests measure time in isolation of all other scenarios and some measure in which a normal user would have invoked the operation.

    For testing memory leaks in VS we normally test our products both retail and debug builds .If there are any memory leaks they appear up in form of assertions in case of running our tests on debug build.We aks

    Joel : Thanks for pointing out Boost.We do provide intellisense for Boost library and we have tests that use Boost. We have done intellisense testing against Boost for our Intellisense parser and I hope our customers will have good experience with Intellisense in Boost library.Thanks a lot for your interest in Boost.

    Majkara : Nice idea oftesting the Boost Library.Thanks for it.

    Asif: Thanks for putting some interesting stuff here.I will try to explain the behaviour of the second question u had.

    Why do i get seperate results from the constructor and 'func' (without /clr, I get same values)

    When the code is compiled as native, the compiler does not creates a copy instance first. It constructed an A directly as a parameter to ‘func’ and this way just one instance gets created so you see both addresses same in case of native.  

    When compiled with /clr, it first creates a temporary A instance (at one address), invoked the trivial copy constructor and then copy it as an argument to func .This way func prints the address of this second object.The compiler can choose to construct an A instance directly as an argument to func, or it can construct a temporary, then copy it as an argument.

    I am looking forward to some more interesting questions.Do let me know if you guys have any regarding the Intellisense and our VC++ compiler.

  • http://social.msdn.microsoft.com/Forums/en-US/vclanguage/thread/3ba4afd7-9109-4426-8dc9-8b16374cc5b5  

    go through this discussion.

  • Hi, I stuck because of this bug in the VC++ compiler (or so it seems). This code is simpilfied for the sake of this discussion, and I cannot implement copy constructor etc on classes/structs that do not belong to me...

  • What happened with the Hotfix for the "VC9 SP1 Hotfix For The vector<function<FT>> Crash" Hotfix?

    http://blogs.msdn.com/vcblog/archive/2008/12/17/vc9-sp1-hotfix-for-the-vector-function-ft-crash.aspx

  • Hi Anders,

    We are currently working on the Hotfix for the "VC9 SP1 Hotfix For The vector<function<FT>> Crash" and it should be posted shortly. We will also update the original blog with the new link.

  • Can vs add a menu or button which is "Rebuild intelisense" so that I can manually full rebuild it? Even it needs 600 seconds would be helpful.

    Another question:

    Why when I open multiple VC IDE(3 or more), and use for a long time(more than 1 day),the CTRL+F show the dialog very slow? If I close all VC IDE and reopen, everything seem OK.

    I really doubt VS has GDI leak or handle leak.

  • This is a bug I found (and reported) in VC 6.0 and is still in 2003.  I was hoping it would be gone im my next generation of compiler>

    #include <ctype.h>

    #include <math.h>

    #include <stdlib.h>

    #include <stdio.h>

    #include <string.h>

    #include <stddef.h>

    static double dmx1,dmx2;

    #define NRC_DMAX(a,b) (dmx1=(a),dmx2=(b),((dmx1)>(dmx2))?(dmx1):(dmx2))

    /* --------------------------------------------------------------------------*/

    int main( void );

    /* --------------------------------------------------------------------------*/

    /* --------------------------------------------------------------------------*/

    int main( void )

    /* --------------------------------------------------------------------------*/

    {

     double test1;

     double test2;

     double test3;

     test1 = NRC_DMAX(7.,NRC_DMAX(4.,3.));

     test2 = NRC_DMAX(NRC_DMAX(4.,3.),7.);

     test3 = NRC_DMAX(NRC_DMAX(4.,7.),3.);

     printf("\n NRC_DMAX(7.,NRC_DMAX(4.,3.)) = %le \n", test1 );

     printf("\n NRC_DMAX(NRC_DMAX(4.,3.),7.) = %le \n", test2 );

     printf("\n NRC_DMAX(NRC_DMAX(4.,7.),3.) = %le \n", test3 );

     return 0;

    }

    Test1 , -2 ,-3 should be the same but are not in the versions I have tested.

  • As someone said. Please give users some control over intellisense.

    eg some settings like

    "Run Intellisense in background", "Don't run Intellisense in background, just do it when I tell you - even if it takes 5mins", "Kill Intellisense, its eating my memory"

    Intellisense certainly shouldn't be run during a compile!

    We've managed to mitigate some of the probs by not letting it write to disk.

  • I am curious about code style Listing 2, where there is so much verbosity! I hope this is only seen here and is not in compiler code :)

    if (even)

    return 0;

    else

    return 1;

    == return !even;

  • Unfortunately this intellisense bug https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=339575 still happens in VS 2008 and is a show stopper for us. :(  I hate it that we can't reopen bugs in connect

Page 2 of 2 (25 items) 12