Testing VC++ Compiler and Intellisense

Testing VC++ Compiler and Intellisense

  • Comments 25

Hello,

My name is Rashid Sarwar and I am on Visual C++ Compiler Front End team. I have been on this team for several months and have learned many things on how to test a complier. In this post I will be sharing with you how we test our Visual C++ compiler and it’s Intellisense.

Testing features in a compiler is not easy as each feature/construct can interact with the already existing features/constructs in many different ways. Coming up with all the possible cases in which the new construct can be used is a big task as the Visual C++ compiler has many features, starting from standard C++ language features to our own Microsoft extensions to compiler. Every time when we have to add a new feature to our compiler like the new C++0x features coming in Visual Studio 2010, we try to come up with the good number of possible cases in which the feature can be used as a standalone feature .We first try to exhaust the new grammar of the newly added language constructs and try to make sure that we are able to compile the newly added language constructs as a standalone construct.

For example:

Lambda expressions in new C++0x can be used in many different ways. A lambda can have an empty capture clause.

A simple compiler test to test empty capture clause would be something like this in which we will come up a source code related to empty lambda capture clause. The automated test will take these sources and compile them with Visual C++ compiler. If let's say any compilation on a valid usage of lambdas on empty capture clause fails than it is an indication that something is there which either compiler currently not supports or is broken.

 

//Source1: testing empty lambda capture clause

int main()

{

 bool even = [](int k){ return k%2 == 0; }(2);

 return 0;

}

 

The above test verifies that we are able to compile correctly the empty capture clause of lambda expressions. But how can we verify that we are generating the correct compiled output? How can we verify that code generated by compiler is correct? We take a further step in the test by looking at the output of the lambda function object. We do run time verification by changing the test in following manner:

//Source2 - testing empty lambda capture clause

int main()

{

 bool even = [](int k){ return k%2 == 0; }(2);

 if(even)

 {

   return 0;

 }

 else

  {

   return 1;

 }

}

Our testing automation framework will look for the return value of the main method. If the return value from main method is 0 than the test is assumed to pass and assumed that we are generating correct code from compilation. If the return value is non zero than the testing framework assumes that the test failed as the result of main method can only be non zero if let’s say we generated wrong machine code.

This is very simple source code to test a big feature like lambda expressions which can be used in many different ways. So we then try to test each feature from the perspective of how it can be used in combination with other different features. Example can be like testing lambda expressions in a class member functions.

Now, I will discuss testing Intellisense for Visual C++. For Intellisense testing we divide our automated tests into two categories. Engine/Component level tests and End to End User scenario tests.

In the End to End User scenario test our automated test will try to simulate user actions in IDE example test can be like:

1.       Creating a new Visual Studio instance.

2.       Creating a new Visual C++ project.

3.       Adding a new cpp file in the project and set the cursor focus in the new file.

4.       Writing some C++ code in the opened file in the editor.  Simple code can be like:

class person{

public :

char * name();

};

 

int main()

{

person p;

return 0;

}

5.       Trying to invoke some intellisense operations like QuickInfo on the code in the file by hovering the mouse cursor over the newly created instance of class person in the main method.

6.       Finally verifying the output of tooltip with the expected output the test expects.

7.       If tooltip output differs from the test expected output than QuickInfo operation is assumed to not working and thus making test fail. The test passes if both expected and actual outputs by hovering mouse cursor over the construct match.

 

The Engine\Component level tests directly talk to Visual Studio to invoke the intellisense operations on the code constructs. We let the test know that at which positions in the code it has to invoke the intellisense positions.

For example on sample code:

class person{

public :

char * name();

};

 

int main()

{

person p;

return 0;

}

Simple test scenario can be like testing QI functionality on variables of user defined types. We will let the test know to invoke QI operation on the position where class person p instance is declared in main function. We will define in the test the (row/col) position and the test when it runs it will look for the specified QI operation on the specified position. It will get the result of tooltip that appears and match it with the expected output.

Our Engine level tests are easy to write and they have less execution time, as the test itself does not need to do all the extra work. But they don’t exactly mimic user scenarios. We write IDE tests to get better coverage of user end to end scenarios .The drawback of IDE tests is that they take a long time to execute. So we try to strike a balance in engine level and end to end scenario testing when we have to test a new or existing feature in our Visual C++ intellisense.

Hopefully this post will give you an idea of how we test our Visual C++ compiler and also its intellisense.

 

 

 

 

 

  • how do you test intellisense performance?

  • Hi, Could you guys please explain the Rich header generated between DOS & PE header?

  • yeah, Intellisense performance

    And, resource leak in vs...

  • Hi, this appears to be a bug (VS2005 SP1), it prints garbage.

    class String

    {

    public:

    char string[8192];  //keep array less than 4096 for correct results

    String(char *buffer="")

    {

    strcpy(string,buffer);

    }

    };

    void func(String str1,_bstr_t str2="")

    {

    printf(str1.string);

    }

    int _tmain(int argc, _TCHAR* argv[])

    {

    func("asd","dfgdfs");

    return 0;

    }

  • You should be glad that your code even compiles. Run cl at /W4 and see what I mean.

    You are converting const char* to char* through that user defined conversion which is a non-standard extension. No wonder it prints garbage.

    Secondly, you should never omit a format string in one of (print/scan)f functions.

    Third, use cout and cin.

    Fourth, redefine func as void func(const String& str1,_bstr_t str2="").

    Fifth, add cctor, dtor and assignment operator to that class.

  • //compile with /CLR

    class A

    {

    public:

    A(int *i)

    {

    printf("this: %x\n",this);

    }

    };

    void func(A a)

    {

    printf("func: %x \n",&a);

    }

    int _tmain(int argc, _TCHAR* argv[])

    {

    func(0);

    return 0;

    }

    why do I get different results printed out from my constructor and from 'func', (compile using clr, without clr, it is correct).

  • //compile with /CLR

    class A

    {

    public:

    A(int *i)

    {

    printf("this: %x\n",this);

    }

    };

    void func(A a)

    {

    printf("func: %x \n",&a);

    }

    int _tmain(int argc, _TCHAR* argv[])

    {

    func(0);

    return 0;

    }

    Why do i get seperate results from the constructor and 'func' (without /clr, I get same values)

  • //compile with /CLR

    class A

    {

    public:

    A(int *i)

    {

    printf("this: %x\n",this);

    }

    };

    void func(A a)

    {

    printf("func: %x \n",&a);

    }

    int _tmain(int argc, _TCHAR* argv[])

    {

    func(0);

    return 0;

    }

    Why do i get seperate results from the constructor and 'func' (without /clr, I get same values)

  • //Correction

    //compile with /CLR

    class A

    {

    public:

    A(int i)

    {

    printf("this: %x\n",this);

    }

    };

    void func(A a)

    {

    printf("func: %x \n",&a);

    }

    int _tmain(int argc, _TCHAR* argv[])

    {

    func(0);

    return 0;

    }

  • The code works fine with my install of VS 2005 SP1. Several of Badar's comments are silly:

    <i>Secondly, you should never omit a format string in one of (print/scan)f functions.</i>

    What does that mean? The string is the format string.

    <i>Third, use cout and cin.</i>

    Bullocks. cout and cin are bloated functions that don't have the control and simplicity printf does.

    <i>Fifth, add cctor, dtor and assignment operator to that class.</i>

    What are you some sort of academic? If you don't need that crap, why add it? In this case, the compiler does it for you and that's perfectly fine.

    Yes, Asif should use const correctly and guard against buffer overruns, but I figured he was simplifying the code to show a compiler error. (He should also be asking these questions elsewhere and needs to brush up on what the /CLR switch does.)

  • Asif,

    I compiled and executed your first code in VS2005SP1 pro(v50727.762) in both release and debug config and using both unicode and multibyte for each of the configs. The result was same for all configurations: "asd" that I believe is correct result.

  • Asif,

    Your second example is interesting. When I compile it without /clr I get same result for constructor and 'func' but with /clr I get different results too. Seems some magic happens there.

    Anybody any explanation?

  • The only way i'm going to be impressed by Intellisense is when it works properly with Boost.

  • > You are converting const char* to char* through that user defined conversion which is a non-standard extension.

    It is a perfectly standard conversion, just a deprecated one. ISO/IEC 14882:2003, 4.2[conv.array]/2:

    "A string literal (2.13.4) that is not a wide string literal can be converted to an rvalue of type “pointer to char”; a wide string literal can be converted to an rvalue of type “pointer to wchar_t”. In either case, the result is a pointer to the first element of the array. This conversion is considered only when there is an explicit appropriate pointer target type, and not when there is a general need to convert from an lvalue to an rvalue. [Note: this conversion is deprecated. See Annex D. ]"

  • vote+1 on boost.

    Save yourself the trouble load your test engine up and parse the boost (you have the code already). If it cannot fetch the names and types, you have a bug.

Page 1 of 2 (25 items) 12