Testing the Visual C++ IDE

Testing the Visual C++ IDE

Rate This
  • Comments 4

Hello folks!

 

Alvin Chardon, VC++ IDE QA member here.  I write to our readers today, not with a specific topic in mind, but with a few different thoughts collected during the Orcas product cycle. Hopefully, by the end of this narrative we can take that collection of ideas and form a coherent statement.  Don’t worry, I’m sure it will all fit together as nicely as native to managed transitions in COM Interop C++\CLI code targeting Excel…don’t ask.   So back in the real world, I would like to start by giving you a brief introduction to Alvin.  I am a Puerto Rican Software and Electrical Engineer (no, the diploma doesn’t say Puerto Rican), that has been in Microsoft for the past 5 years.  All this time my career has been inside the Visual C++ Team.   About a year and half ago I moved from the Visual C++ Compiler Team to the Visual C++ IDE Team.  And let me tell you, it has been quite a change.  C++ is a great language, and being a .NET fan myself, it was an incredible experience to be part of the Visual Studio 2005 release which delivered C++\CLI.  Testing a new language can be quite a challenge, but by definition, compiler testing is so much easier than UI (User Interface) testing.  I have great passion for user experiences, human/computer interactions, software usability, functionality and over all, software quality.  Thus I took on the challenge of testing the Visual C++ IDE for a change.  A functional IDE in which to write code is such an integral part of the development experience, that some even consider it as important as the compiler.  I know most hard core programmers out there prefer notepad-type editors, I know I myself sometimes do, but try consuming the Excel API with C++\CLI using notepad and the Visual Studio Command Prompt.  (“Again, with the Excel?”, you might ask).  I guess it is something that has been on my mind this week.  Why?  You’ll have to wait.  It will all make sense later, I promise.

 

The Visual C++ IDE is great, but we know it could be even better.   One of the toughest challenges facing the VC++ IDE QA Team is the testability of the product.  During Orcas, we have spent a lot time thinking about this problem.  A good software architecture will have a clear boundary between the code that belongs to the UI and the code that belongs to the engine.  Having said that, in Orcas we opted for designing an internal API model to expose the engine’s functionality through test hooks. This enables us to reach into the VC IDE at the component level.  Why component-based tests?  Because they are more robust, easier to write and more maintainable than UI tests.  Don’t get me wrong, I think UI tests are still an integral piece of the quality puzzle we call IDE Testing.  Optimally, you can use component-based tests to make sure your engine and data are correct and then use UI tests to verify the engine and data are correctly and accurately connected to the user interface.  Scenario testing can be done with either, or a combination of both.  Providing your QA team with an arsenal of testing frameworks that include both methodologies, is the key to enabling testability in your product.  Ideally, they can all coexist on the same testing environment and provide seamless UI and component-based testing.  For example, say you want to test that a glyph, that little picture that identifies the type of code construct and its accessibility, is correctly displayed in the Class View.  You could use UI to get the symbol name and glyph description, ask the engine for the code construct type of that symbol, and then validate the correct glyph was loaded and displayed.  In the Orcas timeframe, we managed to enable this sort of seamless test environment in the Visual C++ IDE, (or we will once I integrate that little code change into our test library).

 

One component for which we implemented one of the richest API libraries was IntelliSense.  Ah…I can feel readers waking up and rocking their seats at the sound of the word: IntelliSense.  Yes, IntelliSense, your friend and foe, lover, hater, friend and enemy.  You just can’t live without it when it works to your satisfaction and you just wish you could turn it off when it doesn’t.  One of the toughest challenges for IntelliSense testing, is not only correctness, but the sort of data flow and time bound issues that make our VC++ IDE hang or crash.  Well, in Orcas we recently finished the development of a spanking new IntelliSense stress test harness.  You can input any solution, project or what would have been a compiler command line and stress the life out of it.  We identified different types of tests, some for stressing reads from our store and some to stress writes into our store.  The harness is smart about typing code in the editor in different styles and executing every single possible IntelliSense operation on their allowed C++ constructs.  We measure the execution time, we look for parser Access Violations, failed operations, hangs, crashes, processor CPU % peaks, etc.  Since the technologies being used are targeted at component-level testing, the harness produces consistent repros (VC slang for steps to reproduce an issue), provides for a debugging environment where it doesn’t matter if the IDE has focus or not, even when typing, and still gives us real metrics because the operations get executed through the same code paths as when a user requests them.  Oh, and at the end, it uses Excel to plot all this information and .NET to send a report out to the team.  So you can imagine why my week has been about C++, .NET, COM Interop, and Excel and how stress filled it has been…I mean…stress tests filled.  Testing improvements like this stress harness are a great big step towards making the quality of our IDE even better.  It has been my personal experience as tester and user of the Orcas Visual C++ IDE that IntelliSense has gotten significantly better.  We have gained great momentum during the Orcas product  cycle which will keep building up after a great Orcas release towards an even better and mind blowing Orcas + 1 release.

 

I almost forgot to summarize the end result of our narrative. So what do you get when you add Alvin, C++, IDE, engine level testing, IntelliSense and stress?

 

Un día en la vida del Visual C++ IDE QA Team

 

Till next time.

Alvin

  • Hi Alvin,

    Great post and I hope it will make IDE more effective. We have recently moved to VS2005 from different build and compiler platform. Not to mention that Intellisense is causing a great deal of problem.

    I want to know whether having one solution with 160+ projects and thousands of file can cause problem?

    Due to unbuffered read and writes by Intellisense solution loading and closing times are increasing. Do we have any solution?

    (Current NCB size is around 70MB).

    Platform is sufficient with dual core, 4GB RAM etc.

    Any suggestions?

    thx

    Ketan

  • Just to follow on from Ketan, we have just moved a largish code base (1.3 million lines) with about 40 projects to visual studio 2005. It is much slower, it seems from filemon, due to 4 byte read/writes to the NCB file. Given Orcas is a long time way from full release are there any plans for incremental improvements to 2005 performance?

  • Gracias por tu esfuerzo, Alvin. A ver si conseguimos que VC++ sea un ciudadado de primera categoria en .Net.

    (Thanks for your effort, Alvin). Let's see if VC++ becomes a first class citizen in .Net!).

  • Hello Ketan and David!

    Very interesting timing on your concerns about the small group of byte writes problem in VS 2005.  We are currently working on a hotfix for it.  Last week I signed off on the Private Binaries testing stage.  It will probably take a little while longer to be released to customers.  It still needs to go through the following stages: Customer Sign Off on private binaries, Packaging and Final Testing.  On this latter step, it comes back to me to repeat my testing on the actual MSI we release.

    Using Process Monitor (procmon.exe) to filter only FileWrites from DevEnv.exe to the NCB of the Power Point sources (it is about 60M in size), the size of the log was reduced by 99.5%.  So we got a whole lot better at writing the NCB data in chunks.  Also, I have to say, I didn't notice significant improvements in solution close time.  There are some improvements, but they are mostly noticeable when you open a solution from a network share or from a USB flash drive.  In addition, this fix is already in Visual Studio Codename Orcas, so if you would like to try it out, download a copy of Beta2.

    If you would like to know more about the hotfix process or what testing I did, feel free to ask.  Stay tuned for the release of that hotfix...

    Thanks,

    Alvin Chardon

    Visual C++ IDE Team

Page 1 of 1 (4 items)