Hi, my name is Jianhua Li, and I am a VC++ IDE QA. Today I am going to talk about VC++ IDE / Design Time stress testing.
What is stress testing?
Design Time Stress (DTS) is part of a broader set of reliability quality criteria focused on measuring an application's robustness, availability and reliability under stressful conditions. It's scoped to Design Time Components which are components exercised by the user when working in the IDE, for example, browsing or intellisense.
Why do we need to “stress” our product?
Basically, we want to make sure our product, VC++ IDE, works reasonably under critical conditions, which is far worse than real world coding. By pushing our product to its limits, we can also find many product bugs that can’t be found by “normal” test cases. For example a stress test can expose many product crash/hang problems due to various racing/timing issues; subtle memory leaks can also be detected since it will be accumulated and enlarged during the extensive long time testing. Third, stress testing provides a sufficient simulation of real world programmer’s continuous coding experience. Because of the nature of stress testing, a several hour run of stress testing can be adequately equivalent to several days of real world’s testing, or several weeks’ programmer coding.
How to we perform stress testing?
When we implement our test infrastructure work for stress testing, we considered the following questions:
· What scenarios should we stress?
Inside the Visual Studio product division, different teams have invested on different stress scenarios, for VC++, our favorite is IntelliSense! We believe Intellisense is one of the key design time features of VC++ for our customers. Just think of how often you will use AutoComplete, MemberList, ParameterHelp, and QuickInfo on your daily coding work.
· What are the requirements/goals for stress testing?
Since this is stress testing, the basic requirement will be “continuous running over a long period without any IDE performance degradation and memory leaks”. In practice, this goal will also guarantee that for a real world programmer, after several weeks coding, the intellisense experience will be as smooth as when coding began. There shouldn’t be any delay for the new intellisense operations.
· What types of stress testing should we test?
VC++ design time stress contains 2 different test types: Stress Store Reads, and Stress Store Writes.
For “Stress Store Reads” test type, as its name implies, we just read data from the intellisense “store” by performing intellisense operations without editing the code.
For “Stress Store Writes” test type, we will force writes to the intellisense store by automated editing of the code, e.g., deleting and re-typing in every line on a file. After the intellisense store is updated, we will “read” the updated intellisense store to make sure it produces the expected results.
Based on the above design requirements, we implemented a fully component level infrastructure for VC ++ IDE stress testing. This leverages our existing component level framework, and uses DTE and other Microsoft internal component level libraries.
Below is the high level general overview of VC++ IDE stress testing.
1. Stress Test Initialization: Initialize the run settings which includes:
· stress test solution: the solution on which the stress testing will be performed
· stress type: Stress Store Reads or Stress Store Writes
· stress time: how long the stress testing will last
· stress iteration: how many iterations the stress testing will perform.
2. Open the stress test solution file, get the full list of header and cpp files in all projects in the test solution file, and close the solution.
3. Start the stress test monitor process, which will monitor the memory usage during stress run.
4. Open the test solution file again, do the following loop:
· Open each header/cpp file in the file list obtained from step 2
§ If (stress type == Stress Store Write) do code editing, e.g., deleting and re-typing each line in current opened file. We keep deleting and re-typing every line in current file, without pause during this process, until we are done with the last line in this file. Then we move cursor back to the beginning of current opened file.
§ Get the position of every token in the current file
§ Iterate over every token. For each one, invoke AutoComplete/MemberList/ParameterHelp/QuickInfo as appropriate for that token
§ Collect the intellisense results on a log file.
· If specified stress time is expired or test iteration is reached, quit the loop in this step; otherwise, go to next header/cpp file in the file list, repeat the same steps above.
5. Analyze the stress log file, get the overall intellisense operation pass rate, average, minimum, and maximum runtime for each intellisense operation. Make sure there is no performance issue found. Also, analyze the stress monitor log file, and make sure there are no memory leaks. All these kinds of results reporting and analyzing are done automatically.
· Since our testing is implemented at component level, the file opening, code editing, and intellisense invocations are very fast. Normally in 1 second, our stress testing can request about 10 or even more intellisense operations, and we will keep this pace/speed during the whole stress test run period, which is about 8 hours. No real world programmers can do this, I bet. ^_^.
· For stress run result analysis, we don’t verify the correctness for each intellisense operation, the thing we care about in stress testing is the performance over long period run. So the criteria for stress testing are no IDE crash/hang, no abnormal big runtime for any intellisense operation during 8 hours run, and good overall pass rate, e.g., above 80%.
Some interesting data for Orcas SP1 stress testing.
Now let’s see some interesting data for Orcas SP1 stress testing results.
For Stress Store Reads type testing, on Orcas SP1, we have:
§ IDE continues to run for 8 hours without crash/hang.
§ Over 220k intellisense operations are requested during an 8 hour run, and successful intellisense results are returned for over 98% of those intellisense requests. What does this sound like? Let us make a simulation to real world programming.
1. Assume that during daily coding, an average C++ programmer will request 1 intellisense operation every 5 seconds.
2. that's 12 per minute
3. 720 per hour
4. at 4 hours of actual coding per day we get 2880 per day
5. and 14,400 per week (5 day)
§ Based on above assumption, our stress testing is simulating a real world average programmer’s coding of 76 days / 15 weeks! Now let us look at our performance, for 76 days continues coding, the average runtime per intellisense operation is only about 150 ms, and the peak memory for devenv.exe is only 150 MB.
Please give us your feedback on IDE stress testing.
In Visual Studio 2010, we invested in stress testing even more and extended it to include browsing scenarios, for example GoToDefinition and FindAllReferences. Please let us know of any scenarios that you think should be included in our stress testing. Any comment or feedback is welcome!
PingBack from http://asp-net-hosting.simplynetdev.com/vc-ide-design-time-stress-testing/
You need to have complex C++ test cases such as templates used in functions and classes.
Also you should review all the submited bug reports that dealt with Intellisense crashes and failures.
The crucial variable is not how long it runs against a static sample of C++, but how well it handles all the C++ possibilities seen in real world programming.
You can test against a small subset of C++, and produce tests that don't catch the errors in the real world.
Andrew, thanks for your comments.
Actually all projects in our stress testing are from real world. They do contain complex C++ language features. But you are right, there is no way we can cover all possibilities, and we just want to cover as much as possible.
Recently I had to use Boost Graph Library with VS2008 and Intellisense simply did not work at all. Please make sure your tests reflect real-world usage of C++.
Here's a part of the IDE that's progressively gotten worse ever since VC6 – support for custom color/font schemes. The first thing I do after installing VS is to change the colors to bright text on dark background and set the font to Courier (not Courier New).
In VS2002/2003 some debugger windows ignored the custom colors.
In VS2005/2008 that was fixed but the Disassembly window started ignoring the color settings. It starts with the default colors but if I enter/exit the color settings dialog (without changing anything) the Disassembly window fixes itself. I reported that to Connect but the bug was closed as failed to reproduce. However I’m seeing this on multiple computers with different versions of Windows and different versions of Visual Studio with and without service packs.
In VS2010 CTP it was much, much worse. Random color settings would reset themselves, and changing the font would cause VS to crash when opening a text file. It was basically unusable. I'm waiting to try a real beta before I submit any bugs.
So please, please, take some time to iron these things out. I’m sure the majority of the users are happy with the default settings but there are still many people that like customizing their IDE – colors, fonts, keyboard shortcuts, window layout, etc. These are features that should work reliably.
In VS2008 C# you can use the BACK button on Mouse to perform some IDE action. If however C++ IDE settings were selected after install, the back button does nothing!
Can you make it so the user can decide what the mouse back/forward button does in all the languages.
"Over 220k intellisense operations are requested during an 8 hour run, and successful intellisense results are returned for over 98% of those intellisense requests. What does this sound like?"
What does it sound like? Honestly, it sounds like you need better test cases. If Intellisense is able to return results in VS2k8 in 98% of your test cases, your test cases do not reflect anything even approaching reality. In my experience it'd be more accurate to say that it does *not* work in 98% of all cases.
I can live with that, I've pretty much gotten used to working without Intellisense, but if your test suite is *that* much out of touch with the real world, that's worrying.
I would strongly suggest chucking your test cases since they are fooling you into thinking VS 2005, VS 2008 and, no doubt, VS 2010 is more stable than it is. This is a very common mistake. When testing interfaces, nothing replaces people actually using them--preferably sadistic people who want to prove that the software engineers are complete idiots (I've worked with such testers and they are invaluable.)
(As an FYI, all versions of Visual Studio will occasionally just vanish, cease to be, be no longer in the task list. How about fixing that?)
Hi Jianhua, the new Intellisense (which iirc is SQL-based) does sound interesting, and I'm glad you're testing the heck out of it.
But, if it turns out that for some scenarios Intellisense does more harm than good (i.e. longer build times, UI unresponsiveness) will there be an easy way for folks to turn it off?
As you may know, the way folks currently turn Intellisense off is by renaming MS dll's...
DaveG: yes, there will be a way to turn off intellisense via configuration settings.
Hi Pingpong, can you share more details about how the intellisense doesn't work for Boost graphs library with VS2008? I would like to raise this issue to our team. Thanks.
It should be noted that Stress testing represents just one arm of our overall intellisense testing strategy - we have targeted correctness and targeted performance tests, but the point of these stress tests is to measure if (and how) the experience degrades over a long period of usage.
Stress was very valuable (from our perspective) in detecting and reducing the number of Intellisense crashes and hangs in VS2008 - it took us a long time to get that number up to 98%, because it definitely did not start there.
Here's a real-world scenario I'd like to see but can't seem to make work in the October CTP.
I think this is Intellisense related, but may not strictly be a part of the Intellisense system.
Create a template MFC Appwizard generated app.
In stdafx.h, add:
In Mainfrm.cpp::CMainFrame() add:
OutputDebugString(_T("DEBUG_ENABLED is defined.\r\n"));
OutputDebugString(_T("_DEBUG is defined.\r\n"));
I'd like to see the OutputDebugString statements dim when the Release configuration is enabled.
Then, darken when the Debug configuration is selected.
Would be great if this worked without needing to recompile or anything like that.
With CTP, I can't get #ifdef'd code that is compiled out to grey at all.
Jianhua Li: The Intellisense fails with 'Can't find the identifier' message in the statusbar whenever I'm trying to get parameter information on calls to BGL's graph layout algorithms. I have 'using namespace boost' at the file scope, preceeding the code causing the Intellisense to fail. Similarly, I can't get autocompletion when I type 'circle_', for example. I'd expect 'circle_graph_layout' to be autocompleted.
Platform is VS2008 SP1 with Boost 1.38.0.
My latest in a series of the weekly, or more often, summary of interesting links I come across related to Visual Studio. Greg Duncan posted a link to the Managed Stack Explorer on CodePlex . Brad Abrams announced that the .NET RIA Services May 2009 Preview