VC++ IDE / Design Time Stress Testing

VC++ IDE / Design Time Stress Testing | ASP NET Hosting

VC++ IDE / Design Time Stress Testing | ASP NET Hosting — Mon, 11 May 2009 20:47:24 GMT

PingBack from http://asp-net-hosting.simplynetdev.com/vc-ide-design-time-stress-testing/

re: VC++ IDE / Design Time Stress Testing

Andrew Webb — Mon, 11 May 2009 22:47:18 GMT

You need to have complex C++ test cases such as templates used in functions and classes.

Also you should review all the submited bug reports that dealt with Intellisense crashes and failures.

The crucial variable is not how long it runs against a static sample of C++, but how well it handles all the C++ possibilities seen in real world programming.

You can test against a small subset of C++, and produce tests that don't catch the errors in the real world.

Good Luck.

re: VC++ IDE / Design Time Stress Testing

Jianhua Li — Mon, 11 May 2009 23:18:55 GMT

Andrew, thanks for your comments.

Actually all projects in our stress testing are from real world. They do contain complex C++ language features. But you are right, there is no way we can cover all possibilities, and we just want to cover as much as possible.

re: VC++ IDE / Design Time Stress Testing

pingpong — Mon, 11 May 2009 23:20:38 GMT

Recently I had to use Boost Graph Library with VS2008 and Intellisense simply did not work at all. Please make sure your tests reflect real-world usage of C++.

re: VC++ IDE / Design Time Stress Testing

Ivo — Tue, 12 May 2009 01:00:30 GMT

Here's a part of the IDE that's progressively gotten worse ever since VC6 – support for custom color/font schemes. The first thing I do after installing VS is to change the colors to bright text on dark background and set the font to Courier (not Courier New).

In VS2002/2003 some debugger windows ignored the custom colors.

In VS2005/2008 that was fixed but the Disassembly window started ignoring the color settings. It starts with the default colors but if I enter/exit the color settings dialog (without changing anything) the Disassembly window fixes itself. I reported that to Connect but the bug was closed as failed to reproduce. However I’m seeing this on multiple computers with different versions of Windows and different versions of Visual Studio with and without service packs.

In VS2010 CTP it was much, much worse. Random color settings would reset themselves, and changing the font would cause VS to crash when opening a text file. It was basically unusable. I'm waiting to try a real beta before I submit any bugs.

So please, please, take some time to iron these things out. I’m sure the majority of the users are happy with the default settings but there are still many people that like customizing their IDE – colors, fonts, keyboard shortcuts, window layout, etc. These are features that should work reliably.

Ivo

re: VC++ IDE / Design Time Stress Testing

drd — Tue, 12 May 2009 01:37:56 GMT

In VS2008 C# you can use the BACK button on Mouse to perform some IDE action. If however C++ IDE settings were selected after install, the back button does nothing!

Can you make it so the user can decide what the mouse back/forward button does in all the languages.

re: VC++ IDE / Design Time Stress Testing

Jalf — Tue, 12 May 2009 02:40:41 GMT

"Over 220k intellisense operations are requested during an 8 hour run, and successful intellisense results are returned for over 98% of those intellisense requests. What does this sound like?"

What does it sound like? Honestly, it sounds like you need better test cases. If Intellisense is able to return results in VS2k8 in 98% of your test cases, your test cases do not reflect anything even approaching reality. In my experience it'd be more accurate to say that it does *not* work in 98% of all cases.

I can live with that, I've pretty much gotten used to working without Intellisense, but if your test suite is *that* much out of touch with the real world, that's worrying.

re: VC++ IDE / Design Time Stress Testing

Joe — Tue, 12 May 2009 07:30:19 GMT

I would strongly suggest chucking your test cases since they are fooling you into thinking VS 2005, VS 2008 and, no doubt, VS 2010 is more stable than it is. This is a very common mistake. When testing interfaces, nothing replaces people actually using them--preferably sadistic people who want to prove that the software engineers are complete idiots (I've worked with such testers and they are invaluable.)

(As an FYI, all versions of Visual Studio will occasionally just vanish, cease to be, be no longer in the task list. How about fixing that?)

re: VC++ IDE / Design Time Stress Testing

DaveG — Tue, 12 May 2009 18:42:07 GMT

Hi Jianhua, the new Intellisense (which iirc is SQL-based) does sound interesting, and I'm glad you're testing the heck out of it.

But, if it turns out that for some scenarios Intellisense does more harm than good (i.e. longer build times, UI unresponsiveness) will there be an easy way for folks to turn it off?

As you may know, the way folks currently turn Intellisense off is by renaming MS dll's...

re: VC++ IDE / Design Time Stress Testing

Mark Hall (VC++) — Tue, 12 May 2009 19:30:53 GMT

DaveG: yes, there will be a way to turn off intellisense via configuration settings.

re: VC++ IDE / Design Time Stress Testing

Jianhua Li — Tue, 12 May 2009 23:01:32 GMT

Hi Pingpong, can you share more details about how the intellisense doesn't work for Boost graphs library with VS2008? I would like to raise this issue to our team. Thanks.

re: VC++ IDE / Design Time Stress Testing

Andy Rich — Wed, 13 May 2009 02:02:42 GMT

It should be noted that Stress testing represents just one arm of our overall intellisense testing strategy - we have targeted correctness and targeted performance tests, but the point of these stress tests is to measure if (and how) the experience degrades over a long period of usage.

Stress was very valuable (from our perspective) in detecting and reducing the number of Intellisense crashes and hangs in VS2008 - it took us a long time to get that number up to 98%, because it definitely did not start there.

re: VC++ IDE / Design Time Stress Testing

lviolette — Wed, 13 May 2009 07:58:32 GMT

Here's a real-world scenario I'd like to see but can't seem to make work in the October CTP.

I think this is Intellisense related, but may not strictly be a part of the Intellisense system.

Create a template MFC Appwizard generated app.

In stdafx.h, add:

#ifdef _DEBUG

#define DEBUG_ENABLED

#endif

In Mainfrm.cpp::CMainFrame() add:

#ifdef DEBUG_ENABLED

OutputDebugString(_T("DEBUG_ENABLED is defined.\r\n"));

#endif

#ifdef _DEBUG

OutputDebugString(_T("_DEBUG is defined.\r\n"));

#endif

I'd like to see the OutputDebugString statements dim when the Release configuration is enabled.

Then, darken when the Debug configuration is selected.

Would be great if this worked without needing to recompile or anything like that.

With CTP, I can't get #ifdef'd code that is compiled out to grey at all.

re: VC++ IDE / Design Time Stress Testing

pingpong — Wed, 13 May 2009 18:25:45 GMT

Jianhua Li: The Intellisense fails with 'Can't find the identifier' message in the statusbar whenever I'm trying to get parameter information on calls to BGL's graph layout algorithms. I have 'using namespace boost' at the file scope, preceeding the code causing the Intellisense to fail. Similarly, I can't get autocompletion when I type 'circle_', for example. I'd expect 'circle_graph_layout' to be autocompleted.

Platform is VS2008 SP1 with Boost 1.38.0.

Visual Studio Links #118

Visual Studio Hacks — Wed, 13 May 2009 18:50:33 GMT

My latest in a series of the weekly, or more often, summary of interesting links I come across related to Visual Studio. Greg Duncan posted a link to the Managed Stack Explorer on CodePlex . Brad Abrams announced that the .NET RIA Services May 2009 Preview

IDE colors seconded

Greg — Thu, 14 May 2009 01:56:40 GMT

Agreed. Please let us set the colors of the ide windows/subwindows. Please let us set the IDE to a flat color scheme (no gradients, no reflected icons, etc.)

This greatly helps us that work more effectively with a clean uncluttered interface due to vision issues.

This is simply the same usability testing/evaluation done when comparing how a 22 year old uses the product and how a 50 year old uses the product. The 22 year old is more tolerant of low level changes in the visual interface while the 50 year old is much less tolerant of visible changes in the user interface.

One easy visual test is to take a snapshot of the IDE, run it through an image edge detection filter (Sobel) and see just how many lines and how dense the lines are in the result. A high density of lines (edges) indicates a noisy user interface. This noise reduces productivity.

IDE

Greg — Thu, 14 May 2009 01:58:57 GMT

I reduce the number of buttons in the IDE to 15 or fewer in VS 2008 to reduce clutter and remove tools I rarely or never use.

re: VC++ IDE / Design Time Stress Testing

Ben — Thu, 14 May 2009 22:21:26 GMT

For a "targeted correctness" test, try using Intellisense from within a Google Test macro (TEST_F or TEST). I've found Intellisense to almost never be able to do any auto-completions correctly from within such a macro.

re: VC++ IDE / Design Time Stress Testing

Jianhua Li — Tue, 19 May 2009 03:14:57 GMT

Hi Pingpong,

Thanks for providing the repro information.

We will work on it and update you as soon as possible.

re: VC++ IDE / Design Time Stress Testing

Nick — Thu, 21 May 2009 04:36:23 GMT

Some minor specific suggestions:

- Incorporate a random number of file switches during the retyping of the file, to simulate moving between files. Make sure you include all the file types which VS processes differently (eg: multiple languages, .xml files, browser help windows, etc.).

- Add some tests to jump (using Intellisense commands such as 'go to definition') into STL/boost/MFC header files, and make sure all the objects in the headers are being evaluated correctly, and in the context of the source file you came from.

- Use temporary macros to rename some symbols (eg: find in files -> temp macro record -> Next in Results -> do replacement (paste or manual typing with some arrow-based navigation) -> temp macro end -> playback repeatedly until end of file). Make sure both the macro steps are fast, and the symbol data is still correct.

- Use macros to define some symbols (eg: DEFINE_SYMBOL_WITH_POSTFIX( Symbol, Postfix ) -> class Symbol##Postfix { [...] }: ), and test Intellisense response and correctness as macro is modified.

Just some thoughts, hope they help.

re: VC++ IDE / Design Time Stress Testing

Nibu Thomas MVP — Wed, 27 May 2009 10:23:09 GMT

I'm using VS2005, now we're moving to VS2008, all of my teammates are facing problems with intellisense so we've disabled intellisense by renaming necessary dlls. We now use Visual Assist which quite sadly is bit expensive. After spending money on the IDE it's quite sad we've got to spend few more on intellisense when the IDE itself came bundled with one.

Don't know if you guys stress tested VS2005/VS2008 in the same manner, if you did then you need to change the tests since it wasn't productive.

The whole IDE looks dumb/useless without this feature. Please make sure VS2010 works well in this respect.

Thanks,

Nibu

re: VC++ IDE / Design Time Stress Testing

dritte — Sun, 31 May 2009 14:01:32 GMT

C++ -> File->New->Project from existing code.. doesn't work properly. 'Browse' and 'Add' buttons have no any actions.

Beside, working with external sources files is a horror - i mean about Solution Explorer - its view by Header Files, Source Files are totally unflexible!!!.

It would be great if I could Add a folder(with many subfolders) with sources 'as a link' form any location and Solution Exporere show me that with properly hierarchical SUBFOLDERS TREE, not as only files with alphabetical order !! I know it's posible to change the view to 'Show All files' mode, but in this mode, I cannot hide unnesesery folders. Look how it's done in others IDE - for exaple Code::Block.

Other thing adding about 1000 files and removing them from project freeze my dual core with 4GB for few minutes- it looks like VS updates tree view in Sol.Expl. by every processed file. Suspend Solution Explorer tree view updating for time when files are adding.

re: VC++ IDE / Design Time Stress Testing

Damir V. — Sun, 14 Jun 2009 23:30:06 GMT

Nibu Thomas MVP: Yes, Visual Assist is now quite expensive product (they were quite cheap when they started), but it is totally worth it!

We've been using it for several years on VC6 and now on VS2008. It just an overall good product.

It does have bugs and it isn't always accurate, but it is 100 times better than VS's Intellisense. Plus it has lots of other time saving features that pay for themselves: refactoring, switching between h and cpp, enhanced highlighting, etc.