In the past several months Visual Studio and I have been really busy stress testing each other. This post is a general overview on what we've been up to and what kind of testing we're doing. I've learned a lot about stress testing and I have to say it's actually a lot of fun, so I guess it's worth sharing. I'll try to make this a series of several posts, diving into more technical details in the upcoming posts.
During Beta 1 and Beta 2 it became painfully obvious that the new VS had an obesity problem: it was slow, consumed a lot of memory and the worst thing, with enough modules loaded it stopped fitting into the 2GB address space on 32-bit machines. There were several reasons for this, which Rico Mariani, Brian Harry and others have extensively blogged about. In a nutshell, with a lot of new functionality a lot more modules were loaded into memory. Besides, we now had to fully load the CLR and WPF at application startup. Moreover, there were all kinds of memory leaks all over the place.
Of course this wasn't good, so our management made the right decision to make performance our top priority. Jason really took it seriously and we dedicated a lot of people to work fulltime to make Visual Studio fast and lean. As part of this effort I became a member of a virtual team called "Perf SWAT". This team is responsible for essentially three things: performance, memory consumption and design-time stress.
Performance is clear: we need to be fast. Memory consumption is clear too: when we load, we need to take as little memory as possible, and avoid things such as double-loaded modules, loading both NGEN and IL versions of an assembly, and so on.
As for design-time stress, the goal is once we're loaded into memory, jitted, warmed up and all the caches are filled, we should not continue to grow in consumption. This means find and eliminate all memory and resource leaks. Run-time stress means finding leaks in the CLR and BCL, design-time stress means finding leaks in the VS and tooling. I am responsible for design-time stress testing for the VSL team (managed languages). I need to make sure that there are no significant leaks in 4 areas:
The good news is that we've made tremendous progress since Beta 2 and have brought the product into a much better state: it is much faster, more responsive, takes up much less memory and we also hope to have eliminated all major known memory leaks. A common complaint was that VS was growing in memory during usage and you had to restart it after a certain time. Right now we hope that you can mostly keep Visual Studio open for days (even weeks) without having to restart it.
The official sign-off criteria is that the end user needs to be able to keep VS open for an entire work week without any noticeable performance degradation (this means 5 days times 8 hours a day). We've calculated that in average continuous human usage of 40 hours is equivalent to running our tests for 8 hours (tests are doing things faster than a human).
We have identified and implemented a series of 22 tests for all the 4 language teams mentioned above. Each test covers one continuous kind of activity, e.g. CSharpStressEditing, CSharpStressNavigation, CSharpStressIntelliSense, CSharpStressDebugging, CSharpStressUI, VBStressEditing, VBStressProjectSystem, FSharpStressEditing, and so on.
Each test runs for 8 hours on a machine in the lab and VS memory usage details are automatically logged. We've also developed tools to automatically analyze the stress logs and produce Excel spreadsheets and charts for analysis and reporting.
Several months ago a typical test would start at about 300 MB ProcessWorkingSet and crash after several hours with OOM (Out-Of-Memory exception). None of the tests would even be able to run for 8 hours. After finding and fixing a lot (a lot!) of bugs, we were able to get it running for 8 hours – VS memory usage grew from about 300-400 MB of WorkingSet to over 1 GB over the period of 8 hours (that was anywhere from 200-500 stress iterations).
Right now a typical test starts at about 150-200 MB and finishes 8 hours later at 200-300 MB. Also, instead of 500 iterations, it is able to do 3000-5000 iterations during 8 hours on the same hardware. Which means we made it considerably faster and also reduced the leaks in major feature areas to a minimum (right now a feature is considered not leaking if there is average increase of less then ~5KB per iteration).
I'll try to continue blogging about our stress testing and dive more into the technical details: what we measure, how we measure, how we find bugs and how we'll know when we're eventually done.
This sounds great! As a developer, if VS2010 slows me down, that's a solid reason not to upgrade.
great article! keep em coming.
Didn't understand the following statement pretty well: "right now a feature is considered not leaking if there is average increase of less than ~5KB per iteration". If there is no memory leak, why is there still this ~5Kb memory consumption increase on each interaction?
I think "leaking" here might mean just growth of consumed memory. not actual leaking..
Very interesting. I'd very much like to know more about the tools you use for stress testing and analysis. Do you use any profiling tools to find out performance bottlenecks? Do you use any specific tools to detect memory that wasn't freed?
More interesting question: how does VS2010 pass your own SA tools? :) Would be interesting to see any PREfast results...
I hope you are also running the tests in combination with each other, as well as in isolation. I am likely to perform every one of the activities you mentioned in a single day. As Brian Harry has been blogging about, over-emphasizing repeatable tests at the expense of real-world scenarios is apparently what led to a lot of the performance problems in the first place.
Thanks everyone for your questions! I'll answer them at length in the upcoming posts.
Luciano: suppose the test is editing a file. Undo history keeps track of all changes to that file. The Undo buffer is growing with every edit, but you can't consider it a memory leak. Hope this clarifies.
I really appreciate the transparency that the Visual Studio and .NET teams have been showing over the last couple of years. Don't let the marketing folks shut you up! Real honesty and transparency, even when it's not flattering, makes for much better long-term marketing. Everybody else knows about the problems: it can only enhance Microsoft's credibility when they talk about them as well. (And conversely, failure to talk openly about them makes it seem like folks at MS are out of touch -- like the MS manager I heard about recently who supposedly told everyone on his team that they needed to get rid of their iPhones or they'd be fired.)
Wonderful!!! Please remember that even small changes in performance makes a huge difference in the lives of countless developers. Beta 2 was really slow all around. And using the WPF designer made the memory footprint go through the roof. Can't wait to get the final bits.
Thanks for the great feedback. I sincerely hope it is better in terms of performance than VS 2008 as that is sloath in terms of developing programs which use the data designer with lots of tables / stored procedures.
Hi, which version of VS do you guys use to develop the VS2010 studio? Would be interesting to know if you use VS2010 to debug VS2010! Yes, trivial, but then I can say to whiners that MS' products, even in beta/RC can handle its weight on a serious product like VS.
I really appreciate your honesty. My thrust is bigger now!
Keep up the performance tuning... it really makes a difference. I think if you can make VS2010 faster than VS2008 it will move more people over than "new features" and support for "new technology".
Blow my socks off please!
From comparing of same project build in various environment including Atom 330 and Q8400 both with plenty of RAM and relatively good average seek time HDD in XP I could say that 2010 is mostly depended on CPU performance. Tuning over clocked CPU frequency on Atom shows linear improve in total build time.
Is there any unified benchmark project to measure VS performance?