The official source of product insight from the Visual Studio Engineering Team
At the end of April, we released a telemetry system to monitor and report the performance issues that our customers face during their everyday use of the product. To start off, I would like to thank everyone who installed Visual Studio PerfWatson! These reports have provided valuable insight into where you are encountering issues, and have helped us prioritize our performance investments for the next version of Visual Studio (“vNext”).
As many of you have seen, PerfWatson monitors delays on the UI thread, and submits error reports on these delays with the user’s consent. These reports include information on the duration of each delay and a mini heap dump of the root cause for the delay.
When we receive these error reports, we aggregate the data collected, extract the stacks from the mini dump, and analyze the results to identify the largest and most frequently seen delays. We also aggregate the data collected with the delays we have seen using PerfWatson internally. Periodically throughout each milestone, bugs are generated for each team in the division to track the specific subsets of frames (or methods) that are responsible for these performance issues.
Bugs are prioritized based on a combination of the total number of delays we have seen with that frame and the average length of the delay. This ensures that both very long delays and frequently encountered delays receive our attention. Because we have live data associated with each bug, we can track progress across the entire product, within specific teams, and also across internal “dogfood” builds, enabling teams to easily verify the progress they are making.
Below is an example of a bug we have fixed for vNext. The PerfWatson data shows that this bug was reported 16588 times in total for the released build of Visual Studio 2010 SP1. This data is an aggregation of all results we have collected both internally and from our customers.
Total PerfWatson hits per build. X axes are subsequent build numbers.
As the chart below shows, after fixing the bug, the number of delays seen for this issue has gone down significantly or have completely disappeared with every new build of the product created with the code fix.
If you are interested in learning more about this process, or seeing some of the tools in action, check out this Channel9 video, where Cameron McColl from our product team discusses these mechanics in more detail.
So with all the data we have collected so far, what trends have we seen? Below are three areas that have stood out:
While we had received feedback that loading a solution can be slow, PerfWatson has given us a much clearer picture into where, specifically, delays are most likely to occur. A number of steps take place when a solution is opening, each contributing a certain amount of cost to the overall experience. However, when looking at the data, we’ve found that several tasks were more expensive than we expected. For example: reopening files that were open in the last session, or retrieving the solution settings, such as breakpoints and window layout, from SUO file (Solution User Options) can take more than a few milliseconds extending the overall cost of loading the solution.
One way we have found to help improve this experience is to reduce the overall document clutter. There are several ways that a number of files get opened besides clicking on them in Solution Explorer – Go To Definition, Debugging, Find etc. This experience has now been simplified with the introduction of a “preview tab” for opening files that are not yet needed by the user. This new IDE feature was demoed during the TechEd conference held mid-May (see Channel 9 coverage at 00:27:20 minute marker).
In addition to simplifying certain scenarios like opening files, the Visual Studio team is investigating several ways to optimize the solution open scenario for vNext. One method, for example, we are investigating is moving operations away from solution load time to when there is less competition for system resources. The PerfWatson data is helping us understand which areas would benefit from applying these improvements along with making sure there are no regressions.
Visual Studio has a mechanism for components to queue up non-blocking work for a time when CPU cycles are less constrained (aka, the system is “idle”). These idle operations take place on the UI thread when the message queue is empty. For information on Idle Loop Processing please refer to the MSDN documentation here.
This mechanism worked well when the product was smaller, processors were single core, and only a few operations utilized this mechanism. However, as the product has grown, this mechanism has hit its scalability limits as more tasks have started to schedule recurring computations and expensive IO operations during the “idle loop”.
PerfWatson has enabled us to quickly identify which of these scheduled tasks can become potentially expensive. The following are a few examples that we are tracking as bugs:
For each task, we look at a variety of ways to improve the performance. For tasks which iterate on large data sets, we look at ways to break assumptions that the operation needs to be completed as a single operation. This allows the task to be partitioned into much finer increments, quickly yielding back to the UI thread. For example, our Auto-recover mechanism has been updated to do this. Rather than saving all changed documents at once, we instead save one at a time, yielding back to the UI thread after each save. For others, we’ve rewritten the tasks to execute asynchronously off of the UI thread, allowing us to take better advantage of multi-core hardware and remove expensive IO operations from the UI thread.
Idle issues are not easy to reproduce; however with the use of PerfWatson data collected both internally and from you, our customers, we have been able to identify and open bugs for over 30 idle specific issues.
PerfWatson data indicates that one of the more frequent areas that cause the IDE to be unresponsive is solution build. For vNext, we have made significant investments to improve the overall build responsiveness by moving the build off the UI thread for VB and C# builds (C++ already does this). This feature was demoed at the TechEd conference (Channel 9 coverage at 00:45:00 minute marker).
As part of doing this work, we found that not only could we make the UI more responsive during build but, for C#, we were even able to parallelize the build, improving both build throughput and memory utilization. (VB was best optimized by moving it off the UI thread).
We hope this blog post has given you some insight into how we’re using the PerfWatson data to make improvements to the product. Thanks again to everybody who has already downloaded and installed the extension. For those of you who have not yet done so, the extension can be downloaded from here. As always, we welcome your comments and feedback!
Shibani Basava Software Development Engineer in Test, VS Team
Thank you very much for being so open about this, and especially for telling us what you found and how you're fixing it. This is such a breath of fresh air compared to recent years when it seemed Microsoft did not care about efficient performance. Please continue to keep us informed! I'm excited about using the next VS.
Overall, I have found VS2010 to be much faster and more responsive than VS2008 - nice work! This may slightly off-topic, but I do still find SQL Server Management Studio to have several places where it hangs. I hope that SQL Server vNext will be include the same improvements as Visual Studio vNext. Thanks!
Creating this extension was a great idea and I am glad that it had yielded such useful results.
I wonder if you could share the mechanism that PerfWatson uses to detect delays on the UI thread and capture the mini-dump? Or maybe even release some version of PerfWatson as open source? I can imagine this being enormously useful for other large .NET applications.
Thanks for the blog.
The experiences of me and my team with VS2010 have been vastly different from that of Mark, in that we find VS2010 very slow, much slower than previous versions of VS. Perhaps Mark's projects are of different size and / or in different language than ours (ours: C++, 1-3 MLOC).
Anyway, the issues with "Idle Loop Processing" look promising as targets for fixing. Let's hope you fix enough of them in vNext. Good luck, we'll keep the reports from PerfWatson coming.
I agree with Dean that this seems to be the first VS version since VC6 where MS actually seems to care about performance. They gave it lip service for every previous version, they let us down terribly in VS2010 with "VS2010 is the new VS6", but this post actually looks like we'll finally see some improvement. Please don't let us down!
Somehow you seem to have forgotten to mention the part where all this is used to improve VS2010. Because it is, right...? Surely, you're not asking your customers to provide you with all this data while offering no actual improvements to the product in return. Right?
Since VS6's IDE is gone, VS become more and more slow.I don't know you guys derived from devenv.exe instead of msdev.exe.
The good news is Office is still write in native language,so it still have good performance.The VS team manager made a bad decision to write an IDE in managed language,and they keep this bad idea for about ten years from vs2002 to vs2010.
I hope this guy get a promote or join some other good company such as Google/facebook,just leave VS team please.
Having VS as a managed IDE has allowed a ton of 3rd party plugins to prop up unlike before. Not to mention that rapid development and small releases that MSFT has offered is sooo nice! I welcome the managed IDE and hope more of it will be managed! And to be honest with a modern machine, VS performs pretty good. I know that not everyone can afford to get newer machines, but maybe productivity is a case for buying new hardware.
If you like the managed VS then your solutions are too small. When working on largeish (> 1 MLOC) projects even VS2008 is far more responsive than VS2010 even though VS2010 has better features.
I take a chance to underline what I consider the most annoying VS perf issue for me (apart of course the startup long time):
-> when an unwanted exception pops up at debugging time, I'd like a single click option to: don't bother with this exception type anymore at debugging time
-> now I am forced to open the exception dialog (around 4 to 6 seconds on my fast laptop), find the exception type, deactivate it! What a loss of productivity each time I do that, and it is typically a few times a day!!