My name is Li Shao. I am a Senior Software Design Engineer in Test on the VC++ team. In this blog, I would like to share the performance enhancements we've made in VS 2013 Preview to improve the C++ IDE and build system.
Performance is a vital part of software quality. Over the last couple of releases we have made significant performance improvements in areas such as solution load, IntelliSense and code navigation. In VS 2013, we focused our performance effort on a few other key scenarios such as configuration switching, responsiveness and build throughput. We also improved the way we monitor real-world performance, something we refer to as Performance Telemetry.
Based on feedback from C++ developers, we focused our performance tuning effort on configuration switching. We used a real-world application with 800 projects as a benchmark solution for driving performance gains. One of the major contributors to configuration switching time we discovered had to do with the complexity of the file structure in the solution navigator. We had some inefficient ways of re-computing file expansions state and filters – by focusing on this code plus a few additional bugs along the way, we drove some very significant performance improvements to configuration switching times. Below is the data before and after the performance improvement for this real-world application:
Figure 1: Performance improvement in C++ configuration switch operation
For this solution:
In addition to performance tuning, we have also added telemetry so that we can find additional opportunities to improve performance across all solutions. Below is a snap shot of the telemetry data for switching configuration:
Figure 2: Telemetry data on configuration switching
X-axis is the time to perform configuration switch operation in milliseconds. Y-axis on the left is the number occurrence in each bucket. Y-axis on the right is the cumulative percentage in each bucket.
From the telemetry data, we can see that the median elapsed time for configuration switching is now less than 100ms. There are less than 10% of the instances taking more than 6s. If you still feel that Visual Studio 2013 configuration switching performance is not up to your expectation, please help us by providing a repro and profile using Visual Studio 2013 Preview Feedback Tools.
One major infrastructure investment we have made to improve performance is to move the C++ project system into a multi-threaded model. This provides agility for most code in the project system to run on any thread, allowing for concurrent execution and better responsiveness. This work also reduced the sources of hangs. For example, in VS 2012, an individual project was loaded in about 3 chunks, and each happened on the UI which included I/O. In VS 2013, with the multi-threading work, we are able to do much of the work (including I/O) off the main thread completely. The parts that remain on the UI thread are executed in smaller chunks so responsiveness is a lot more guaranteed than before. My colleague Andrew Arnott will discuss this work in more detail in future blog post.
For the editing experience, we now have telemetry infrastructure in place to help us monitor real-world typing responsiveness. Figure 3 shows the Telemetry data for typing responsiveness in VS 2013. You may remember Eric Knox's blog post on VS 2012 typing responsiveness. Compared to VS 2012, VS 2013 is on par with VS 2012 with 98.8% of C++ keystrokes processed in under 50ms (Figure 3.A). In the >50ms range, we are also seeing more data points shifting from the 100-250ms (VS 2012 0.48%, VS 2013 0.30%) bucket to 50-100ms (VS 2012 0.37%, VS 2013 0.57%) bucket, compared to VS 2012 (Figure 3.B). The VS 2013 Preview C++ typing median responsiveness is 15ms (Figure 3.C).
Figure 3: VS 2013 C++ editor typing responsiveness telemetry data
In addition to typing responsiveness, we've also fixed a variety of performance bugs, including an issue that could result in an O(n2) algorithm when saving files that are checked into source code control. This can greatly improve the experience when dealing with large applications.
For the VS 2013 Preview release, we have made various fixes to improve compiler throughput. Based on our internal measurement, we have improved the compiler throughput in scenarios where you are not using the multi-processor compilation flag (the non /MP case) by 10% (Figure 4.A).
For those of you who are using the MP compilation switch, we have also reduced the compiler overhead and improved the overall build throughput on machines with many cores. Figure 4.B shows the throughout improvement when building a large internal application on a 48-core machine. In this example, the overall build time was almost cut in half in VS 2013, compared to VS 2012.
Figure 4: C++ build throughput improvement
In VS 2012, for a Windows Store app which has references to one or more WinMD projects (a project that generates WinMDs as output), a change to the source file of the WinMD projects will trigger a full build of the app. In VS 2013, we have made a fix so that the linker timestamp of the WinMD does not change when there is no content change to the WinMD file. This speeds up overall build when making non-content changes to the referenced C++ WinMD. Figure 5 shows an example improvement with one of our internal Windows applications. The same change in the WinMD project would take 15 minutes to build and cost almost the same time as building the entire solution in VS 2012. In VS 2013, it only takes a minute to do the incremental build.
Figure 5: C++ windows store application incremental build throughput improvement
Note that you may not see the full performance improvement if you have managed WinMD projects as part of the reference chain. The reason is that unlike C++ WinMDs, managed WinMDs contain both type signature and implementation, which means any change will cause the WinMD timestamp change. We'd love to hear from you in the comments section below or on our UserVoice site if this scenario is important to you.
As I mentioned earlier, we have made performance improvements in the IntelliSense and code navigation (browsing) features last release. To help us to continue to improve on the performance of these scenarios, we now collect telemetry for Intellisense/browsing operations including Member list population, Auto-completion, Quick info, Parameter help, Go to definition and a few other related operations. Figure 6 has the examples of the trend data that are now available to us. For example, in the trend data below, we can see that the median elapsed time for auto-completion, member list and go to definition are 105ms, 175ms and 226ms respectively. The majority (90%) of the auto-completion and member list operations are now completed within 200ms. We'll be using this data to prioritize where we invest to improve performance over time; you can help by opt-ing into our Customer Experience Improvement Program (see this blog post for more details)
Figure 6: Telemetry data for Intellisense and code navigation
Above are the important performance improvement and infrastructure work that we have done for VS 2013 Preview to improve the IDE and build throughput experience. We appreciate your continued feedback. If you have not already done so, please download the VS2013 Preview release, and provide us with feedback in the comments section below, through UserVoice, using the VS 2013 preview feedback tool to help us prioritize the work we do to improve the experience when working with Visual Studio.
my open tabs don't load files anymore since update 2. used to be it would fail to do that rarely, now it is all the time. the file that stores these is also now a mess compared to when I used 2010. funny, when I updated to update 3, it worked again, but only twice (as in two VS starts). more funny is that when I manually open these files (I snip a screen shot - across at least two monitors - to know what tabs/files were open) the files open exactly where they were (same panel, same tab location) so it knows what it is supposed to do, only it doesn't do it.
yes, it takes a long time to load a couple dozen files manually, in a sln of over a 1000 files.
Thanks, Li, looking forward to trying VS 2013 out.
The build time and config switching improvements are very nice, and are greatly appreciated.
Too bad the results were compared too an editor that types text slower than me. Show an improvement compared to vs2005/2008 and you'll have something to brag about. 2010/2012 are a distaster where it comes to performance, beating those doesn't mean much sadly.
"Figure 4.B shows the throughout improvement when building a large internal application on a 48-core machine. In this example, the overall build time was almost cut in half in VS 2013, compared to VS 2012."
I'd love to hear some more detail on this point. Can you summarize the improvements responsible for this, and/or the characteristics of the projects likely to see the greatest benefit?
@ ide problems with tabs not opening files anymore
It could that the .suo file is corrupted. Could you open a connect bug using the VS 2012 feedback tool (VSUpdate is for VS2012). The link to feedback tool is visualstudiogallery.msdn.microsoft.com/f8a5aac8-0418-4f88-9d34-bdbe2c4cfe72.
For the file loading performance issue, could you use the same feedback tool that I provided above to report a performance issue? The profile will be taken for your repro while you go through the reporting process.
Thanks for providing feedback!
(This did not post the first time - a common problem with the blogs here, for years, you know, so here it is again - hopefully just the one.)
"file loading" problems are the time it takes for me to pick them out of the solution list manually, among 1000+ files in there.
The tabs not loading at start is not new from what I have seen around. It works sometimes. Like I said, it worked after update 3 (maybe it broke bad after update 1, not 2) for a couple of starts, then MIA. I have also (over the months) done the IDE export sans window layout and then import that back, and that too worked, but again, only for one or two starts. And I have to do my window layouts again.
I had a string of lucky days by doing this: I opened a fourth panel (a panel is that which holds the tabs, you know) and put that on a fourth monitor and left it open. It worked. Made it through five or six days. More than any streak in a long while. That stopped working, too.
Like I said, when I pick the files out of the solution list, these open exactly where they, exactly to the same tab slot. It also opens up the panel (container) where at start it was not there. When I start, all I see are the main (empty) screen with all the menus/buttons (you know), and the output panel I have on a third monitor.
Anyway, no way to recreate this. I won't waste anyone's time on Connect (mainly mine). But it is not uncommon. It should be investigated.
@ ide problems with tabs not opening files anymore
I will forward this to the appropriate team who owns this feature. Do you have an email that you can share so that we can contact you as needed? You can send to lishao at microsoft dot com.
@ S. Colcord Can you summarize the improvements responsible for [build speed improvements]
Unfortunately, we cannot divulge the "how".
I'll find the right person to share the characteristics of the projects most likely to benefit. Look for a comment soon.
@S. Colcord, re: characteristics of projects likely to see the greatest benefit
Large projects building on many-CPU machines are likely to benefit whereas small projects and/or building on machines with fewer cores are unlikely to see a benefit.
Does this help? If not, ping me at email@example.com. Thanks!
Not being positive about this post, the performance results etc. But most software developers won't care about these statistics. Visual Studio must be extremely responsive. VS2010 was disastar, VS2012 improved a lot (not anyway near to VS2008).
Why can't VS2010+ versions open the solution files as swiftly as VS2005-08 would do? What has changed (especially for C++ projects). Developers don't have a 48 core machines, most have 2-8 core machines. VS IDE must be well multithreaded - many operations would just hang the IDE (entire application!) - it must be improved. IDE must be very responsive!
@Li Shao [MSFT], PLEASE LISTEN TO THIS REQUEST. THIS IS A REAL "PITA".
This is off topic but its very much related to the blog's comment system which "ide problems with tabs not opening files anymore" mentioned and we all share.
People are complaining about this for 7+ years on IE, PowerShell, Windows Server, TechNet, Visual Studio, .NET, VCblog, WindowsTeam, other MSDN blogs and Office blog (who decided to take action and replace this junk comment system with third-part one).
The primary problem is, when we add comment as anonymous users, the comment system eats our comments if the session timeout! The session timeout is set to 10-15 minutes, which is not enough for reading article and posting comment. The application should prevent the submission and/or post the error and preserve the comment. Moreover, the other validations, such as the max char count and max URL in a post, upon failing eats the comments and no error message is shown. The page is not using AJAX to prevent it from posting and discarding the comment message.
Secondly, from ASP.NET jargon, the form comments system is using a technology called "Post-back method". The Telligent guys who developed this blog system is a third-party company by ex-MSFTies. They failed to deliver blog based on the state-of-the-art technology called MVC and responsive UI (for mobile and low resolution screen scenarios) which ASP.NET has adapted few years ago and rigorously used on other Microsoft websites. Visual Studio 2013 has bring excellent OOTB responsive UI support for MVC.
So, there should be appropriate error messages, preserving comment message in case validation fails, responsive UI and no post-back.
Interestingly, if you search for ASP.NET MVC tutorial on Bing, the first link from http://asp.net/... walk you through the example of blog system! So, whoever at MSFT decided to purchase blogs system product (back in 2001 ??) is perhaps either retired or left the company. So please "pitch" this idea to the sleeping bosses who matters and make this wonderful platform more wonderful by using amazing people from ASP.NET and Windows Azure teams. John Papa and Mads Kristensen can be consulted and these people can come up with a universal blog system product in days (over the weekend perhaps)! We need better comment system and better error handling.
We can only report this issue over and over again. Everyone is having problem with comment system and its a pretty stale attitude. You are also a stakeholder of this system. If you don't forward this complain, no one else will. Because everyone seems to think the same "this is not my job".
Please convince the bosses and share our pain with them for sake of humanity! Please ask them to dump Telligent contract and borrow those people at Microsoft just for few days for this gig.
You feedbacks are taken. For the solution load, do you have a solution that you can share that is loading slower in VS2010 than VS2008? We would like to take a look. You can either open a connect bug or share with me at lishao at Microsoft dot com.
Thanks for your feedback. I have forwarded your request to our program management team. Hopefully we can work something out.
@Nate: That was beautiful. Thank you.
@Li Shao we outside Microsoft don't really have a channel to report bugs like this (The blogging software isn't on Connect, for example), so we're reliant on someone like you picking this up and bringing it to the attention of the team responsible (whoever and wherever they are). I hope you manage that. Thanks for responding to the issue. :)