The official source of product insight from the Visual Studio Engineering Team
Welcome back to the 3rd and final part of the Visual Studio 11 Beta Performance series. This week’s topic is Debugging. As I mentioned in the 1st post debugging is a key component in your continuous interaction with Visual Studio and we heard from you that the compile, edit and debug cycle had hiccups and was sluggish. I would like to introduce Tim Wagner, from the Visual Studio Ultimate Team, whose information below describes the work done to improve the Debugging experience for you.
Visual Studio provides an array of features across the entire application lifecycle, but it’s still the core “compile-edit-debug” loop that tends to dominate a developer’s workday. Our goal is to ensure that these central experiences work well so that a developer can stay in the zone, focused on coding and debugging in a tight cycle. In this post I’ll describe some investments we’ve made in the build and debug part of that cycle (collectively referred to as “F5”).
First, the punch line: We’ve made significant performance improvements in the Beta release! Take a look at this video comparing //BUILD and Beta C++ F5 experience side by side:
When we shipped the Visual Studio toolset at //BUILD, we focused initially on getting a functionally complete F5 experience in Visual Studio for Metro style apps, knowing that we still had work to do on this scenario to improve its performance. Between //BUILD and Beta we pulled together people from across many teams at Microsoft – Windows, C++, C#/VB, debugger, Project & Build, XAML, and others – to create a virtual team focused on making this scenario perform as quickly as possible. In this post I’ll take you inside the work we did, illustrate our progress between the //BUILD and Beta releases, and shed light on the kinds of changes and tradeoffs we made to improve your experience with the tools. Although we’ve made performance, memory, and responsiveness improvements in other areas (and other application types) with respect to building and debugging, this post is specifically focused on the F5 experience for Metro style apps on Windows 8.
Let’s start with some background on exactly what F5 entails. We usually think of the F5 key informally as a shorthand for “start debugging”, but depending on the state of your application, it can involve an entire range of activities:
We defined the three temperatures as follows:
Cold – The first F5 of an app, including its registration. This will include a full build and deploy.
Warm – A one-line change after the cold F5 has completed. Warm is faster than cold because builds are incremental rather than full, the app is registered, various caches are primed, and so forth. Warm is the “sweet spot” where we expect most F5’s to occur as people iterate in tight compile/edit/debug iterations, and so it’s the most important of the three scenarios.
To test the languages, we used the Grid template that ships with Visual Studio to find the first level of performance issues. Using a small example like this has two primary advantages in the initial testing phases: It’s small enough to capture and view the sampling traces easily, and fixed overhead tends to dominate (the rendering of the UI mode shift in Visual Studio is a good example; animated transitions, which were present in //BUILD and then removed for Beta, is another). On the downside, small examples like this fail to uncover breadth, end-to-end, and scalability issues. To catch those, we also look at larger test cases with both synthetic and real world apps, including the Zune and Bing apps you can download from the Beta app store. Real apps are the best proxies for actual developer experience, and synthetic load testing helps us find non-linear problems of scale – for example, we found an issue where we were accidentally turning warm cases cold by deploying more than what actually changed, behavior that stands out with larger apps and especially by plotting deployment times against app sizes.
As you’ve heard in previous posts, we use several methods to do the actual performance measurements, including an automated bank of performance analysis machines that we call “RPS” (Regression Prevention System) which can track detailed performance results over time for scenarios like the table shown above. We also use this lab for manual performance testing to “compare apples with apples” when we look at changes or want to compare detailed perf traces for different approaches. Here’s a graph of trends over time between //BUILD and Beta for the matrix above, minus the “F4” case, using our lab machines:
C++ has the longest F5 times of the four languages, due in part to its nature (include files, link step), so I’ve broken it out to keep the graph above more readable. The yellow bars represent performance targets for the native case.
One thing you can notice from both graphs is that the trend over time is towards improved performance, with occasional regressions. Some of the regressions are intentional – for instance, we fixed several functionality (correctness) bugs where more time was consumed in order to get all the cases correct. Temporary regressions are usually the result of integration issues, when two more pieces of code reach a common branch but have unintended interactions with one another.
In this section I’d like to provide more insight into the C++ case, how we approached improving its //BUILD-era performance to arrive at Beta, and what we’re continuing to do to improve.
When we released //BUILD, we knew that we had some low-hanging fruit in terms of performance improvements. We used Windows ETW sampling (which will be the basis for all profiler sampling in Visual Studio 11) to collect traces and formulate a coarse grained breakdown of the time. Here’s an example of the cold and warm breakdowns for C++ as they ended in Beta:
Using diagrams like this, you can quickly apply some Amdahl’s Law analysis – cold C++ F5 is dominated by the full build time. In the //BUILD version of this graph that was even more true. As a result, improving other elements won’t have much of an effect until/unless building gets faster, so this is where we’ve focused our improvements.
Between //BUILD and Beta we improved the C++ build by pre-compiling headers for C++ templates and – most significantly – by switching to “lazy” wrapper generation. At //BUILD, wrapper functions were generated for all methods of a WinMD type in every file in which that type was referenced, leading to redundant code generation, larger object files, and longer link times. For Beta, we changed the generation policy to only do work if a method is actually referenced, greatly reducing the number of generated wrappers and resulting in faster builds and smaller PDBs. We’re continuing to look at compiler improvements here that could make this even faster, such as generating wrappers only for the first reference.
Compiling the XAML UI language and its interaction with the host language’s type system (either managed or native) is a significant part of the overall build pipeline. For both managed and native, one of the changes that improved performance was reducing the amount of up-front type information loaded by the XAML compiler. In fact, of all the system assemblies, our traces showed that only four contained types that are typically referenced by Metro style apps. Loading the remaining assemblies only when required (a type lookup fails) reduced startup time for the compiler without changing the semantics of type referencing. In addition, XAML files refer repeatedly to the same types; we took advantage of this fact by adding some additional caching to make subsequent references faster and to reuse type information between compiler passes.
Managed Metro style projects benefited from an additional change to the XAML compiler: change detection. The output of the first pass of the compiler (*.g.i.cs or *.g.i.vb files) is used by VS for IntelliSense and does not need to be generated unless the XAML files are modified. To exploit this, a quick “up to date” check was implemented that skips this step entirely whenever possible. This change also speeds up other scenarios, such as repeatedly tabbing to/from an unchanged XAML file.
Even with these improvements, XAML compilation in the Beta release still dominates small and UI-intensive Metro style native and managed app builds. To make further improvements (especially for native, where the compilation times are slower), we’re currently investigating ways to checksum user types, so that we can skip the XAML codegen step entirely when the surface area of the user’s code is unchanged (i.e., the user’s changes affect the bodies of methods, not the shape of types). We’re also continuously looking at the quality of the T4-generated code for XAML, and welcome feedback on its layout and implementation.
We’re very excited about introducing F4 with the Beta release and look forward to hearing your feedback! Note that there are a few cases in Beta, which will be fixed before we release, where F4 may not activate the app – the workaround is to use F5, which is always available.
The default deployment in Visual Studio is to launch the application on the same machine that Visual Studio itself is running on. (We refer to this as “local” deployment.) There are two additional debugger targets: remote, which is useful if you want to deploy to specialized hardware for testing (such as ARM devices, where you cannot run Visual Studio), and simulator, which can be used to test rotation/resolution options, location, and other settings that are difficult or impossible to vary on the host machine when deploying locally.
cpp 1 line
cpp no change
cs 1 line
cs no change
js 1 line
js no change
vb 1 line
vb no change
Internal performance testing, even with real apps, is ultimately a proxy for the experience of real developers. Telemetry helps us understand what happens after we ship. While the Windows 8 Consumer Preview / Visual Studio 11 Beta is still in its early days, we already have some data flowing back. For F5, one of the best indicators is the data that tracks overall deployment times; this doesn’t capture all of the F5 user experience, but it includes many of the expensive portions (build, deploy) and it helps us discover if we’ve missed any cases. The chart below shows the data we’ve received to date, showing the distribution and cumulative population for bucketed deployment time. Around 89% of deployments take 2 seconds or less. The data also confirms that the warm and hot paths are working as intended, since the majority of deployments are sub-second (and hence not cold). Of course this is still early data, from a population likely dominated by smaller apps and examples, so we will continue tracking this along with your feedback.
We hope these and other improvements we’ve made in Beta are providing you additional productivity gains and making the experience of developing Windows 8 Metro style applications a great one. Please help us continue to improve through your feedback and suggestions!
Tim Wagner – Director of Development, Visual Studio Ultimate Team
Short Bio: Tim Wagner has been on the Visual Studio team since 2007, working on a variety of areas, including project, build, shell, editor, debugger, profiler, and – most recently – ALM and productivity areas like code review, diagramming, and IntelliTrace. Tim has been involved in performance improvement crews for the past two releases and continues to look for ways to make Visual Studio a better user experience. Prior to joining Microsoft he ran the Web Tools Project in Eclipse while at BEA Systems, and has been involved in several startups and research organizations over the years.
The Last Word
As mentioned above this brings to a conclusion the Performance Series on the Beta of the Visual Studio 11. I greatly appreciate you taking the time to read and comment on these posts. Even though this is the last of the series let me reiterate that we are always open for feedback as hearing from you is critical in knowing where you feel we still need improvement. As always I appreciate your continued support of Visual Studio.
Thanks, Larry Sullivan Director of Engineering
This post leaves a lot to be desired... In particular, the "deep dive into c++" consisted of a single paragraph that described the change at a high level, ( one sentence if you are only counting the parts relevant to full-experience application development and not metro).
Are there any other compiler improvements that someone from the visual c++ team can comment on with regard to build performance and link time?
Seems like this is very much focused on speeding up build/debugging of Metro apps. What about build/debugging of everything else?
As Adam commented, the "deep dive" apparently consisted of speeding up how WinRT metadata is generated and nothing else.
Great. But you still have a C++ compiler that is absurdly slow compared to, say, Clang. (but at least it implements so much more of the C++ language.... oh... wait, never mind). So really, you've got a slow compiler, an *amazingly* slow linker, and a frequent I/O bottleneck on writing out PDB files. One would think that it was possible to squeeze out a bit more performance, given that, well, other compilers can do it.
Of course, since the VC++ team seems to be too busy inventing new proprietary languages to actually improve their compiler, I guess we'll have to take what we can get. So hey, it's better than nothing. Thanks.
I agree with the previous comments. In addition, the lack of XP targeting support makes it impossible for us to upgrade to VS11 -- we simply cannot abandon that many users just yet.
c++ intellisence is still slow and buggy
That side by side video really drills home how much easier to read the solution explorer was before the colour change... Any news on that yet?
Agreed. The solution explorer on the left is a billion time better than on the right.
On performance, are those two compiling the same code? On the same machine?
I would like to see a performance comparison between VS2010 and VS11 Beta. I believe you can reach massive performance improvements between alpha and beta software. But we don't build with alpha software we build with VS2010.
Hello Adam & Jalf,
On the VC++ compiler & linker side, we have multiple improvements especially multithreading the compiler for Dev11. As a result we have seen large compile time improvements when building some of the large internal MS products. We will follow up with a more detailed blog about build time improvements in Dev11 VC++. Stay tuned and keep an eye on the VC Blog at blogs.msdn.com/.../vcblog
Lead Program Manager
@Ayman Shoukry: *Very* much looking forward to improved build performance. I'm hoping that at least some of the improvements relate to templates; my own tests show VC performance degrading exponentially with the number of template instantiations in a compilation unit once that number goes above 500 or so. A single file doing nothing but instantiating a single small template 8k times took about 5 minutes to compile.
I can notice some minor improvements after trying VS11 BETA, but still I think it's great to have.
As for the next topic, I'm suggesting that you guys talk about performance (and perhaps UX-related matters as well, if any) with plugins? Some plugins can cause a significant performance degradation after installed, for example currently most-used refactoring tools. The areas where performance degradation occured are in VS startup, and mostly while typing/editing code (C#, XAML, etc.). Has there been any performance improvement step taken regarding to plugins, especially in those areas?
@Ayman: oh, I'm looking forward to more information about that. Thanks for the heads-up.
Given how few (at least for several months) apps are likely to be built for Metro, rather than current/legacy apps based on Windows as it is today - it's very disappointing (perhaps even political maneuvering?) to see that the subject of the development / debug / build experience for current Win32 applications has been totally avoided. As I say these are currently the VAST majority of real world use cases......
I'm trying to work on a sizable Silverlight project in VS11 and it absolutely crawls while editing XAML files. It may be Resharper's fault, I'm not sure. It actually has locked up a few times telling me its waiting for a background operation to complete.
It would be really nice if I knew WHAT background operation was running and if I could terminate it (since it has become unresponsive).
I have been using VS 11 Beta as my primary development environment since it was released now. I have noted a couple of bugs that I have yet to report on connect.
About C++ performance:
Performance of full builds seems to be down somewhat going from 5:30 to 6:00 minutes. If it's true that you have multithreaded the C++ compiler I could see that causing inefficiencies when we were already massively parallel in VS 10.
C++ Intellisense is MUCH better, but there are still numerous times when I get the busy dialog for 20-30 seconds, but at least it doesn't pop up without a user action any more.
Debugging is much improved in some ways. Stepping time is for example much better, but it seems to have massive performance problems the first time you show a specific type in the watch/locals/autos. Especially our string templates can block the debugger or 20-30 seconds the first time you break into the debugger.
Debugging this is seems that most of the time is spent concatenating strings generating the type names for symbols in msdia110.dll. The performance was also exacerbated when I implemented visualizers for our types.
Hopefully once I have created a connect bug for this you will be able to do something about it, because waiting several seconds each time I hit a breakpoint is not productive.
You write "Managed Metro style projects benefited from an additional change to the XAML compiler: change detection." and "This change also speeds up other scenarios, such as repeatedly tabbing to/from an unchanged XAML file."
Does this also fit to non Managed Metro projects that contain XAML like WPF? If not, please port that performance feature to all XAML editing. I guess that is an easy one for you.