Visual Studio Performance – Providing Feedback

Visual Studio Performance – Providing Feedback

Rate This
  • Comments 16

With VS 2013 Preview now available for download, we’re excited to share how we’ve been using the feedback and data you’ve shared with us to drive performance improvements to Visual Studio.   In this first post, we’ll walk through how we’re prioritizing and scheduling performance investments based on your feedback, and outline the best ways for you to provide additional feedback on performance.  We’ll also highlight a few product improvements available for you to try out today.  Over the next few weeks, we’ll follow-up with additional blog posts that walk through the details on how we’ve evolved our performance telemetry systems (including "PerfWatson”) to make it easier for you to help us find and fix product performance issues you encounter, and we will share with you a sampling of performance gains we’ve made, including improvements to our C++, Web, and core IDE experiences.   If you are interested in understanding how we’ve factored performance into our engineering plans for some of our newer experiences, there’s some great content already available in Thursday’s post on Windows Store XAML apps, and in a few days we’ll share more about how we built the Synchronized Settings capabilities into VS 2013 Preview, keeping performance at the forefront of that experience.

As we share the work we’ve done, we’d especially love to hear back from you in the comments section below about your experiences with the product and the overall usefulness of these performance blog posts.  We’ll use this to improve the quality of the content for future blog posts.

Using your feedback to prioritize performance investments

With that out of the way – let’s start by discussing where we look for feedback, and how we’ve been incorporating your feedback into our performance plans.  You’ve been very clear that you’d like us to invest further in Visual Studio performance.  In fact, since launching our Visual Studio UserVoice site, we received over 5,000 votes requesting that we “Improve the performance of Visual Studio” making it the 4th highest voted suggestion ever on our site (just under support for better Lambda expression debugging):

Uservoice suggestion to improve performance of visual studio

Performance also shows up as an area of particular interest in developer surveys, in Connect bugs, on twitter, in forums, in blog post comments, and most recently through VS 2013 Preview’s Send a Smile feature:

Send a smile

We pay very close attention to feedback from all of these sources as we plan each release, and combine it with telemetry data from people who have opted into our Customer Experience Improvement Program via “Help -> Customer Feedback Options” to help us identify where we have the largest opportunity to make an impact on performance.  Things that we factor into this decision when prioritizing and scheduling performance investments include:

  • How much user feedback have we received about a scenario and how important is the scenario to those users?
  • Based on performance telemetry, by how much would we have to improve a scenario to really make a perceptible difference?
  • Based on feature usage telemetry, how many users would benefit from any given investment?
  • Do we have enough information on where we need to make a change/ can we accurately model product usage to ensure we’ve had the intended impact?
  • Will this be a breaking change, and who else do we need to work with before publishing? (Ex: Popular 3rd party extension vendors)
  • What is the opportunity cost to other productivity enhancements we could be making instead?

For example, one of the top areas of feedback for the past few releases has been about improving solution load times.  While we made significant improvements in VS 2012, looking at performance telemetry (summarized below), there’s still opportunity to further improve this experience in a very perceptible way.  (We’ll discuss the work we’ve done to further improve this scenario in more detail in an upcoming blog post – stay tuned!)

Solution load chart

The key thing to note here is that performance work is prioritized and scheduled based on a combination of direct feedback and actual product usage data.   By choosing to participate in the Customer Experience Improvement Program and trying out product pre-releases like the VS 2013 Preview, you are influencing where we invest and helping us validate improvements, even if you don’t have time to send us written feedback. 

What about performance bugs?

Sometimes, performance issues are simply bugs that may not affect a large number of users, but can be very painful or completely blocking to someone who hits an issue.  We take these issues very seriously and try to publish workarounds as soon as possible.  When the complexity and risk associated with fixing these issues is small, we try to schedule fixes for delivery with our VS updates (like the recent VS 2012 Update 3 release).  The tools and techniques we use to discover and fix these issues are a little different than our aggregate feedback and telemetry systems.  We still use systems like the “PerfWatson” system we blogged about last release to auto-detect classes of issues like UI hangs, but we augment this with bug reports filed through Connect - preferably using the Visual Studio Feedback Tool  [VS 2012 Download, VS 2013 Preview Download].

While this list certainly isn’t comprehensive, here is a sampling of some of the more notable of performance fixes that were published between VS 2012 RTM and VS 2012 Update 3:

  • Fixed issues causing a “Please wait for a background operation to complete” dialog to show when copying code to the clipboard
  • Fixed hangs on close when trying to access files on remote shares missing a specific Windows Update
  • Reduced Symbol Loading from 30+ seconds to less than 3 seconds when symbol or source paths were not available [more info]
  • Improved IDE Start scenario when Team Explorer is left open and OS has not yet “warmed up” by ~600 ms
  • Reduced VS 2012 Virtual Memory consumption at startup by 95 MB
  • ~99% improvements to specific version control scenarios when working with extra-large TFS repositories (Ex: 600 MB ‘tf add’ reduced from 2 hours -> 3 min)
  • Page load times reduced from minutes to seconds when loading TFS work item web pages from deployments with hundreds of team projects
  • Eliminated a common 6+ second UI hang when saving web application files

When fixes are larger in scope, or are likely to break 3rd party extensions or SDKs that have already shipped on top of the current VS release, we still try to publish workarounds quickly, but may defer a fix until our next VS Release.  This gives our partners the time they need to try out the fixes and report any resulting issues, and lets us better coordinate with teams who may also be doing feature work in an impacted area.  In addition to the items above, the VS 2013 preview also includes targeted improvements in the following areas:

  • Fixed an extra-long hang when working with large numbers of Windows Phone project types and multiple build configurations
  • Eliminated a common 3-5 second hang when using “Show on Code Map” or its Find all References features.
  • Eliminated a 7 second hang when trying to bring up the Add Data Binding dialog in the XAML editor
  • Numerous fixes to improve IDE start-up time
  • ~26% improvement when adding a new Lightswitch Client to a project
  • Numerous fixes to eliminate delays when opening specific tool windows (Ex: Test Explorer opens up to 1.5 seconds faster)

When these fixes are combined with new performance features and investments we’ve made across the product, we hope that you’ll find the VS 2013 Preview to be notably faster than our previous releases.

Summary and Preview into Upcoming Performance Posts

I hope this post has been helpful to communicate how we think about and prioritize performance work for Visual Studio based on your feedback.  If you’d like to use any of the feedback systems noted above to share your performance suggestions or issues with us, here’s a few quick links to get you started:

Over the next few weeks, we’ll be having members of our engineering teams come share details on specific performance features investments we’ve made in VS 2013 for C++ developers, Web Site developers, and to Core IDE experiences.  We’ll also share details on advancements we’ve made to our performance telemetry systems to auto-detect and report on key product scenario performance.   As we do, we’d love your feedback in the comments section below about how we can make performance-related blog posts even better for you.

clip_image001[5]

Nathan Halstead - Program Manager Lead – Visual Studio Fundamentals Team

Short Bio: Nathan Halstead has worked at Microsoft since 2004, and currently leads a team focused on continuous improvements to Visual Studio performance and reliability.  He’s an avid skier, plays in an adult recreational ice hockey league, and loves playing with new technology.

Leave a Comment
  • Please add 2 and 6 and type the answer here:
  • Post
  • I appreciate the fact that the team is trying to pay attention to performance.  My frustration is that this could have been done over the last three years and left out all of the work for a Metro UI look.  I have yet to see a single vote on uservoice post the 2010 release asking for monochrome icons, all caps menus, and all of the other nonsense that has gone into the product.

    More to the point regarding performance – a smiley face or a frowny face???  Really????  Developers are not kindergarten children and should not be treated as such.  There are a good number of high quality developers that have the ability to both think and express themselves.  From my point of view, this is even more juvenile than “clippy”.

  • I do have to say that after using 2012 for a while and then having to work on a project in 2010, the performance improvements in 2012 were really noticeable. Glad to see this is a continued priority for 2013.

  • I don't know how smart it is to focus on performance -- Scott Hanselman isn't going to be happy unless you permanently bind shortcut key Shift-Ctrl-P to a Powershell script which uses SignalR to spin up a new MVC 5 project that includes support for the AJAX Toolkit and .NET 2.0 incorporating a Web Essentials front-end running Modernizr to surface an Amazon S3 interface talking to Azure Mobile Services with Twitter-based logging. It's almost certain that nobody would ever actually use such a feature, but it would make for quite a demo at BUILD 2014.

    Performance? Bah!!

  • Sorry Scott, almost forgot ... this may be too obvious to mention but that Shift-Ctrl-P key binding also needs to involve NuGet in some way. Thanks.

  • Great work, sounds good! And VS2012 indeed seems to be much faster than VS2010 was (however I still dislike the new UI, even after several months of usage). Regarding performance: When are you going to fix the XML editor? It hangs VS for 30 seconds to reformat a mid-size XML file (around 1MB).

  • @LS - Thanks for the feedback.  Regarding the smiley/frowny - is there a visual metaphor you think would be more appropriate as an icon?

    @Mike - We'll let Scott know he needs to add some more complexity into his demos  ;)  On a more serious note, Scott's been great at sharing any feedback he receives on performance, and helping teams collect the information they need to make improvements, so our thanks if you have shared feedback with him through email, his blog, or twitter.

    @Ooh - It would be helpful if you could add your votes to that suggestion on our UserVoice site, and any comments about the scenarios you are encountering in the XML editor that are still problematic.  Having your comments and feedback added to that item will help us prioritize what we fix, and your votes will help influence when (or if) we invest further here.  Here's the link:

    visualstudio.uservoice.com/.../2255809-improve-the-performance-of-the-xml-text-editor

  • "@LS - Thanks for the feedback.  Regarding the smiley/frowny - is there a visual metaphor you think would be more appropriate as an icon?"

    Couple of ideas: thumbs up/down or Up arrow/down arrow? Must admit, the smiley faces seem a little out of place.

  • Your blog contain really very useful information. Thanks a lot for sharing this information here.

  • @ Nathan

    Don't use icons at all.  They are as useful as facebook likes.  That is to say, they hold no value whatsoever.

    Instead, if you want feedback, allow the developer to submit meaningful, written feedback.  The downside of this is that someone on the VS team would have to read the data.  Even though that can be a someowhat daunting task, it will bring the VS team closer to what developers are actually thinking.

    Of course, uservoice already provides this sort of feedback.  The downside of uservoice is that many of the submissions probably do not provide true value to MS.  Sadly, the ones that do, appear to be ingored since the issues with the most number of votes seem to be declined.

    Perhaps you could enlighten us on the semantic value that the VS team believes it will derive from someone clicking an icon.

  • @LS

    I agree with everything you said, but I think you're reading too much into this smiley icon.  The point of it isn't to get serious feedback, but it's simply MS trying hard to "fit in" and be popular in this social networking craze we're in.  It's interesting how these things look natural in real teenager apps, but when companies/websites that don't normally cater to this try to do it....it looks odd.   That's a hint to the company trying it that they should stop.

    I wish MS would stop trying to fit in; they have the ability to make awesome things, but these days they are too distracted by chasing how to have teenager-friendly software.

  • I don't know how you guys do it on a daily basis...I'd be embarrassed to show that user-voice screenshot when there are legitimate user-voice items with even more votes that are either: declined with no real reason, or have been left there to stagnate.

    Look at this one: visualstudio.uservoice.com/.../3041773-bring-back-the-basic-setup-and-deployment-project-

    ...this one has over 5700 votes and was declined in classic MS style:  a few generic words with no real reason.

  • @LS - Thanks for following up - your concern makes much more sense to me now.  One thing I should have clarified in my post above is that the "send-a-smile" launches a plain-text feedback form for you to share the reasoning behind your smile/frown.  We use the choice of smile/frown to help us clarify the intended tone of the feedback, which may not always be clear (especially if we translate the feedback from a language we don't natively speak on the team).  There's also an option to provide an email address and indicate whether the feedback is about a bug, so we can follow-up 1:1 if needed to resolve an issue.

    Our goal with this feature was to provide a low-overhead way of sharing any feedback you might have.  When incoming feedback volumes are low (such as the first week or two after a release), we typically read every comment.  As the volume of feedback increases, we use clustering techniques to help us categorize the feedback so we can identify trends and any hot-issues we may need to react quickly to.

    Sorry for any confusion - I agree if this were a simple like/dislike mechanism, it wouldn't be very useful for soliciting performance feedback.

    @StartedWithDOS  - I understand your frustration with the decision not to bring back the setup and deployment project type.  Unfortunately, I don't have much context on this decision, and can't really elaborate any further than what was already published to UserVoice.  However, I have passed your feedback along to the team that closed out that UserVoice item so they are aware of the dissatisfaction you share wih other commenters on that item.

  • @ Nathan, thank you very much for the explanation.  Your methodology makes quite a bit of sense.  Before you explained things, I thought the VS team had taken a complete leave of their senses!   :)

  • "fixes that were published between VS 2012 RTM and VS 2012 Update 3:

    Fixed issues causing a “Please wait for a background operation to complete” dialog to show when copying code to the clipboard"

    YOU CAN ONLY DREAM ABOUT THAT! I HAVE UPDATE 3 AND THIS ****** DIALOG IS ANNOYING ME DAILY! AND NO SORRY FOR CAPS LOCK, I GOT THIS IN VS!

  • Whats the easiest way to ask the VS team to remove the perspective switcher 'feature'. I hate it that my Windows keep jumping around uncontrollably whenever I start/stop debugging. Why doesn't stuff just stay where I put it? At the very least 'leave my windows where I put them' should be an option.

Page 1 of 2 (16 items) 12