Everything you want to know about Visual Studio ALM and Farming
Brian Harry is a Microsoft Technical Fellow working as the Product Unit Manager for Team Foundation Server. Learn more about Brian.
More videos »
Up until Christmas, I was blogging almost every day about the progress we’ve made on addressing the Beta 2 feedback that VS 2010 had significant performance issues. Since then, our efforts have hardly taken a break. I saw active investigations all the way up to Christmas eve and then a bit of a lull until Jan 4th when activity resumed in full force.
The improvements are so many and so dramatic that I really couldn’t enumerate them all here. I think the best way to summarize it is to talk about the feedback we are getting from people as they use new builds. Since our renewed performance effort started, we have produced 3 Super Limited Community Technology Preview releases (SLCTPs – yeah, a mouth full :)). These have been delivered to virtually anyone who has expressed performance dissatisfaction, and for whom we could find contact information. With each one we contact as many recipients as we can and solicit feedback on the changes.
About a week ago, we delivered SLCTP3. A few hundred people (outside Microsoft) have received it and so far we have close to a hundred survey responses from them. We also made a huge push to get everyone in Developer Division upgraded to a January build. I think, as of late this week, we managed to get over 80% of the division upgraded. Lastly, we’ve reached out to partner teams like Blend, Exchange, CodePlex and Oslo to upgrade and give us feedback on the new builds. The difference between now and Beta 2 is staggering.
Before I go into the details, let me remind you of the results we got from our initial Beta 2 surveys.
Our initial external customer survey said that approximately 30% of Beta 2 testers where somewhat or very dissatisfied with Beta 2 performance (the earler version of the “Performance” chart below). After seeing that and running an internal survey of dogfooders, we found that about 70% of internal users were somewhat or very dissatisfied with Beta 2 performance.
Now, at SLCTP3, it looks very different. Our external LCTP3 results look like this. And remember, all of these results are from people who complained about Beta 2 performance (not the 70% of people who didn’t). We didn’t give any of the LCTPs to or survey people who were already happy.
And our internal dogfooder survey looks like this. The “improved” questions are relative to Beta 2. The Ready To Ship and Feature Satisfaction questions are absolute. The performance feature satisfaction question was 70% somewhat or very dissatisfied after Beta 2.
As you can see, the picture looks totally different. The very dissatisfied numbers are nearly gone and the total dissatisfied are way better. I have no illusion that we can make everyone happy – we could work on it for 2 more years and still have a few people who would like even more but I feel like we have really turned the corner. We aren’t done but we are now ready to go out with a broad Release Candidate and solicit feedback from thousands rather than hundreds.
Beyond the statistics, some of the verbatim customer comments tell an even more compelling story…
And not to ignore any negative feedback:
We are following up on all of these with survey respondents.
We are not done with performance or functional improvements but I believe we have addressed all of the major issues and are now ready to make sure we’ve hit all of the combinatorics, edge cases, etc. That’s what a Release Candidate will help us do. In parallel with getting an RC ready, there are a few performance areas where we are still working diligently. Probably the biggest one is solution build performance. Many build scenarios are faster than VS 2008 but we’re also still tracking a number of applications where it is slower (generally between 15 and 20%). We are also putting some finishing touches on some remaining editor/WPF painting and responsiveness issues. I think even without them you’ll find the RC performance to be quite good but we’ll continue to polish it while we wait for RC feedback.
We’re working on getting the release candidate ready now and I’ll let you know as soon as it is available.
It’s been a hectic couple of months working through all of the feedback and resulting issues. It’s not quite what I was expecting to be doing but that’s the way it goes. I’ve got more fodder for posts on things we’ve learned along the way and I’ll try to get a few of them written over the next month or so.
Thanks for helping us work through this and deliver a product that you are going to love,
Brian
I notice the levels of dissatisfaction for local help remain higher than other areas. Will VS2010 stick with the local webserver mechanism for local help? I found that to be an incredibly clunky and slow way to access the docs, compared to the VS2008 help viewer.
I've noticed the same thing. I wrote several posts on the plane flight back from Redmond and one of them is about help. I decided not to drown people by posting them all at the some time. I'll post the one on help in the next day or two.
I have to agree with Brock about the local help. I had gotten very good at finding exactly what I wanted in the local help using the index. The new MSDN web interfaces are nice and all but the light/scriptfree modes are terrible for *browsing* content and the traditional mode is too sluggish to be usable. Please at least just give us a .chm with the .NET Framework reference documentation!
Whilst we all appreciate the effort put into addressing the performance issues, performance is just one side of the coin (or perhaps even just one side of the die) when it comes to dissatisfaction with VS2010. This beta has been the most frustrating release in memory. It's been an absolute battle at every turn just to retain the features and performance we already have with the current product.
However, now that we can put aside performance as a blocker, I hope some of the major issues logged on Connect can be addressed. Here's some of the most serious:
The default platform target has been changed from Any CPU to x86. Up to 95% of your customers have voted against this awful decision, yet our feedback appears to be falling on completely deaf ears:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=455333
and
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=455103
Then we have the fantastic new data contract checking feature, which has been excluded from the Professional version. Over 90 voters so far have taken the time to express their complete dissatisfaction with this decision:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=481327
And finally, as others have said, the new offline help system has been utterly panned:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=505032
Are any of these being looked into? I'm especially interested in hearing some feedback on the default platform target change.
Does this mean that you don't send out the SLCTP 3 to any more customers meaning that one will have to wait for the RC to test the improved perf?
How will you deciede which scenarios you actually will make improvements on once the RC is out and you start to recieve feedback on scenarios that still are too slow? Does the scenario have to be more important after the RC than before?
Brain, I'm following your blog about VS2010 and performance issues for some weeks now. I found it great to get an inside look into how you guys are finishing the product. I've never had the opportunity to observe somebody dealing with such issues as honestly as you do.
The way the team analyzes and follows up the feedback you all get is very interesting for me.
Even though lots of comments you've got are even more complains, you seem to know that it is the right thing you all do... I mean getting it done and sharing the information you have learned!
I keep my fingers crossed for you and I'm looking forward to working with the RTM productively.
Thanks for the insight about your performance push. It's wonderful the way you guys are open about what's been good and not so good with VS 2010.
I'll bet the marketing people are cringing at each post, but you're earning a vote of confidence from the real consumers of the product.
Just re-reading my post/rant above, sorry if that comes across a little stronger than I intended Brian. Didn't quite mean to sound so negative there, especially since you're posting such good news about the performance gains you've been able to achieve.
I do also appreciate the openness your posts and it's great to see issues being acknowledged and the actions that are being taken to put things right.
Believe it or not I am actually looking forward to VS2010 despite my grumbling!
Brian, when the dust settles on VS10, would you be able to write a post one day explaining what changed around VS10B2 to suddenly make MS care about VS perf.
People have been complaining for *years* about VS, it's relentlessly mocked in public by MS staff, and there's been very little sign of anyone caring about perf for release after release, even when some of the fixes were obviously trivial service-pack level stuff (the lame add-ref dialog defaults, blocking the IDE for 3 minutes when you hit F1, etc). Nothing was done.
Then suddenly, some threshold of unbearability was crossed and it apparently became P1 for the whole of devdiv. I can't help wondering if someone senior listened to Jim Allchin's folk record and thought "Gee, that could be me next... "
As an outsider, I'd be fascinated to know how/why it happened.
Lot's of questions :) I'll take them in order.
We are most certainly looking at both functional and performance issues. You've mentioned a few and all comment on those, but quite honestly, most of the fixes we are making are functional ones - only a fraction are performance. I'm only focusing on performance because in the Beta 2 surveys it was the highest single area of customer concern.
That said, I'm not sure you are going to like any of my answers to your specific issues.
x86 vs Any CPU - I'm with you. I raised the same issue right after I discovered the change was made. They made some adjustments, like setting the default back to AnyCPU for library projects. The change wasn't made for stupid reasons. I can't remember all of them but I do remember that one of the issues is that Edit & Continue doesn't work in 64-bit apps. They were very concerned that E&C wouldn't work by default for anyone who was running on a 64-bit OS. We can have a whole nother conversation about how whacky it is that an important feature like E&C doesn't work on 64 bit apps and I'll be in your camp but it's not changing for this release.
Static analysis in Pro - Boy, you want to talk about a controversial issue: What features go in what SKU... I can't tell you the hours of arguments I've been involved in over the years. Ultimately, we have a variety of SKUs at different price points. We work hard to make sure that they all make sense (at least to someone :)). We look at what competitors are doing, we do pricing studies with customers (build your dream SKU and pick your price), we talk to our sales team, we consider the impact on the overall ecosystem, we try to create appropriate "themes" for each SKU that avoids it seeming like random choices, etc. Over time, we make adjustments - like merging Database Professional with Dev or Moving unit testing from Team System to Pro. For now, based on all of these things, we've concluded that most of our static and dynamic analysis features belong in Premium. It's been visited and revisited and won't change for this release. We may shuffle some stuff around for the next release, we'll have to see.
I've got a post on the help system that I'll post today. I'll also read the thread you referenced in detail. I'd like to understand the help feedback better.
Nillebo,
We are still sending out SLCTP3 to people who really want to try it and give feedback. However, no more of that feedback will be incorporated into the RC at this point - the RC ready for final verification and a last few fixes.
However, the feedback will not be wasted, as we will continue to fix issues after the RC.
Yes, our scrutiny of potential fixes increases after the RC but any issue that looks like a real problem for customers will be thorougly investigated and fixed if at all possible. We're very hopeful that many of you will try out the RC and let us know about any additional issues we've missed. It's a big enough product that we've surely missed some.
kjopc,
The marketing guys are less pointy haired than it might seem :) They want the same thing we all want - happy customers and a successful product. We do sometimes debate where the right line is with transparency but, in all, I'd say it's a healthy tension.
Will,
Sure, I can look at writing such a post. I don't think I'd start from the premise that we've never cared about perf before though :) That said I agree, there's an increased awareness and sensitivity that would be worth talking about how we are thinking about it.
With respect to the performance improvements, they sound pretty nice, but I'm a little concerned by this:
"Build time is 10-15% longer for the same solution compared to 2008"
and this:
"Intellisense pop-ups are slow to pop up"
To the average developer, these are likely the two most noticeable performance concerns in the whole system. And remember that the vast majority of developers are going to be coming from VS2008, not beta 2. I'm sure the improvements you've made feel great if you've been using beta 2 for a while. But to the average developer, coming from 2008, what you're describing above is going to feel like a step back. Never mind the WPF build performance problem (which I have to assume you're planning on addressing, because a 500% hit is absurd).
I would view ANY performance degradation to intellisense and solution build (vs. 2008) as being a showstopper.
It seems like your major problem here is that you've framed the comparisons wrong. Tracking performance improvements against 2010 beta 2 might have some limited utility internally, but to the vast majority of developers, you're basically saying, "Hey, look, we suck less than something that wasn't really release quality anyway." That's not much of a benchmark. Sorry.
With that said, like others, I very much appreciate the openness. It's nice to have a view into the process and it definitely does improve my level of confidence in what Microsoft is releasing (in a general sense...obviously, I do still have concerns with this one).
Actually, we've been VERY careful not to fall into the trap of just comparing to beta 2. Only a very small fraction of our survey questions are Beta 2 relative - most are either 2008 relative or absolute "Are you satisfied with...".
I covered Intellisense and build performance in one of the closing paragraphs in the post. We are still actively working on both. Both have improved a great deal but they are among the last issues we are still really driving on.
We have a fix in the works that looks like it will cut IntelliSense dropdown rendering time by about 40%. At that point, I think IntelliSense will be golden.
Build is more complicated because somethings are faster and somethings are slower than 2008 - it's litterally about a 50/50 split. We've been collecting a bunch of different solutions to test build performance against and I'd say we are on the "long tail" which is to say there's no set of few common things causing many things to be slower but rather the issue causing slowness in one solution tends to be different than that causing slowness in others.
I haven't followed the intimate details of the VS2010 performance problems. I appreciate that MS is responding to customer feedback and being very diligent to address the issues.
My question is ... if (what I assume to be) the best development team at MS (the VS development team) has had such difficulty using WPF and the .NET Framework to build a performant, non-trivial application, even though they have access to the best and brightest minds at MS ... how can mere mortals hope to?
It would be much more illuminating if you described the specific problems and how they were overcome, e.g. looping constructs, use of memory, locks, ... so that the rest of us can consider how to better design our code.