One of the areas we’ve gotten a lot of performance feedback from Beta 2 on is loading solutions. We’ve made a ton of progress addressing the causes of the slowness. Like on the virtual memory issue I talked about yesterday, we’ve picked a set of “real world apps” – some of the same ones and some different ones to measure solution load time. Most of them are pretty big. I haven’t forgotten that I own you some data about how big these “real world” apps are. That’s next on my list. Again I’ve obfuscated the names to protect the identities of some of the customers who shared their apps with us but where the abbreviation matches with the VM charts you saw, they are the same app (for example: AG and CB).
The good news is that in most scenarios, we are now faster than VS 2008 (Orcas), in fact in some cases substantially so, for example: AV and TZ are almost twice as fast. Here is a chart comparing them (Main 21130.00 is the build from the Main branch on Nov 30th). You’ll notice we still have a few regressions with bugs filed on the more significant ones. As you can see a couple of measurements are still missing.
The first line (Open Dev10) is just measuring the raw time to start VS (for comparison to loading a solution). Ultimately, we are not going to get the raw startup time of VS 2010 to be as fast as VS 2008 but we continue to make progress on it and I don’t believe that is the most important metric. You'll see below a 1s startup regression in the fix queue and there's more improvements in the works, however, having startup be aa bit slower isn’t deadly if the rest of it runs pretty fast.
I suspect you are wondering what is “High-end hardware” and what is “Typical hardware”.
Here’s our list of active investigations for solution load:
You’ll notice a couple of our internal customers I mentioned in an earlier post (Expression Blend and Exchange) in the list of active fixes and investigations. BTW: Don’t read that 12/19 (RC) to mean anything. I don’t know why it says that – our RC plans have nothing to do with 12/19. I suspect they are just saying that by getting in by 12/19 it will be sure to make it into the RC (which is definitely true).
I’ll continue to update you on other areas as we make progress and I get time to write about them.
Brian
"...we are now faster than VS 2010 (Orcas),..."
Isn't Orcos VS 2008?
All this info is excellent news, btw.
Ack, stupid typo. Fixed. Thanks.
You mentioned that VS2010 startup perf will not be as fast as VS2008 and that you "don’t believe that is the most important metric"
While I would agree that it is not the *most* important metric, I would still say it's pretty darn important.
I (and many other devs that I know) use VS for much of my text editing (XML, text, SQL files, etc) and being able to startup the environment and quickly edit a file is very important to me. In fact, I'd say I probably open VS five times more often to edit a simple file than I do to work on a full project (I usually keep these loaded throughout the day). I already find VS2008 slow (and its almost unusable when editing something over a WAN-based UNC connection) so the thought that VS2010 will be even slower is disapointing.
Even scarier is that my dev box is a Xeon X5550 with 15k SAS drives and 6GB of RAM and VS2008 still takes 5-7 seconds to start.
I've installed VS2010 Beta 2 on my personal laptop and I stopped testing with it due to the lack of performance with small tasks.
I am however looking forward to the next RC. For all the perf issues, there's also a lot of amazing features in VS2010 that I can't imagine living without!
I agree with Steven because I use VS for my text editing as well.
Please make VS 2010 loads faster than VS 2008.
Uncle, you're right. I'll go look into it. I'm not comfortable with a significant regression here either. I'm not sure we can get exact parity but I suspect we can do better (some fixes are in the queue, I know, but it's not one of the scenarios I've been tracking closely).
I'll stand by my statement though that it's important but not as important as some of the other things we're working on.
You guys definitely use VS differently than I do :) I think I start it about once every few days and leave it running a long time.
I would have thought 'typical' is a laptop.
But your high end system appears to be a SFF PC and your 'typical' system is a server?
RichB - I suspect that's because they're the kind of machines they can easily rack-up in their labs, rather than because they're supposed to be exact replicas of actual machines people are using.
Not sure how they reconcile the differences with the pathetic I/O performance of a lot of laptops - perhaps they don't..
These "Low-end" and "Typicals" are based on hardware we have in the lab. Most of our lab hardware is rack mounted and that's going to define something about it. It takes a lot of hardware to test VS (lots of teams, lots of tests, lots of builds, lots of runs).
We do testing on laptops, net books, VMs, desktop machines - but the regularly reported, semi-repeatable, automated testing is done in the lab.
Although it isn't perfect, I'm comfortable that choosing based on CPU speed, core count, memory, cache, memory speed, graphics card and disk subsystem is enough to fairly represent a class of machine. Although other things, like chip sets, bios, etc can have some effect I don't expect them to affect the validity of the tests.
I'm not sure what makes you guys think a Compaq dc7800 is a server. Don't you remember when desktop PCs actually sat on the top of your desk?
Dan - nobody did say the DC7800 was a server - RichB said it was a SFF PC, and I said I thought it had been chosen because it could be easily racked-up (c.f. a laptop, which is not very rackable at all).
Interesting. My machine has higher specs than the "high end" system, yet it takes twice the time as the "typical" machine to load VS2008. The thought of VS2010 taking twice as long as THAT is disturbing.
Kevin,
We'd like to know more about why you're seeing slower load performance. Can you e-mail DevPerf@Microsoft.com with some more information about your machine and the operating system environment (if you can run a DXDIAG report and attach it, that should have everything we need for a first cut).
As a side note, these tend to be warm times (meaning VS was already loaded and unloaded first) because they're more consistent. We also typically make sure that background NGENing has completed. If not you'll see more JITing and typically slower load times.
David Berg - Performance Engineering
Thanks for sharing with us with everything. Despite I didn't yet tried beta I got strong feeling I already like it.
On my "typical" (gamers just call it wooden-pc You know ;) machine with 1gb ram i tend to run VS really often (while reading blogs/xna tutorials to test some features/samples) so I actually care about starting times.
Can You make a post about what exactly happens in those 7.1s ?
It seems quite a bit - I do understand that VS is complex system, and this is from 0 to Ready state, but is it the same overhead for opening project?
Were You thinking/testing delaying some parts (in another thread), to achieve faster response and semi-Ready state, even if not everything was actually ready?
For example I think that VS 2008 doesn't actually runs faster with /nosplash switch, but it certainly feels that way.