Everything you want to know about Visual Studio ALM and Farming
Brian Harry is a Microsoft Technical Fellow working as the Product Unit Manager for Team Foundation Server. Learn more about Brian.
More videos »
Someone asked me the other day what we were using as our “Real World” apps so they could compare them to their own apps to understand how the testing we are doing will apply. Here’s a high level overview that characterizes several of them. There are others that I haven’t listed but this should give you a decent flavor. Many of these came from actual external customers and the names have been changed to protect the innocent :)
I’m not trying to tell you that no one has any apps bigger than this but we believe these represent a pretty good picture of a range of largish real world apps across a variety of technologies.
Brian
Wouldn't it make sense to perf test by *stressing* VS rather than targeting typical largish real world apps, especially since your target memory usage is a very generous 1.5GB?
I'm currently developing on a codebase with 80 c# projects and 2700+ source files.
1.5G VM space is not 'very generous' given that on B2 even a very small mixed C++/C# solution can be >1G VM with the right (wrong) extensions running.
I think Brian made the point several posts ago that in fact they'd found that targetted 'stressing' had not, in fact, led to good real-world performance - I suspect it mainly led to a great deal of ego deflation the week after B2 shipped.
The current effort seems to be on getting real people to capture real profile data on their own machines, which is big a step forward if it's done widely.
The current effort seems to be targeting "adequate" performance rather than the "terrible" performance of B2.
Being that stellar performance was actually a supposed be a feature of VS2010 they should take the problematic real world test cases and multiply them by 2-10x, see where things collapse, and fix the problems.
Do you test solutions with mixed c#/vb projects and database projects (GDR)? I have very poor perf in vs2008 when loading a solution wich includes 2 or 3 dbprojects.
Sorry, I've been waiting for some data to reply to some of these. Brien, it sounds like you and I come from the same school of limit testing :) For years I have said the way to figure out what you should test up to is to imagine the most you think a reasonable user will do and multiply it by 10 and that's what you should test. People always vastly surpass your expectations.
The first time I learned this lesson was in about 1990 when I was working on one of the first commercial LAN based email programs. We concluded that no one would have more than several hundred email messages in their inbox (remember this was before spam and rampant use of email - most people had 10-20 mail messages :)). Within six months, we had customers calling and complaining because they had 3,000 or more messages in their inbox and it was slow :)
A few years later when I was doing SourceSafe, we testing our file compare algorithms up to about 100,000 lines (and felt pretty good about that). We have customers calling us complaining about perf of 500,000+ line files.
Hence my philosophy was formed. If you look at what I've published on TFS limit testing, you'll see that philosophy in practice: http://blogs.msdn.com/bharry/archive/2005/11/28/497666.aspx. And that was back in 2005 (V1). Most of our test goals are 10 times that now.
Anyway back to the point...
When I was gathering the data for this post, it was the first time I actually looked at how big the solutions are and as I was typing them, I was thinking - these are big but they aren't REALLY big. I wonder how big some of our internal solutions are that we are testing.
So I tracked down the numbers for Blend: 30 projects and over 3,500 files. That feels even better than what we have above. I'll track down Exchange too. I won't be surprised if it's even bigger.
However, to support what Will said, I do believe there is a lot of value in using a real project. Using really large fictitious projects could very well mislead us. One of the advantages of using real projects is real development teams can actually use the product against them for an extended period of time and get a feel for the perf beyond just some numerical analysis.
I think enhancing our testing to really probe the limits of what VS can do would be a good idea but it still won't substitute for real apps.
amselem,
Yes, we do testing with lots of mixed projects. I don't think any of the big end to end apps we are using have DB projects in them but I know the DB team is testing them. In fact, yesterday, I just got the results from their latest testing and it's showing generally a 25%-75% improvement over their results for the GDR they shipped (about a year ago?) on most tests.
If you'd be willing to try out a new build and see how it works for you, please let me know and we'll get one for you.
Brian, it would be relevant to also provide the number of lines of code as defined as PDB seq point. For example:
NUnit 15K LoC,
Paint.NET 50K LoC,
NDepend 95K LoC
R# 0.5M LoC
.NET Fx v3.5 1.3M LoC
http://codebetter.com/blogs/patricksmacchia/archive/2007/10/03/how-do-you-count-your-number-of-lines-of-code-loc.aspx
http://codebetter.com/blogs/patricksmacchia/archive/2007/10/05/why-is-it-useful-to-count-the-number-of-lines-of-code-loc.aspx
There are indeed a lot of reasons to look at lines of code. I'm not convinced that understanding VS performance is one of them. I have no reason to believe that these apps are any different than what you would find in nature. Many of them were selected because they were known to be "problematic" for Beta2 (which is part of why we chose them).
The team is really busy tracking down performance bug reports, looking for optimization opportunities, etc. I just can't see asking them to go count the KLoC right now. Maybe at some point down the road we will. If there's another app you are particularly concerned about, please try it out. If you'd like a newer build than Beta 2, please let me know - I'm happy to get it for you.
I think it would be interesting to know what about a project impacts VS performance. Is it the number of files? lines? types? etc. Of course it's likely a combination of many factors but I expect the people who investigate VS performance issues to have some basic ideas about what's going on.
Most people typically don't just have one type of project in a solution. Hopefully you are testing many different types of projects as well, and also have many dependencies between them all, in a single solution. Your posts have really been interesting though.
Yes, we do test mixed solutions. The SW app in particluar is very mixed. If I remember correctly, it has ASP.NET, WPF, Silverlight and C++ projects in it, in addition to corresponding unit test projects.
Mike,
I don't know that I can give you a very good explanation of that, because the short anwer is almost everything can affect VS performance, but let me talk about the things that seem to have the MOST impact on VS performance.
Probably the biggest is how much stuff you have loaded at once - projects in a solution, editor windows open, tool windows open, designers, etc.
Another thing that has a big impact on designer performance is the complexity of the design. For example, very large work flows in the workflow designer, very large architectural models, very complex forms in Web, WPF, Silverlight, WinForms, MFC, etc.
Our switch to WPF and hardware acceleration has made hardware a bigger factor in performance than it has ever been before.
Size of the source code (lines of code) tends to mostly affect compiler performance and some of the symbol browser kinds of operations, but not a ton else. And it's usually pretty linear.
Those are probably the biggest things.
250 project, over 10,000 source + header files, almost all of it native C++. Now I really wish I could send it to you; the lack of native test cases has me very worried now.
Why in the world would we have this? Well, the code grew up in VC6 and it has a horribly tangled dependency graph. We have subset solutions for most development tasks but our production builds (and my work debugging build breaks) all happen against the "everything" solution because it's the only one that can properly encompass the dependency graph.
Yes, it's painful.