scarroll's Blog

Hey all,

Marc (the QA lead) and myself (the Dev lead) of the VSTS profiler are sitting in front of Booth Dev-2 (Visual Studio Team System) in the Technical Learning Center.

The TLC should be opening up soon to customers, so I'd like to encourage you all to drop by.  We'll be happy to help you profile your app or answer any questions you may have.

Also, mark your calendars as Marc and I will be delivering a Breakout Session entitled DEV313 - Improving Code Performance with Visual Studio Team System.  We'll be covering a little bit of education about how to interpret the data coming out of the profiler and giving LOTS of demos of great new profiler scenarios in the next version including:

Web/Load Test integration

Comparing two performance reports

Filtering

Line level sampling

and many more.

Sawatdee Khrup!

So I arrived in Thailand for Visual Studio Launch which will be in BecTero Hall here in Bangkok on Tuesday morning (tomorrow as I write this).  I hope to see all of our Thai developers there.  I spent the morning getting ready for a presentation and waiting for the torrential downpour to finally stop.  When it finally stopped, I went and checked out Pan Tip Plaza, the giant IT shopping complex and I wandered around the nearby shopping districts until I couldn't stand the heat anymore.  It is 93 degrees here... and this is the cold season! 

Anyways, let's talk food.  I had a great meal tonight at a place called Benjarong.  They specialize in Thai Royal Cuisine and so I had to check it out.  I got the set menu.  9 courses, $20, and more food than any one person could actually eat.  So anyways, the second course was this amazingly tasty Tom Yum Gung.   Really good stuff, with these giant cloves of garlic and what looked like little green beans floating in it.  I bite into the "green bean".  The world flashes white.  Extreme agony in my lips and tongue.  I am on fire.  I actually yelp.  It's actually a tiny green chili.  A tiny green chili... of DEATH!  No matter what you read in books, in situations like this if there is a glass of ice water in front of you, you are going to drink that glass of ice water.  And of course, the books will be right and this will make it worse.  And now you are really in trouble.

The wait staff notices my distress and (bless their hearts) bring me ice cream to ease my pain.  Remember that this is only course 2 of 9.  Lest any of you think I am some lightweight making a big deal out of nothing, I will tell you that my wife is Thai.  When we dine with my in-laws, my mother in law actually makes 5 different types of fish sauce for putting on each dish to control the spice level.  The pecking order is:

Mama Bear (my mother in law can eat fire.)
Papa Bear (fire with a splash of lime.)
Steve (hot enough to earn the respect of my mother in law)
Sister Bear (my wife, medium)
Brother Bear (my brother in law, a total wuss.)

Anyways, the ice cream did its job and thank god because there was some amazing dishes still to come although my lips are still in a little bit of pain.  For future reference, the chili was a Prik Kee Nuu Suan.  Beware!

I have to get back to working on these slides.  Hopefully I'll have some good customer stories tomorrow.  Come out and see us tomorrow at the launch event.  I've been practicing the little bit of Thai I've picked up from my family. :) (Aahaan thai phet khrup!)

UPDATE: msdn2 seems to be flaking on this... I stuck a msn cached version below.

 

I just noticed that the Visual Studio Profiler docs are up on msdn2. 

Here's a good list of what each column means in English.

http://msdn2.microsoft.com/en-us/library/ms148209

 

http://cc.msnscache.com/cache.aspx?q=2268729578875&lang=en-US&FORM=CVRE10

Here's another fun one from the internal support alias.

When you profile with the VS Profiler, you are profiling the Release version of your code (I hope!). As you know from having debugged Release code in the past, sometimes things get a little funky. Well, a good rule of thumb is that anything that is funky in the debugger, is also a little funky in the profiler. After all, we share the same symbol information (i.e., the stuff in the PDBs for native code and the managed metadata in the EXEs and PDBs for managed code.)

One optimization that is particularly tough for the profiler is called COMDAT folding. There's a good explanation of this from a debugger standpoint here: http://blogs.msdn.com/oldnewthing/archive/2005/03/22/400373.aspx

The basic concept is that at link time, all identical functions at code generation time are folded in to a single copy of that function. This is particularly useful for limiting the code bloat in template code. You will see COMDAT in action with things like default destructors and simple methods like field accessors. Since there is only one copy of that function in the binary with several source level functions associated with it, if we take a sample in it, there is no way for us to know which is the correct function. So we just pick the first one in the list, just like the debugger.

Hope that helps!

Online registration for the Meltdown DirectX conference ends tomorrow July 22. I tried my best to target this talk to the Game Developer. If you are a grizzled veteran perf guy, I'm going to show you how to get the most out of our tool with advanced features and scenarios. If you are just getting in to perf work, I'm going to show you how easy it is to get started with the profiler wizards. The talk is at 4pm on both days of the conference for you late risers. ;-) If you have any profiler issues, I'm gonna be around to try to help you out. Remember, we are just about to start V2 planning in earnest, so if you have a feature request or feedback, this is a great time to get your voice heard. Here's the official abstract: Visual Studio 2005 Team Edition for Software Developers: Profiler Tips and Tricks Speaker: Steve Carroll (Bio) This talk will give an in-depth discussion of profiling techniques using the new Visual Studio profiler available in Team System. The talk will cover how to choose the right profiling method for your application including the tradeoffs between sampling and instrumentation profilers. Also, we’ll talk about each of these profiling techniques works. Common tips and tricks will also be presented as well as common mistakes we’ve seen in the field. We will also drill down on what all the data that the profiler produces means and how to interpret it. Advanced techniques for “off-roading” with our command line tools in advanced scenarios will also be presented. We’ll wrap up with a demonstration of using the profiler on a DirectX application.

The most common reason for this to occur in Instrumentation mode is that the dll you instrumented is appearing in two different processes. We don't aggregate data from multiple processes. You can see this easily by using vsperfreport.exe -summary:function foo.vsp. If you get foo_1234_FunctionSummary.CSV and foo_5678_FunctionSummary.CSV, that just means your instrumented dlls were loaded in 2 processes with IDs of 1234 and 5678. In the IDE, you can add the Process ID column to verify that the two copies of the function have different IDs. This can happen easily in ASP.Net if you crash the w3wp.exe process while the monitor is still running. A second w3wp.exe process will start up with a new process ID. It also happens if you have multiple worker processes on a machine running the same app.

I'm speaking at the Microsoft Meltdown 2005 conference at the end of the month. The topic is tips and tricks for using the Visual Studio Profiler. Since this is a game developer audience, I'm going to try to tailor this talk to your needs and go in to how to do some more advanced stuff with the API and the command line tools.

If you've got something you really want to see covered in this talk, now's the time to post a comment or drop me some mail.

When we get closer I'll put a quick talk outline so you can smack me down if I am talking about stuff you don't care about.

Hey folks.  We have a public chat scheduled for tomorrow.  I just checked the schedule and we have a bunch of the devs, testers, PMs, who are responsible for the dev and test skus of Team System signed up to be on for tomorrow's chat.  If you've never tried it before I think this would be a good time to test the waters.  If you have any burning question about our technology or even if you just want to yell us (constructively), this will be a great time to get your voices heard.  Of course, we read every single bug and suggestion and forum post that customers send us, but I think the chat format is more of a dialogue, since you can more easily ask a real person followup questions.

Team System Developer & Test SKUs Public Chat

Every 3^rd Wednesday of the Month @ 10am PST

Join us to discuss the Profiler, Test Tools (Unit, Generic, Manual), Web Load Testing, and Code Analysis (FxCop & PREFast).  We have questions for you, will answer questions from you, and will chat about the exciting new technology.

http://msdn.microsoft.com/chats

 

 

My illustrious co-worker, Ian, is compiling a central resource of all the most essential profiler blog posts.

I had to help a high school kid solve this problem yesterday.  It’s pretty interesting I think.
I sent it around the team and people seemed to enjoy it so here you go.  (In the old days, I would have saved this puzzle for an interview question, but pure puzzle questions are discouraged now.)

A census taker knocks on Mary’s door. 
Census Taker: I need to know the age of your three children
Mary: The product of their ages is 36.
Census Taker: hrmmm
Mary:  The sum of their ages is the same as my house number.
Census Taker: I still don’t have enough information.
Mary: The younger two are twins
Census Taker: Ok, I’ve got it now.

What are the ages of Mary’s children?

ANSWER IS NOW IN THE COMMENTS (along with a afterschool special style wrapup of what we've learned).

Total tangent here, but I just made a really great meal and I must blog about it.

Side Dish: Roasted Grape Tomatoes

Directions:
1) Grab a pint or so of grape tomatoes.  They have them at Costco out here, but I've seen them in supermarkets.  In a pinch, use cherry tomatoes. 
2) Brush liberally with extra virgin olive oil
3) Add salt.  I prefer Kosher for grilling for the texture.  Add enough so you can see it.
4) Add black pepper.
5) Sprinkle with thyme.  Fresh is best, of course.  It should stick to the olive oil
6) Grill on medium low in a grill basket about 10 minutes.

Main Dish: Spencer Steaks with Garlic Tarragon Butter Sauce

Directions:
1) RIGHT NOW (or at least a few hours before you are serving) take out a stick of butter and leave it to soft.
2) Add minced garlic, salt, and minced tarragon to the butter.  You can substitute thyme in a pinch.  (I have a TON of tarragon growing here.)
3) mash up with a fork until well blended.
4) Spoon out onto wax paper.  Roll up the wax paper to form a log.
5) Put in the refrigerator to harden.  Note: you can use this garlic tarragon butter in anything from mashed potatoes to stirfrys so make plenty of it.  It should freeze well.
6) Grill up the steaks.  We used Spencer Steaks from Larry's Market.
7) When the steaks are done, put a pat of the butter on it.  Tent with aluminum foil and wait until it melts.

Compound butter is super easy and versatile. We also made some chipotle butter (mince the chipotles in adobo, and follow the directions above) for my wife's mashed potatoes.  Good stuff.  It really takes your steak grilling to the next level.

Ok.  Thanks for your indulgence.  I promise I'll get back to writing about perf now!

The symbols for several key CLR Beta2 Symbols are finally on the symbol server.  Go forth and profile!

If you had previously sampled some managed apps and were disappointed with the stacks, please give it another try.  Unfortunately, the DLLs that had their symbols missing were pretty crucial to actually seeing what was going on in your managed app.

Thanks!

Sorry folks.  I want to update this blog about this issue.  We just made another symbol publication attempt to the symbol server of the CLR Symbols.   We hope this one will go through.   Just to sum up:  The symbols distributed with the sdk for mscorwks, mscorlib, mscoree, and mscorjit are corrupted and will not be loaded by the debugger or profiler.    There are no symbols on the external symbol server either.

Several attempts have been made to update them, but each has failed.  I went to a meeting yesterday where a single person was assigned to drive the resolution of this  issue.  I will update this space as soon as the symbols are fixed.  I am really sorry about this inconvienence.  All I can say is that now there is a single person responsible for symbol publication and we will make sure not to make this mistake again at release.

I will update this space as soon as I get word that the symbols are live. 

A lot of my graduate work was in the area of parallel computing so we spent a lot of time with Amdahl's law.  Amdahl's law states that if you can speed up some fraction F of the program by some speedup S, the overall speedup of the program is:

1 / [ ( 1 - F ) + ( F / S ) ]

So if you can speed up half the program by 8x, F = 0.5 and S = 8, the overall speed up is only about 1.8x.

By the same token, if you profile a program and find that you can really speed up a part of your program that is only 5% of inclusive time, the best possible speedup is 1.05x.  The moral of the story of course is that you should focus your effort on optimizations that will affect the greatest possible portion of the program.

Which leads me to the question of really long sampling runs.  One thing we've noticed about our users is that they tend to run their app for a really long time under the sampler.  I'm not sure I understand this technique.  You can get a statistically significant number of samples pretty quickly.  Remember that you can use vsperfcmd with the -timer option to set the sampling frequency but even with the default on most processors you are going to get several thousand samples even in a few seconds.  That should be more than enough to tell you where the hot section of your program is.  It doesn't really matter if you get enough samples to tell if some part of your code is 0.5% of your inclusive time or if it's actually 0.7% because Amdahl says you should be ignoring it anyways.  Likewise, you won't care if another function is 50% or 53% of samples because you know that's the bit that needs to be fixed.

I asked RicoM about this a few days ago and he told me that he brings people in to the MSFT developers labs, warms their apps up and then just takes about 15 seconds worth of samples and its usually enough to find where all the allocations / time is coming from.  One possible reason that he suggested where this might be a useful scenario is when you are trying to figure out why an app runs well in the first hour but then slows down in the second hour, so you might want a longer run to try to catch whatever changes.

But it's certainly possible that I'm missing something...  Does anyone have a good reason to do longer runs for everyday sampling jobs?

There was a problem with the original publication of the CLR Symbols for some binaries (mscoree, mscorwks, maybe others).  This was causing the profiler to be unable to load symbols for these binaries.  The end result was a lot of [mscoree.dll] function names in all profiling reports.  I have been informed that these should be up on the public symbol servers very soon (today or tomorrow).

Thanks for your patience.

 

UPDATE!! This is still broken.  More to come....

SORRY!!