Windows Performance Blog : benchmarking

Measuring Performance in Windows Vista (configuring the system)

mayers — Tue, 30 Jan 2007 22:58:00 GMT

Ok, so it's been a long 'couple of days' since my last post but I haven't disappeared, completely. I'm back from vacation, caught up on email, and starting back into writing about Windows performance. One thing I've noticed over the past week or so is that with the pending General Availability of Windows Vista, we're starting to see more benchmarking of our new OS.

One in particular, over at Tom's Hardware, caught my eye; as a good example of how to do measurement of the OS. Overall, Tom's found that the OS performed pretty well at an application level and that some of the intensive 3D graphics operations weren't quite there, yet. What impressed me most was their methodology. They really understand that some of the changes that we made in Windows Vista to help the user can really wreak havoc with benchmarks.

To quote their article:

Knowing that Windows Vista has its SuperFetch feature, it is important to set up your test system to receive maximum performance that is reproducible... Vista learned about our preferred applications: Microsoft Office Outlook launched noticeably faster, and Skype launched almost instantly ...they are available much more quickly by relocating frequently accessed files from the slow hard drive into the quicker main memory.

Tom's understands that Windows Vista adapts to the user so, to make sure that their results were reproducible, they followed a checklist to ensure maximum reproducibility of the results. We've been working with several partners on how to accurately benchmark Windows Vista and as a result have developed a set of guidelines that we use for our own in-house methodology. These aren't necessarily recommended for everyday use but will ensure that your benchmarks run smoothly. We'll be publishing these guidelines as a whitepaper (which is currently in editing) but I'd like to give the blog readers a preview.

The easiest way to approach Microsoft's recommendations by breaking them down into three groups - configuring the system, preparing the system and preparing the workload. Today I'll discuss bits involved in configuring the system:

Prepare graphics

We recommend turning off animations (of windows, menus, dialog boxes, etc) that introduce artificial delays. I love watching a menu slide down into place instead of popping up onto the screen but, in a benchmarking scenario, these animations could impact accurate measurement. This is especially true if you are timing a sequence of events. Although they improve user experience, animations may erroneously impact your responsiveness measurements.

One place where Tom's differs from our methodolgy was in using Aero Glass. At Microsoft, we believe that glass is efficient and, recommend leaving all Aero features (except animation) at the system default settings. If the system enables compositing, keep it on. The OS wants to enable transparency? Then by all means, you should run a benchmark with transparency turned on.

Prepare User Account Control

If your workload requires elevation, then UAC prompts can really create some hurdles. Put bluntly, UAC is not friendly to scripting (imagine if you could script clicking ‘Accept' on an elevation dialog - shudder). This promotes good security but, again, makes for benchmarking headaches.

We don't recomend completely disabling the feature - just the prompts. We recommend going into the security policy manager and configuring the system to automatically accept UAC prompts for Admin group users. Thus, the user runs as a normal user most of the time, elevation still happens but the process is automated. We want benchmarks to measure any perf impact of the feature (which we believe to be negligible :) ) but not be hampered by the prompts.

Turn off system restore

This one is really your call - Windows Vista has the ability to roll files back to previous versions which has saved my clumsy bacon on more than one occasion. Although the impact on standard users is minimal, the work that system restore does in the background may affect the repeatability of benchmarks(this really depends on the amount of disk activity). The bottom line on this one is: if you disabled it for your Windows XP benchmarking, go ahead and do the same for Windows Vista.

Well, I hope that this has been helpful in both understanding the steps that we take at on the perf team to prepare a system for benchmarking; we recommend doing the same for your own measurements. My next post will look at how to prepare a system for actual benchmarking runs.

‘till next time,
-M

The results are in

mayers — Thu, 14 Dec 2006 12:32:00 GMT

As I said in yesterday’s introduction, my job as an engineer on the Windows Vista team is to improve performance. I wanted to look at a study that measure a key area that we focused on for Windows Vista – consistent responsiveness during the times that matter most to users (when starting up their machine, after being idle, and when you are under the gun running tons of apps, etc.).

To objectively measure how we did, I’ve been working with a company named Principled Technologies. If you’ve been involved in the (admittedly somewhat niche) specialty of perf testing over the last decade you likely know the people if not the company. We commissioned Principled Technologies to develop, run, and document the results of a set of tests that compare the performance of Windows Vista RTM and Windows XP on common business tasks. Today they published their findings here. I am, of course, really excited by the results.

Now you should read the whole report, but I wanted to talk a bit about their key findings:

l “Windows Vista was noticeably more responsive after rebooting than Windows XP on several common business operations.”

As I alluded to, yesterday, superfetch is the key driver behind this. I want to point out, though, that 'after rebooting' can be seen as a proxy for lots of cold operations. Rebooting is just the easiest to reliably measure. The second bullet is:

l “Overall, Windows Vista and Windows XP were roughly equally responsive on most test operations. Windows Vista was more responsive on some operations, and on those operations on which it was more responsive, Windows XP typically responded only a half a second or so faster.”

This is great, especially since Windows Vista is doing considerably more out of the box (e.g. UAC, Defender, search indexing, etc.). One of the most interesting bits for me was their 3rd highlight... you can run Aero without guilt!

l “Windows Vista Aero had little effect on the responsiveness of Windows Vista. Over 95 percent of the response-time differences between tests we ran with and without Vista Aero were under a tenth of a second, and all of the differences were under one second.”

We put quite a bit of effort into making sure that the new visuals were as efficent as possible and it really paid off.

For the truly technical, as you would expect, the report lists exactly how PT developed and ran these tests. The short answer is that they used a range of machines (laptops and desktops, 512M to 2GB, mix of graphics cards & processors, high-end and bare minimum, etc). I encourage you to dig into the report to learn more about the perf of Windows Vista compared to XPSP2. All in all, we were more consistent - better on cold and still doing well on warm; users can come to their machine and begin working, regardless of what state the box is in.

Anyhow, in the next couple of posts I think that I'll focus on some of the things that you will run into when designing and running a performance evaluation like this one. For example, one thing that PT did that I think is important is that they ran the same workload three times on each system before beginning their timed runs. This put the system into a quiescent state by allowing SuperFetch to learn and tune itself for the work it would be facing - similar to what it does in the wild, for real users. This is important to consider because otherwise you will get weird data that is much less repeatable.

Let me also take a second to note that a lot of the advice I am going to be giving in these next posts will be targeted specifically at performance benchmarking. I am not talking about how to maximize the performance of the everyday system (we'll do that in later posts). Perf tests are almost always automated to ensure consistency and repeatability. So I’ll focus on the benchmarking impact of some key features in Windows Vista (such as SuperFetch and UAC) which may not generalize to all situations.

Well, now that we've gotten all that straight, go read the report, and come back in a couple of days. In my next post I'll start talking about preparing a system for accurate, repeatable benchmarking.

-M