Windows Performance Blog

More OpenGL...

mayers — Mon, 23 Apr 2007 20:53:00 GMT

Just a quick follow up on Prashant & Michael's OpenGL post, I'd like to point readers to the OpenGL Pipeline newsletter. The Khronos OpenGL ARG Working Group has written a nice article addressing concerns about the performance of OpenGL in Vista:

In the article, they reach three conclusions:

Windows Vista fully supports hardware accelerated OpenGL;
OpenGL applications can benefit from Window Vista•s improved graphics resource management;
OpenGL performance on Windows Vista is extremely competitive with the performance on Windows XP.

We've put a lot of work into optimizing the Vista experience and it's great to see that confirmed by the computing community.

-M

OpenGL and Windows Vista

mayers — Thu, 05 Apr 2007 00:20:00 GMT

Good afternoon,

One area that I get asked about quite a bit is graphics in Windows Vista - most recently, OpenGL. Despite having a graphics background, I'm not too current wrt Vista. To get more info about OGL in Windows Vista, I asked Michael, a PM on the graphics team, and Prashant an analyst who focuses on graphics perf, what the real scoop was. Here's what they had to say:

We get a lot of questions along the lines of “Why did Microsoft drop OpenGL support from Windows Vista?” that leave us scratching our heads. Microsoft doesn’t implement hardware-accelerated OpenGL directly – we offer a mechanism that allows hardware vendors to integrate a hardware-accelerated implementation of OpenGL (called an ICD or installable client driver) that utilizes their hardware into Windows. This has been the case since Windows 2000, and hasn’t changed much in Windows Vista. These ICDs are not included inbox with Windows, and are installed when you get the latest driver package either off a hardware vendor’s website or pre-installed in an OEM machine. The major difference this time around was WDDM, which required that all display drivers be re-written for Windows Vista. This also meant that the OpenGL implementations be re-written, which is something that hardware vendors are continuing to work on as we speak to get the best performance.

Another thing that has left a lot of people confused has been around OpenGL applications and how they work with the new desktop composition system, called DWM. DWM is implemented using Direct3D 9, and as such it was originally thought that OpenGL applications could not interoperate with DWM and DWM would need to shut down in the presence of an OpenGL application. This is not the case. Windows Vista provides a mechanism for hardware vendors to use to integrate an OpenGL application with DWM, which acts in the exact same manner as D3D9 and GDI integration with DWM via shared surfaces (a new feature of WDDM).

'Till next time,

-M

Measuring Performance in Windows Vista (configuring the system)

mayers — Tue, 30 Jan 2007 22:58:00 GMT

Ok, so it's been a long 'couple of days' since my last post but I haven't disappeared, completely. I'm back from vacation, caught up on email, and starting back into writing about Windows performance. One thing I've noticed over the past week or so is that with the pending General Availability of Windows Vista, we're starting to see more benchmarking of our new OS.

One in particular, over at Tom's Hardware, caught my eye; as a good example of how to do measurement of the OS. Overall, Tom's found that the OS performed pretty well at an application level and that some of the intensive 3D graphics operations weren't quite there, yet. What impressed me most was their methodology. They really understand that some of the changes that we made in Windows Vista to help the user can really wreak havoc with benchmarks.

To quote their article:

Knowing that Windows Vista has its SuperFetch feature, it is important to set up your test system to receive maximum performance that is reproducible... Vista learned about our preferred applications: Microsoft Office Outlook launched noticeably faster, and Skype launched almost instantly ...they are available much more quickly by relocating frequently accessed files from the slow hard drive into the quicker main memory.

Tom's understands that Windows Vista adapts to the user so, to make sure that their results were reproducible, they followed a checklist to ensure maximum reproducibility of the results. We've been working with several partners on how to accurately benchmark Windows Vista and as a result have developed a set of guidelines that we use for our own in-house methodology. These aren't necessarily recommended for everyday use but will ensure that your benchmarks run smoothly. We'll be publishing these guidelines as a whitepaper (which is currently in editing) but I'd like to give the blog readers a preview.

The easiest way to approach Microsoft's recommendations by breaking them down into three groups - configuring the system, preparing the system and preparing the workload. Today I'll discuss bits involved in configuring the system:

Prepare graphics

We recommend turning off animations (of windows, menus, dialog boxes, etc) that introduce artificial delays. I love watching a menu slide down into place instead of popping up onto the screen but, in a benchmarking scenario, these animations could impact accurate measurement. This is especially true if you are timing a sequence of events. Although they improve user experience, animations may erroneously impact your responsiveness measurements.

One place where Tom's differs from our methodolgy was in using Aero Glass. At Microsoft, we believe that glass is efficient and, recommend leaving all Aero features (except animation) at the system default settings. If the system enables compositing, keep it on. The OS wants to enable transparency? Then by all means, you should run a benchmark with transparency turned on.

Prepare User Account Control

If your workload requires elevation, then UAC prompts can really create some hurdles. Put bluntly, UAC is not friendly to scripting (imagine if you could script clicking ‘Accept' on an elevation dialog - shudder). This promotes good security but, again, makes for benchmarking headaches.

We don't recomend completely disabling the feature - just the prompts. We recommend going into the security policy manager and configuring the system to automatically accept UAC prompts for Admin group users. Thus, the user runs as a normal user most of the time, elevation still happens but the process is automated. We want benchmarks to measure any perf impact of the feature (which we believe to be negligible :) ) but not be hampered by the prompts.

Turn off system restore

This one is really your call - Windows Vista has the ability to roll files back to previous versions which has saved my clumsy bacon on more than one occasion. Although the impact on standard users is minimal, the work that system restore does in the background may affect the repeatability of benchmarks(this really depends on the amount of disk activity). The bottom line on this one is: if you disabled it for your Windows XP benchmarking, go ahead and do the same for Windows Vista.

Well, I hope that this has been helpful in both understanding the steps that we take at on the perf team to prepare a system for benchmarking; we recommend doing the same for your own measurements. My next post will look at how to prepare a system for actual benchmarking runs.

‘till next time,
-M

The results are in

mayers — Thu, 14 Dec 2006 12:32:00 GMT

As I said in yesterday’s introduction, my job as an engineer on the Windows Vista team is to improve performance. I wanted to look at a study that measure a key area that we focused on for Windows Vista – consistent responsiveness during the times that matter most to users (when starting up their machine, after being idle, and when you are under the gun running tons of apps, etc.).

To objectively measure how we did, I’ve been working with a company named Principled Technologies. If you’ve been involved in the (admittedly somewhat niche) specialty of perf testing over the last decade you likely know the people if not the company. We commissioned Principled Technologies to develop, run, and document the results of a set of tests that compare the performance of Windows Vista RTM and Windows XP on common business tasks. Today they published their findings here. I am, of course, really excited by the results.

Now you should read the whole report, but I wanted to talk a bit about their key findings:

l “Windows Vista was noticeably more responsive after rebooting than Windows XP on several common business operations.”

As I alluded to, yesterday, superfetch is the key driver behind this. I want to point out, though, that 'after rebooting' can be seen as a proxy for lots of cold operations. Rebooting is just the easiest to reliably measure. The second bullet is:

l “Overall, Windows Vista and Windows XP were roughly equally responsive on most test operations. Windows Vista was more responsive on some operations, and on those operations on which it was more responsive, Windows XP typically responded only a half a second or so faster.”

This is great, especially since Windows Vista is doing considerably more out of the box (e.g. UAC, Defender, search indexing, etc.). One of the most interesting bits for me was their 3rd highlight... you can run Aero without guilt!

l “Windows Vista Aero had little effect on the responsiveness of Windows Vista. Over 95 percent of the response-time differences between tests we ran with and without Vista Aero were under a tenth of a second, and all of the differences were under one second.”

We put quite a bit of effort into making sure that the new visuals were as efficent as possible and it really paid off.

For the truly technical, as you would expect, the report lists exactly how PT developed and ran these tests. The short answer is that they used a range of machines (laptops and desktops, 512M to 2GB, mix of graphics cards & processors, high-end and bare minimum, etc). I encourage you to dig into the report to learn more about the perf of Windows Vista compared to XPSP2. All in all, we were more consistent - better on cold and still doing well on warm; users can come to their machine and begin working, regardless of what state the box is in.

Anyhow, in the next couple of posts I think that I'll focus on some of the things that you will run into when designing and running a performance evaluation like this one. For example, one thing that PT did that I think is important is that they ran the same workload three times on each system before beginning their timed runs. This put the system into a quiescent state by allowing SuperFetch to learn and tune itself for the work it would be facing - similar to what it does in the wild, for real users. This is important to consider because otherwise you will get weird data that is much less repeatable.

Let me also take a second to note that a lot of the advice I am going to be giving in these next posts will be targeted specifically at performance benchmarking. I am not talking about how to maximize the performance of the everyday system (we'll do that in later posts). Perf tests are almost always automated to ensure consistency and repeatability. So I’ll focus on the benchmarking impact of some key features in Windows Vista (such as SuperFetch and UAC) which may not generalize to all situations.

Well, now that we've gotten all that straight, go read the report, and come back in a couple of days. In my next post I'll start talking about preparing a system for accurate, repeatable benchmarking.

-M

Greetings

mayers — Wed, 13 Dec 2006 22:01:00 GMT

Hi, all & welcome to the Windows Performance Blog. My name is Matt Ayers and I’m a Program Manager on the Windows Client Performance team. I occasionally write about Windows but I always seem to fall into the ‘always a blogs-maid, never a blogger' category – most recently, I borrowed some space from Tom Archer to talk about ReadyBoost. Well... we've decided to create a perf team blog to talk about key features, technical analysis and other topics near & dear to our hearts. We'll try and have a new topic every few weeks - if there's something you'd like us to talk about, please mail wprfblog@microsoft.com.

Anyhow, to jump right in, let’s chat about one of our major goals for Windows Vista: consistent responsiveness. General customer feedback about XP is that the OS performs pretty well, most of the time… but not necessarily all the time. We found that this inconsistent performance stems from having the wrong data is in memory; put another way, the memory is in a cold state. The most well known cold state occurs right after you start the system but there are many other ways that the system can get ‘cold’.

Another common example would be switching back to an application, such as a web browser after running several other applications or even a single application with a large data set like a video editor or a 3D game. If the machine doesn’t have enough memory to hold all of the other applications plus the browser, then some of the code and data for it will be re-loaded from of disk. For Windows Vista we focused on eliminating more cases where the system performed inconsistently. This is a continuation of work we did in XP which improved application launches.

Our innovations in memory management deliver the core of these improvements. In Windows Vista, SuperFetch* pays attention to what data the user needs and when they need it. Based on this history, it identifies data that are likely to be most useful to the user and proactively places them in memory before the user actually requests them. Additionally, SuperFetch will use this knowledge to keep important pages in memory, even if they haven’t been used for a while.

Anyhow, a detailed discussion of SuperFetch will be better served by its own post in the future – for now, I want to use it as an example of a feature that we use to improve the responsiveness of Vista. SuperFetch, along with ReadyBoost, low-priority I/O and loads of analysis has allowed us to add new features to Vista and improve the overall responsiveness of the OS.

Over the next few weeks, I’ll discuss these features, as well as the best way to measure their impact on the OS. So thanks & stay tuned.

-M

*trivia: this is the ‘mystery feature’ referred to in one of Richard Russell's 'pigs can fly' posts