Engineering Windows 7 : Graphics

Engineering Changes to ClearType in Windows 7

e7blog — Tue, 23 Jun 2009 10:00:00 GMT

One of the many passions held by Bill Gates is a passion for reading and so his desire to make reading on PCs a fantastic experience has been an effort ongoing for many years. In the 1998 COMDEX show, Bill Gates unveiled ClearType – hard to believe it was that long ago. Back when it was announced, very few of us had LCD monitors and those that did invested several thousand dollars in one that was 15” and 1024x768 (today one like that costs less than $100). The notions of smoothing and anti-aliasing have been around for many years and are common in the world of typography, animation, and games. ClearType took this to new levels by building on the properties of LCD panels. ClearType was subsequently included in Windows XP and continues in Vista and Windows 7—each release saw changes in the underlying technology, the fonts that support the technology, and the APIs available to developers. It is fair to say that over the years we have learned that there are a set of customers who simply find ClearType rendered screens less than appealing and wish to turn it off. We recognize this and want to make sure we provide the appropriate controls. ClearType is also part of the Windows platform and provides APIs callable and controllable by developers of applications. There is a conventional view that ClearType is a "visual preference" and through this post we want to show how there are elements that are such a preference but there are also elements that are APIs used by applications, just like applications can choose fonts, colors, and other attributes as required. This post goes into the details of Windows 7’s implementation along with some history and background. Greg Hitchcock is the development lead on ClearType and has worked on it since the start. He’s also one of the most tenured members of the Windows 7 engineering team with only 6 folks having been at Microsoft longer -- Larry is one of them :-)!

--Steven

Based on feedback, we want to clarify how font rendering works in Windows 7 and give some background on how we chose ClearType font rendering to be the default in Windows. For those that dislike ClearType and want to change the system default setting to bi-level rendering, as were defaults in Windows Millennium, the quick answer is:

Enter Appearance into the start menu search
Select Adjust the appearance and performance of Windows from the Control Panel
The setting that should be changed under the custom option is: Smooth edges of screen fonts, which should be turned off

The longer answer, as we will describe in this post, shows that changing the default setting is not as “black and white” as it may seem. As you have noticed, Windows 7 also includes a new ClearType tuner in the control panel which affords fine-grained control over rendering—we’ll talk about that some below as well.

ClearType

ClearType is a technology developed to improve both the appearance of font rendering and reading performance on computer displays. As most people spend over 80% of their time on computers reading on the screen, improvements in this area greatly improve the overall experience of Windows. The ClearType technology has continued to evolve and the latest improvements have been made in Windows 7, as discussed in this earlier E7 post.

In simple terms, ClearType works by using the underlying geometry of colored sub-pixels in the display as if they were full pixels—gaining extra resolution while at the same time using principles of human vision to remove the perception of color artifacts. Further details on the technology and how it uses human visual perception are described here. More specifically, the ClearType technology is optimized for LCD panels with red, green, and blue (RGB) striped sub-pixels that are oriented vertically, although it performs reasonably well on CRT displays (especially those that are aperture grille based) and even LCD panels with horizontally oriented RGB stripes. Although this might seem counterintuitive, through informal studies, we’ve found that about 70% of users prefer ClearType even on these non-optimal displays. Of the 30% who preferred other rendering techniques, their biggest concern with ClearType in these non-optimal cases was the loss of text contrast.

Other Types of Font Rendering in Windows

Given the complex world of many display types and a wide variety of users and their visual systems, how did we go about implementing ClearType into Microsoft Windows? Microsoft did not rush headlong into making ClearType the default rendering. The technology was first released in 2000 with the Windows CE product. The Windows CE environment is usually quite controlled in terms of the hardware used, so it was quite easy to verify that ClearType worked properly on each device, and either tune ClearType or adjust the hardware to optimize the onscreen reading experience. The first release of ClearType on the Windows platform was with Windows XP in 2001.

Bi-Level Rendering

Example of Bi-Level rendering. Note if your browser scales this image the text will not be correctly represented.

Prior to Windows XP, two types of font rendering were supported in Windows. The first type of font rendering supported was bi-level rendering, more commonly referred to as “black and white” rendering, but sometimes also called aliased text. With bi-level rendering, two colors represent the font, the foreground color and the background color. This was the first type of rendering supported by TrueType when Windows 3.1 was released, and also the essential method of displaying fonts in bitmap form from Windows 1.0. Bi-level rendering, especially when generated from outline technologies like TrueType, is very difficult to optimize for low screen resolutions. Significant effort needs to be put into the font hinting for TrueType in order to get the best bi-level quality. It can reasonably take a skilled person 6 months to a year per font of hinting time to get this level of rendering detail. That would be multiplied by four for a four-member family. If the character sets are larger, as in some system fonts, it can take even longer.

Font Smoothing / Grayscale

The second form of rendering, known as font smoothing, became the default rendering in Windows 2000 and was first released as an option for Windows 95 in the Plus! Pack. Font smoothing is a hybrid grayscale anti-aliasing technique designed to improve the contrast of fonts over traditional anti-aliasing techniques. There are two factors that differentiate font smoothing from more traditional text anti-aliasing.

First, traditional anti-aliasing works by overscaling the font outline data and then downsampling. Font smoothing uses the same technique, except that it applies font hints prior to overscaling the outline data. Although we don’t have enough space here to fully describe font hinting, I can simplify it enough to say that it often uses a method called “grid fitting” to snap the vertical and horizontal edges of the font outlines so that they are aligned with the pixel grid. In this situation, most horizontal and vertical stems of a font, when overscaled, cover 100% of the underlying pixels, and when downsampled return the text foreground color, which is usually black. Diagonal and round features of the font will not have full coverage of the pixel and thus will return some shade of gray, reflecting the coverage of the underlying pixel. It should be noted that when text displays the “jaggies” (or more formally aliasing) this usually occurs on round or diagonal parts of the glyphs—exactly the areas receiving gray coverage with this method. This way of anti-aliasing is beneficial due to the higher contrast of the stems in the font at a slight cost of some spatial accuracy.

The second differentiating factor is that the fonts determine the exact size that the font smoothing turns on or off. Most fonts that provide this level of information turn on grayscale anti-aliasing below 9 pixels per em (PPEM.) That is the equivalent of 7 points on a 96 PPI screen. Above 9 PPEM, anti-aliasing is turned off until the main stems of the font are around two pixels wide, which is around 13 to 20 points, depending on the typeface. Once the main stems are two pixels wide, anti-aliasing remains on as the sizes increase. Two pixel wide stems are usually chosen because there is usually enough “backbone” of foreground colored pixels to keep the stem contrast high. If the font does not have specific sizes for font smoothing, system defaults are used. There are independent defaults for both regular and bold typefaces. So although font smoothing was the default, most fonts, when displaying text at typical reading sizes, would render them bi-level.

Defaults for Font Rendering

With the addition of ClearType to Windows XP, there were now three types of font rendering available (Bi-level, Font Smoothing, ClearType). During the time period that Windows XP was being developed there was a clear transition underway from desktop systems with CRT monitors to laptops and desktops with LCD displays. Since this transition was far from complete, we felt that the default value for font rendering in Windows XP should be grayscale font smoothing, the same as Windows 2000. OEM’s who provide Windows XP on their systems could change this default, and in fact, by the time Windows XP SP 2 shipped, many of them had set the default to ClearType. It should be noted that OEMs can always change these settings as part of configuring a PC.

In Windows Vista, the system’s default font rendering was changed to ClearType. It is important to clearly understand what is meant by default font rendering. In Windows, the default font rendering is the rendering used when the application does not choose an explicit type of rendering. Some have confused this default value to mean that all applications must use this value. This view is inconsistent with the way text APIs worked when introduced in Windows 95’s font smoothing. In GDI, the API for choosing the current font has the rendering type explicitly as input. It is expected that there are situations where the application knows best what type of rendering should be used. For example, in displaying a print preview page with small text, traditional font smoothing might be the best choice for rendering. Likewise, if an application was targeting on-screen reading, it might be best to use ClearType as the rendering for that application. In some situations, like remote terminal services, the application might choose to use bi-level rendering to reduce the bandwidth of text information that needs to be sent to a remote client.

There are many examples where applications decide one way or another to use rendering other than the system default—just as applications that choose to use different fonts, colors, sizes, or other text attributes. The most typical example is in applications that wish to have reproducible layout and flow of documents. By being specific about which way to render text, the applications can be certain of how text will flow across different PCs. Another common example, as mentioned above, is Print Preview where the ability to properly render representations of higher resolution output, particularly for small text sizes, is much improved. We recognize that for some it is counter-intuitive that an aspect viewed as a “display” property is something that applications can choose to “over-ride”. We’ve designed rendering so that the default case is to respect the setting, but applications, including Windows itself, might have elements that require explicit rendering techniques.

Although each application can make the choice on a per-font basis of which rendering to use, the majority of applications choose the default rendering. Therefore, making the decision to change the default for Windows Vista was not taken lightly. The trends in the hardware displays were strongly showing a rapid movement from CRTs to LCD-based displays as we have shown in earlier blog posts based on the Windows XP and Windows Vista real-world telemetry. Even though there were still CRTs in use, feedback from Windows XP customers was positive on the quality of ClearType rendering on CRTs. After we made the choice, the feedback on the decision to enable ClearType as the default for Windows Vista was overwhelmingly positive.

Even with the default rendering set to ClearType, there are some scenarios that can change the default. If an OEM is providing Windows on their hardware, they can change the default. In some situations, and this was more common with font smoothing in Windows 95, the hardware may not meet the minimum requirements for the rendering technology. In the case of both font smoothing and ClearType, a minimum of sixteen bits per pixel display resolution is required. (When rendering to an off-screen bitmap in GDI, it is important that it not be the default color depth of 1 bit per pixel if you desire to capture ClearType text.) In some cases, when optimizing for system performance, font smoothing (both ClearType and grayscale) can be disabled. In a similar fashion, using Remote Desktop to connect to another computer or session usually disables ClearType by default.

Changing the Default Rendering in Windows 7

Windows 7 maintains the same defaults as Windows Vista. There are several ways for the user to change the default values for font rendering in Windows 7. For those that want the default rendering to be bi-level, the user interface for this selection is in the performance section of the Control Panel. From the root of the control panel you can access it by selecting System and Security –> System –> Advanced System Settings –> Performance (Settings…). An easier way is to enter “Appearance” into the start menu search, and then select “Adjust the appearance and performance of Windows.” The setting that should be changed under the custom option is: Smooth edges of screen fonts, as shown in the figure.

The option of no font smoothing as the default value is considered to be an uncommon setting, so it is a little more difficult to find than other settings. If the user prefers to change the default font rendering selection to the Windows grayscaling anti-aliasing technique described earlier, in Windows 7 that is now done through the ClearType Tuner.

ClearType Tuner

The quality of the ClearType text can be optimized for you and your monitor. The ClearType Tuner is a new control panel component for Windows 7. Because there are differences in monitor characteristics and differences between readers’ eyes, there are font rendering options that can only be optimized by a reader looking at text on their monitor. The ClearType Tuner uses various samples of ClearType, presented in the form of an eye-test, to make fine grained adjustments to the ClearType algorithms. Each wizard page tunes a parameter such as monitor gamma (relationship between voltage and brightness), your sensitivity to color artifacts, and your preference for letter heaviness.

In order to switch between ClearType and grayscale, the setting “Turn on ClearType” on the opening page of the ClearType Tuner can be toggled.

Either way, the user is taken through the rest of the ClearType Tuning wizard for two reasons; if an application explicitly enables ClearType rendering, it is useful for that experience to be tuned, and some graphics platforms have more fine tuning of the rendering for both gray rendering and ClearType.

Font Design and Font Rendering

The availability of higher resolution font rendering techniques like ClearType has had a significant impact on the design of fonts for onscreen reading. From the early days of the printing press, as new technologies and printing styles were developed, typefaces were redesigned to take advantage of those technologies. For example, many typefaces still in use today incorporate “ink traps” into the design so that ink would not clog up key features of a glyph. This is an aspect of making specific design choices in the font in order to work the best with the technology. In traditional typeface design, the term font refers to a typeface at a given size. So a 10 point Times New Roman would be a different font from a 24 point Times New Roman. In the days of metal cast typography, each of these sizes were designed by a punch cutter to be optimized for the medium for which they were to be used, often with changes in stem contrast, x-height, or character spacing for a given size. The advent of photo typesetting in the mid-twentieth century was a step backwards in this regard, as it used one size as a type master, and then used optics to scale that master size to any other presentation size.

Microsoft Windows has taken the more traditional approach to digital outline fonts, and through a combination of font hinting and new typeface design we attempt to optimize each size for the medium for which they were intended. With Microsoft’s initial release of TrueType for Windows 3.1, the traditional typefaces Times New Roman, Arial, and Courier New were used as core fonts. In the creation of these fonts, one master size was chosen for the outline data, usually something around 10 or 12 point, and, similar to the technique used in photo-typesetting, the outlines could be scaled to any requested size for a given display resolution. But, going back to the more traditional ways, each size was carefully examined and changes were made to the basic design through font hinting—including changes to critical features like stem contrast, x-height, or glyph spacing. As earlier mentioned, hinting fonts to be tuned for a low-resolution medium like full pixels on a 96 PPI screen was very time consuming. To help in this process for Microsoft Windows, we commissioned or designed in-house new outline typefaces designs that were optimized for the world of 96 PPI bi-level rendering. These custom fonts include Tahoma, Verdana, Georgia, Trebuchet MS, and even Comic Sans MS. These fonts still needed to be hinted to tune the individual sizes, but because the typeface was designed with the medium in mind, it was a more straightforward process and less time consuming.

Even with typefaces tuned to the display medium, 96 PPI pixels on a screen are still larger than many of the features we’d like to show in a typeface—and that is where ClearType helps us. Therefore, with ClearType, it made sense to commission a new set of fonts that were optimized for this new medium. Now the existing fonts for Windows still work well with the technology, but this project was an attempt to get the very best design for onscreen reading using ClearType. This led to a new set of fonts that shipped and were tuned for Windows Vista. The ClearType Collection fonts of Calibri, Cambria, Consolas, Corbel, Candara, Constantia, the new user interface font Segoe UI, and the Japanese font Meiryo were designed for this medium. As part of the engineering work on these font projects along with the default setting of ClearType, we decided in the hinting process to do the fine, size-specific hinting only for ClearType, and not for bi-level rendering. This allowed us to focus our efforts on the fine levels of detail and quality for the vast majority of customers.

ClearType Fonts in Windows 7

A reasonable question for us to ask ourselves is what is the experience like in Windows 7 when bi-level or hybrid font smoothing is chosen as the default?

As mentioned earlier, not all applications will choose to render with the default settings. Microsoft Office and Internet Explorer will default in some cases to using ClearType rendering. Some applications that use fonts tuned for ClearType and not bi-level rendering may choose ClearType rendering to maintain the benefits of the font designs. Some applications need higher precision glyph widths like sub-pixel positioning or “natural width ClearType,” and would reflow if they were changed to bi-level or grayscale rendering. Other applications like Adobe Reader have their own built-in text rendering engine that is independent of the Windows graphics platforms. Likewise, platforms like Java on Windows also use their own rendering techniques.

In some situations with the Windows 7 Explorer, ClearType rendering will remain on so that Segoe UI will keep its optimal design. Changing the system font from Segoe UI to some other font could be problematic, leading to issues like reflowing dialog box entries, missing text due to wrapping, unlabeled buttons, etc. We know many would value global changes to the fonts used by Windows, but to maintain to reliably across resolutions, DPI, and languages to name a few issues means we cannot have total flexibility on the system font settings at this time.

Given the challenges of turning off ClearType, there are a few mitigations in the fonts to handle some scenarios where ClearType is not available. In the ClearType font Calibri, since it is the default font for Microsoft Office, an unusual technique was used to attempt to improve the quality of the font rendering when font smoothing grayscale was selected. In this case, as opposed to the normal situation where font smoothing was disabled at lower text sizes to remove the blur, at these lower sizes the font enabled grayscale in order to improve the character shape. Also, at a few key sizes, the Calibri font had some bitmap fonts embedded in the outline file. These bitmaps kick in when bi-level rendering is requested. These bitmaps were intended to handle the case where Calibri was being used in a Remote Terminal situation and the default for Remote Terminal was not set to ClearType for performance reasons.

ClearType Research on Performance

As mentioned earlier, one of the goals behind ClearType is to improve the performance of reading text on computer screens. We have supported several areas of research looking into measuring this work. The research is done at universities and published in peer-reviewed journals. We have another Microsoft blog, that among other things related to fonts, also describes some of the research work on reading performance. Since those blog entries give more detail and background, we’ll just describe some of the performance highlights.

We’ve measured an improvement in word recognition accuracy of 17% using ClearType over bi-level rendering.
We’ve found a 5% speed improvement in reading speed and a 2% improvement in comprehension (this is remarkable) using ClearType over bi-level rendering. A 5% reading speed improvement may sound small, but the cumulative effects can be huge given the amount of time people spend reading.
We’ve found the reading speed improvements of 5% continue over longer spans of text, and we’ve found that non-traditional reading tasks like document scanning are about 8% faster with ClearType over bi-level rendering.
We’ve found that reading sub-optimal text causes eye fatigue by increasing squinting and decreasing the blink rate. (This may seem obvious, but prior to this work there was no understanding of the physiological mechanisms of eye fatigue.)

ClearType Research on Rendering Preferences

Another research question we’ve asked ourselves is why do some people prefer bi-level rendering over ClearType? Is it due to hardware issues or is there some other attribute that we don’t understand about visual systems that is playing a role. This is an issue that has piqued our curiosity for some time. Our first attempt at looking further into this involved doing an informal and small-scale preference study in a community center near Microsoft. This was done with two identical laptops, one with ClearType and one with bi-level rendering. They were placed side by side and participants were asked which version they preferred. This was done with three different samples. Here were the results:

	Prefer ClearType	Prefer Bi-Level	No Preference
Sample 1	33	1	1
Sample 2	33	2	0
Sample 3	33	2	0
Average %	94%	5%	1%

Comments:

35 participants.
Comments for bi-level rendering:
Washed out; jiggly; sketchy; if this were a printer, I’d say it needed a new cartridge; fading out – esp. the numbers, I have to squint to read this, is it my glasses or it is me?; I can’t focus on this; broken up; have to strain to read; jointed.
Comments for ClearType:
More defined, Looks bold (several times), looks darker, clearer (4 times), looks like it’s a better computer screen (user suggested he’d pay $500 more for the better screen on a $2000 laptop), sort of more blue, solid, much easier to read (3 times), clean, crisp, I like it, shows up better, and my favorite: from an elderly woman who was rather put out that the question wasn’t harder: this seems so obvious (said with a sneer.)

Two other additional preference tests were performed with 28 of 30 participants preferring ClearType to bi-level rendering in one study and another with 52 of 55 participants preferring ClearType. Combining these three tests, we get 113 of 120 participants preferring ClearType rendering over bi-level rendering. It is important to note that in a forced preference test like this, just because someone preferred ClearType, it does not mean that they also don’t like bi-level rendering. It is just a preference towards ClearType.

Further examination of those who prefer bi-level rendering is of great interest to us and we continue to research this topic and to work with university researchers as well. We expect to see published papers on this topic in the future.

Future Research

Going forward, much of our research is in finding ways to make the highest quality text rendering more accessible to everyone. Each visual system has its own characteristics, and just as the ClearType tuner allows us to tune the algorithm for display characteristics, it would also be nice to tune for visual system characteristics. For example, in the United States 7% of the male population is color blind. We believe that we can improve the ClearType algorithm so that text for a colorblind reader is even better than for a reader without colorblindedness. Researching ways to improve text rendering for those with high color sensitivity and lower visual acuity would be just as important for us.

Conclusion

Making the screen the best possible place to read is an exciting opportunity for us. It blends the engineering challenges of working with many display technologies and human visual systems with the artistic challenge of creating a beautiful set of glyphs, where every subtle typographic nuance is important. In doing this, we need to keep in mind how the science of reading must guide us in making the experience optimal for us—humans. Each rendering technology has advantages and disadvantages for different people; depending on the application in use there are tradeoffs involved. Many of these issues go beyond the ability for people to easily discern choices. Our job is to work hard to provide a great platform for developers as well as tools that people can use to make choices and control how they use their technology. Our goal should be that the out-of-box experience just works. We think that, most of the time, we’ve accomplished this and we also recognize this area is complex and there is a wide spectrum of feedback.

The team at Microsoft working on these problems has been together since 1990, developing fonts and font-rendering solutions, and working to get a better understanding of the science of reading. The team is made up of engineers, type designers/artists, and psychologists and we work with many other experts throughout Microsoft in attempting to tackle this tough, yet vitally important task. You spend over 80% of the time at the computer reading, so it should be as pleasant an experience as possible. The following article from IEEE Spectrum describes some of the issues we deal with related to the technology, art, and science of text.

--Greg

Engineering Windows 7 Graphics Performance

e7blog — Sat, 25 Apr 2009 10:00:00 GMT

One of the areas of any release of Windows that receives a significant amount of testing and scrutiny is the performance of graphics—desktop graphics all the way to the most extreme CAD and game graphics. The amazing breadth of hardware supported for Windows and the broad spectrum of usage scenarios contributes to a vibrant ecosystem with many different goals—from just the basics to the highest frame rates on multiple monitors possible. In engineering Windows 7 we set out to improve the “real world” performance of graphics as well as continue to improve the most extreme elements of graphics. This is work we do in Windows 7 and work our partners do as they work to improve the underlying hardware/software combination through drivers (note: Windows Vista drivers continue to work as they did in Windows Vista, but we've also been working with partners on updated drivers for Windows 7 which many of you have been testing through Windows Update downloads). This post looks at this spectrum of engineering as well as the different ways performance is measured. Ultimately we want to inform you about what we have done in engineering Windows 7, while we leave room for the many forums that will compare and contrast Windows 7 on different hardware and in different scenarios. This post is written by Ameet Chitre, a program manager on our Desktop Graphics feature team. --Steven

If you have gone online to check out or purchase a new PC, you must have noticed that “faster graphics” and “great performance” are always some of the key selling points. People have come to expect faster systems which enable them to edit photos, do email, watch high-definition videos and play the latest 3D games all with greater ease, often shuffling between these tasks seamlessly. Quite a few of these users refer to the enthusiast community blogs and various review sites which run graphics benchmarks and report results evaluating how fast the graphics of new hardware or software performs. Traditionally graphics performance has been measured and analyzed through 3D games but it also impacts what we call “desktop scenarios” - such as when you are using the Windows UI, moving or maximizing windows, or scrolling within Word or IE etc. The performance requirements for these desktop scenarios are quite different from3D games. In fact, this is the reason in Windows Vista Experience Index (WinEI) we give you rating for these two scenarios separately, highlighted in the image below:

Figure 1. WEI sample with Graphics capabilities highlighted.

Graphics performance is usually assessed through many benchmarks. These can be classified into 2 broad categories:

Scenario benchmarks: These report performance of particular scenarios, e.g. the frame rate when flying over a terrain in a flight simulator. Many of the popular games have in-built benchmark modes that demonstrate how fast the graphics perform in scenes that are typical for that game.
Synthetic benchmarks: These highlight the performance of a particular aspect of graphics such as the ability to draw a number of lines as fast as possible.

However, there are plenty of things that we all do on our PCs that don’t have benchmarks tracking them that are still quite critical to make fast. In these cases we use the instrumentation within Windows to obtain timing information and then analyze the performance.

This blog entry discusses various aspects of graphics performance - both gaming and desktop graphics performance. It calls out the changes we made in Windows 7 to address user feedback as well as to take advantage of modern hardware to improve graphics performance.

Desktop Responsiveness

Many have experienced scenarios where an application, or Windows itself, stops responding momentarily. This is type of a performance issue that can be impacted significantly by the performance of graphics in the PC. We categorize these as desktop responsiveness issues. Improving responsiveness, both in real terms and by avoiding non-responsive moments, is one of the key ways that performance is improved in the system. It is also hard to measure.

Measuring desktop responsiveness is a hard problem since a number of issues which affect responsiveness aren’t easily reproducible and there is a great variety of them. They are rarely caught by either kind of benchmark as these issues are dependent on real-world combinations of factors. For Windows 7 we spent a great deal of time looking at these performance glitches using a mechanism in test versions of Windows 7 which has the ability to record key OS events and when they occurred. During real-world testing when we encounter a responsiveness problem, the tester can hit a record key and enter a small description of the issue encountered. The event history with diagnostic information called a “performance trace” is written out to a file and uploaded to a server where a team of performance analysts parse the data to figure out the cause of the responsiveness issue. This process has been successful to the extent that today most responsiveness issues can be quickly tracked down and root-caused.

Using this methodology, we analyzed thousands of desktop responsiveness traces where the tester experienced a frozen desktop anywhere from 100msec to several seconds. The type of issues ranged from an antivirus blocking disk access for all applications while updating itself on the vendor’s website to an application doing network access from a UI thread. In a significant portion of these traces, we found that a GDI application is waiting on another GDI application which is experiencing slowdowns due to excessive paging activity. This was the single-most frequently occurring cause of all desktop responsiveness issues, which without this data we probably would not have assumed. Based on these investigations, we worked to improve the architecture in these two key areas:

GDI Concurrency: Improve responsiveness when multiple applications are running. This required a re-architecture of the code around GDI (Graphics Device Interface) synchronization objects or “locks”.
Reduction of overall memory footprint of Windows: Reduce the memory overhead of composition in the Desktop Window Manager (DWM), which is the component responsible for rendering the desktop. This helps reduce overall paging activity and thus is especially important for low-memory PCs, especially using shared graphics memory.

These are described in more detail below.

GDI Concurrency

A number of performance traces we investigated in the context of desktop responsiveness pointed us to the design of a key synchronization mechanism in GDI. The performance challenge happens because the design of GDI in Windows Vista allows only a single application to hold a system-wide exclusive global lock. While this seems obvious in hindsight, when this decision was originally made the performance characteristics of different parts of the system made this optimistic implementation perfectly reasonable.

Figure 2. Existing architecture of GDI concurrency.

GDI applications running simultaneously vie for this global lock in order to render on the screen. The application that accesses the global lock prevents other applications from rendering till it releases the global lock. The situation often gets exacerbated when the application that is holding the lock needs to page in a large amount of memory from the disk since moving the memory from the disk to RAM takes a relatively long time. The above picture shows two GDI applications running simultaneously, contending for the global lock. If App X gets hold of the lock, it can render to the screen while App Y is unable to do so and waits for App X to finish.

Figure 3. Windows 7 architecture of GDI concurrency.

The solution to the problem was therefore to reduce the lock contention and improve concurrency by re-architecting the internal synchronization mechanism through which multiple applications can reliably render at the same time. Contention due to the global exclusive lock is avoided by implementing a number of fine-grained locks which are not exclusive but aid parallelism. The increased number of fine-grained locks adds a small overhead for scenarios where only a single application is rendering at a time.

Special attention was paid to GDI application compatibility as changing internal synchronization mechanism in the most widely used API stack could potentially give rise to timing issues such as deadlocks and rendering corruption.

This work also resulted in better rendering performance of concurrent GDI applications on multi-core CPUs. Multi-core Windows PCs benefit from these changes as more than one application can now be rendering at the same time.

After the GDI concurrency work was implemented in the Windows 7 builds leading to the Beta, we saw a large reduction in the number of desktop responsiveness issues reported by our testers which were caused by one application blocking another one due to GDI. To further validate the scalability of our new implementation, we wrote tests that draw 2D GDI primitives and measured the rendering throughput by launching simultaneously multiple such applications. The throughput is measured by adding together the frame rate (FPS) of each application window. Below is a sample of these results on a quad-core CPU system.

Figure 4. GDI Concurrency and Scalability.

Without the Windows 7 GDI concurrency, the rendering throughput of these applications is effectively limited to the performance of a single CPU core. Since only a single application can acquire the global exclusive lock while the others are waiting, this scenario doesn’t benefit from multiple CPU cores. This demonstrates that GDI applications in Windows 7 are now much less dependent on one another. This benefit will not need any new display drivers; it will work on any Vista (WDDM 1.0) and newer display drivers.

Desktop Graphics - Reduced Memory Footprint

Another area which affects system responsiveness is memory usage. Simply put, increased system memory (RAM) usage leads to an increased paging activity which directly leads to reduced system responsiveness. Thus, for the best responsiveness, all applications, processes and OS components need to use as little system memory as possible.

In Windows Vista, the amount of memory required to run multiple windows scales linearly with the number of windows opened on the system. This results in more memory pressure when there are more windows or if the monitors have higher resolution. It gets worse if you have more than one monitor. As part of investigating various means to improve system responsiveness, we saw a great opportunity in reducing the usage of system memory by DWM. In Windows Vista, every GDI application window accounts for two memory allocations which hold identical content – one in video memory and one in system memory. DWM is responsible for composition of the desktop through the graphics hardware. Hence, it requires a copy of the same allocation in video memory, which is easily accessible by the graphics hardware. The duplicate copy present in system memory is required because GDI is being rendered utilizing the CPU completely in the operating system without any assistance or “acceleration” by the graphics hardware. As the CPU performs all the tasks for rendering GDI applications, it requires an easily accessible cacheable copy of memory.

Figure 5. Existing memory allocations.

Windows 7 saves one copy of the memory allocation per application window by getting rid of the system memory copy entirely. Thus, for a GDI application window visible on the desktop, the memory consumed is cut in half.

Figure 6. Windows 7 memory allocations.

We achieved the reduction in system memory by accelerating the common GDI operations through the graphics hardware - the WDDM drivers accelerate these to minimize the performance impact of the CPU read-back of video memory. This was necessary as performing these operations otherwise on the CPU would incur a heavy performance penalty. In order to decide which GDI operations to accelerate, it was important to understand the usage pattern of various GDI applications. We profiled the top 100 GDI applications to learn more about their calling patterns and frequency and nature of the GDI operations.

Figure 7. Calling patterns and frequency of GDI operations for 100 GDI-based applications.

Based on real-world application statistics, a tiny snapshot of which is seen above, we worked with our graphics IHV partners to provide support in their drivers to accelerate the most commonly used GDI operations. Windows 7 systems with these updated drivers, called “WDDM v1.1” will thus benefit from this memory savings work. Please note that WDDM 1.0 drivers continue to function and are fully supported on Windows 7. You might have seen Windows Update providing these 1.1 drivers during the Beta—these drivers are themselves in Beta.

Figure 8. Desktop Window manager memory consumption comparison using WDDM 1.1 v. WDDM 1.0.

The above data shows that the memory savings become more and more pronounced when you have multiple application windows visible on the desktop. Since you save a lot of system memory, the paging activity gets reduced – as a result, your system responsiveness improves for the same workload.

Certain trade-offs had to be made for the desktop responsiveness improvements which benefit a wide range of systems. For example – the elimination of the duplicate system memory copies which “speed up” certain operations introduced slightly reduced performance as the CPU now has to read data back from the video memory. An analysis of real-world application statistics showed that these operations were rare. However, certain GDI micro-benchmarks which issue these operations show some performance degradation. This is something to note if you are running existing benchmarks that stress specific GDI operations repeatedly, which isn’t necessarily a reflection of real-world performance. Our observation has been that these slow-downs do not impact the end-user functionality directly and that the memory savings directly result in Windows 7 being much responsive overall. The improvements overall are definitely noticeable on memory constrained PCs with shared memory graphics.

Gaming Performance

No article on graphics performance is complete without talking about gaming, which is still the most widely analyzed and discussed aspect of graphics performance. There are a number of popular benchmarks such as 3D Mark as well as in-game benchmarks which are really a mode in which you can run your game where it renders the game scenes and animations without any user interaction. This area has thus been well tracked by the gaming industry through various industry benchmarks, which are pretty realistic and representative of actual games. The different benchmarks and tests are widely discussed and gamers all well-versed in the subtleties of these measurements and translating them into recommendations depending on their hardware, drivers, and gaming expectations.

For Windows 7, we have worked closely with our Graphics IHV partners, helping them improve the WDDM drivers’ gaming performance with specific changes to how Windows 7 works under the hood, while maintaining the same driver model and compatibility. Our continued investments in performance tools has helped us and our IHV partners track down and analyze various gaming performance bottlenecks and fix them in subsequent driver releases. The fundamentals of the Windows Display Driver Model remain unchanged in Windows 7. Some policies around GPU scheduling and memory management were changed to enable better performance in certain scenarios.

Because these benchmarks are very sensitive to the specific hardware, firmware, drivers, and overall system and because these are so widely measured and discussed elsewhere we are going to leave these comparisons to third parties. Like many areas in Windows 7, our commitment is to engineer even better performance across many dimensions. We believe it is better for you to experience these efforts directly. In comparing Windows 7, we would encourage the comparison using Windows Vista SP1 and keep in mind the difference you might see in WDDM 1.0 v. 1.1 and that the 1.1 drivers are still under development.

In Closing…

As you can see, in engineering Windows 7 we have worked hard to improve the architecture for graphics for real-world performance. It benefits both ends of the hardware spectrum – by enabling low physical memory systems to run a leaner and faster Windows and at the same time enabling multi-core PCs render multiple graphics applications much more efficiently.

-Ameet

Advances in typography and text rendering in Windows 7

e7blog — Fri, 13 Feb 2009 11:00:00 GMT

Even with the pictures and videos so commonplace on PCs, many of us spend most of our time looking at and interacting with text. Yet few of us stop to think about the depth of technology required to render text well and that this is an area that continues to benefit from improved technology in displays, graphics cards, as well as the APIs available to developers. In Windows 7, The support for text and fonts in GDI continues to provide the foundation for compatibility and application support. Building on the foundation of the modern DirectX graphics infrastructure, Windows 7 enhances the text output available to developers with DirectWrite. This is a new API subsystem and one that over time you will see adopted more broadly by applications from Microsoft, independent software developers, and within Windows itself. This post will also talk about improvements to ClearType and the Fonts, both available as part of the improvements to the GDI-based text APIs. This work was introduced at the PDC (pointers towards the end of the post). This post is by Worachai Chaoweeraprasit, a development lead on our Graphics feature team. --Steven

One of the high-level goals of Windows 7 is to have even better graphics – graphics with higher fidelity. To that end, my team is looking into how to improve one of the most basic graphic elements in Windows, and that is text – the thing that’s always right in your face, but we hope you’ll never actually see it.

The need for good text

About 80% of the time people spend with their PC is to either read or write. This should come as no surprise when you realize that text is essentially how the machine talks back to you, and until we have a technology that would allow it to interject thought directly into our brains, text would probably continue to be the way we receive information from the computer screen.

Studies have shown that good text leads to better productivity. Essentially we are wired as human to be incredibly good at capturing words and making a smooth, rapid transition between them – the basis of reading. We’re so good at it that we can do it unconsciously with incredible speed given that the text is optimized for that process. This might explain why many can sink in to a good book for hours, but some quickly become tired after staring at the computer screen for a while. Any visual-related factor that could disrupt the reading process effectively slows us down. Good text, therefore, is text that is tuned to support the human reading process with minimal distraction possible.

The evenness of the white surrounding each letter, word, line, and paragraph plays a huge role in keeping the pace of reading while the black elements holds our attention together. A line too long, a word too tight, a paragraph too uneven, any of these conditions take us farther and farther away from the message being delivered but closer and closer to the mere medium delivering it. The art of text is essentially to make the actual text itself disappears before your eyes, so that the ideas it delivers reappear in your head. The study of how to prepare proper text is known as typography. And, as a typographer would say: good typography is not to be seen; only the bad ones are. As a platform, the role of Windows is to deliver great presentation of text and offering software developers great tools for creating the best presentation possible in the context of the software they develop.

Improving current techniques

People tend to develop habits and often over time these become the preferred way of getting things done. The more mundane the activity is, the easier we become attach to it, and the harder we’re willing to change. When it comes to text on your screen, the same screen you look at days in and days out. It could quickly become awkward if that completely changes overnight – even for the better. So, how do we go about improving on what we all become so used to? We want to make sure to support what is there and improve it, while supporting existing methods. But, before we get to understand the improvement, let’s first take a closer look into the current implementation really is and what challenges it presents over the years.

The current implementation is the product of text rendering design based on device pixel. The dimension of text at a certain size eventually translates into a fixed number of pixels in horizontal and vertical direction on the device surface. A 10-point text would translate to roughly 80 pixels height on a typical printer device of 600 dpi, while the same text would merely acquire 13 pixels on a 96 dpi monitor. This physical screen condition was hardly adequate for the quality we’re seeking for good text on screen.

Fortunately, the advent of ClearType during the past decade has largely improved the clarity aspect of quality. ClearType leverages the anatomy of the LCD pixel structure and takes advantage of the human visual system to distribute the energy typically emit to a whole display pixel, across the neighboring sub-pixels in the LCD’s typical 3-color channels making up each individual pixel, to create the visual illusion of higher resolution raster quality on a lower resolution device. As the result, ClearType text looks significantly sharper than the typical text on an LCD display, mitigating a large portion of the quality problem on a display technology that would become hugely popular a few years later.

Another pleasant design of the original ClearType in Windows was that it has improved the clarity of text without breaking application compatibility – that is, it doesn’t change the actual size of each individual glyph in either direction, nor did it change the distance between the two adjacent ones. This is the reason one could turn it on or off at will without having to “store” the selected option in the document or application. It is entirely per-user rendering preference. In Windows 7 we also improved the ClearType Text Tuner in keeping with our theme of being in control of your PC experience, by providing even more granular choices when tuning ClearType (and of course you can still turn it off).

But like many other things in the world, the coin comes in two faces. While it is able to preserve backward compatibility, it is limited by its own leverage unable to advance the state of the art. The width and height of the individual glyph and the nominal distances between the adjacent two remain fixed to the rounded number of screen pixels at a given size.

One of the graphics improvements we made in Windows 7, therefore, is to move from the physical pixel model of the past, and instead creating a new design around what we call the “device independent pixel” unit (or “DIP”), a “virtual pixel” that is one-ninety-sixth an inch in floating-point data type. In this model, a glyph (or any other geometric primitive for that matter) can size to fractional pixels, and be positioned anywhere in between the two pixels. The new ClearType improvement allows sizing and placement of glyph to the screen’s sub-pixel nearest to its ideal condition, creating a more natural looking word shape and making text on screen looks a lot closer to print quality.

The following figure shows the side-by-side comparison of the same word between the original or today’s ClearType (above) and the Windows 7 improvement – Natural ClearType (below), which does require calling the new APIs to render. Notice the width of the letters in the word and the spacing between them, as well as how the more consistent width and spacing improves the overall appearance of the entire word. Note that all the letters are placed with its nominal spacing and there is no kerning adjustment being applied here. A great article by Kevin Larson – a researcher in the Advanced Reading Technology team, discusses in details the scientific aspect of word recognition.

The ability to be more precise in approximating the screen placement of natural text also lends itself to a very nice side-effect, and that is the fact that text can now be placed on the line with no regards to the actual display device’s resolution. It means a UI designer can design an application UI knowing it’ll look the same on all other screens as it appears on his or her screen regardless of what type of display device the users might have. This fact is also particularly handy for software localization where the translated text produces the same layout everywhere.

This improvement could also offer a more realistic view of a print document on screen, or make the screen document looks closer to its print counterpart. It could also improve the quality of document zooming. Imagine document zoom that could go in and out in the same manner as what you would see when pulling the actual print page closer and farther away from your sight. It could mean a more joyful experience for online reading.

Fonts and Font Management

The Font is the heart and soul to typography, much like photo is to photography. A lot more fonts are shipped with Windows these days while even more are developed around the world. Windows Vista shipped with 40% more fonts comparing to Windows XP. Windows 7 is expected to ship with 40+ new fonts, just to underscore this trend. We’ve also added some additional viewing/categorization capabilities using the Windows 7 Explorer to improve working with a large library of related fonts.

The default common controls’ font dialog and the font chunk in Windows 7 Ribbon are also updated to be more intelligently selective of what fonts to be present to the user of the current user’s profile. Depending on a number of settings including the current UI language, the user locale, and the current set of keyboard input locales, the font list hides fonts of languages not typically used by the user of different culture and locale. For example, all the international fonts are automatically hidden away from a typical English user to reduce clutter and promote better productivity in common system applications such as NotePad, WordPad and Paint. Third-party application utilizing the Ribbon or the common controls’ font dialog could also have the same benefit. The user still retains the option of selecting any desired font back to the view by explicitly marking it in the Windows 7 Control Panel’s Font applet.

Operating System	Fonts shipped “in-box”
Windows XP SP2	133
Windows Vista	191
Windows 7	235 (currently planned)

This growth, however, introduces some new opportunities for improvement. We’ve long treated fonts as system-wide resources. It gets “installed” on the machine and kept in a single flat namespace managed by the core part of the operating system. It may be interesting to some that the font named “Arial Black” isn’t really in the same grouping as “Arial Narrow” or “Arial”. This is because as far as the operating system is concerned, they are just different fonts with different names. And because font is uniquely identified by its name, you can’t have multiple versions of the same font at the same time.

Because font is system resource, non-traditional usage of font such as font embedded within the document, and font used exclusively in an application is done through the mechanism known as private installation, which involves making sure the font name is unique before installing it programmatically but doing so by hiding it from others to see. Private font is just like font installed publicly as far as the operating system internal is concerned.

An important improvement in Windows 7’s new font system is the notion of “font collection” which allows partitioning of fonts sharing the same usage into a separate namespace. The system collection is similar to what exists today and is created and managed by the system whereas custom collection can be created and managed, as many as needed, entirely by the application program. This allows document to have its own set of fonts local to it, and third-party application or plug-in to ship with its own font used exclusively within the program. This partitioning not only reduces unnecessary system-wide font update and allows update to happen only locally as needed, it also allows access to multiple versions of the same font in different collections.

The new font system also improves the way fonts are organized within the collection. It supports the notion of weight-width-slope variation where fonts with the same stylistic root but vary in its weight (thin, light, bold, black, etc.), width (wide, narrow, etc.), or slope (italic, oblique) are grouped together in the same font family. For instance, “Arial Narrow” becomes a variation or face in the “Arial” family. This grouping model is advocated by the CSS recommendation.

Font Art

Fonts also represent art and artistic expression. The technology helping create font is therefore the artist’s tool of expression. An important technology called OpenType emerged during the past decade. It enables new ways type design can be realized. OpenType allows designer to define how glyphs interact and transform in stages. The designer then exposes this function as an executable unit known as the “font feature” for application programmable access.

OpenType was an offshoot of the TrueType Open technology Microsoft developed in 1994-95. The TrueType Open technology added the GSUB, GPOS, BASE, JSTF, and GDEF tables to the TrueType format. The primary usage at the time was to help with the creation of Arabic font due to the inherent complexity of the task. Microsoft chose to rename the technology to OpenType in 1996 and Adobe added their CFF glyph outline format to the technology in the same year. Today OpenType is used to improve readability of text as well as to express new and exciting type design in various languages.

However, despite its long-time presence and availability, the usage of OpenType in the Windows world remains largely in specialized programs. The Windows native graphics system has not fully embraced OpenType for its mainstream usage of text. This absence discourages many designers as there is no standard way in Windows to test the feature they produce. Likewise, its limited exposure doesn’t encourage discoverability for mainstream application developers. Improving this and transitioning to this improved rendering technology is a multi-step and multi-release investment done so as to maximize the benefit while minimizing the disruption that might be introduced as incompatibilities. Windows 7 takes another step on this path. We know for many that care deeply about this area there is a strong desire to move faster. We are doing our best to balance the speed of transition with the desire to maintain compatibility.

Windows 7 new text system not only uses available OpenType features internally but also allows access to any feature made available in the font in the high level programming interface, making it easier for application developer to discover and exercise the font feature in mainstream scenario. Windows 7 also ships with a brand new OpenType font “Gabriola” developed by a well-respected typographer John Hudson. Gabriola makes heavy use of contextual letterforms and offers an unprecedented number of stylistic sets for different usages of the font in different occasions. The figure below enumerates all stylistic sets available in this font; notice the subtleties and not-so-subtle way to distinguish each stylistic set.

The figure below also demonstrates the power of contextual letterforms in the eighth rendition of Gabriola’s stylistic set (“ss07”) to produce different ways the same word is rendered depending on where it’s at in the line.

New APIs

Rendering text is complex and involved, even though it seems like something that should be straight forward. There are probably hundreds of ways to format text in a document and often many paths that ultimately yield the same results. HTML/CSS is a complex standard and is a great example of the richness of how text may be formatted and typeset. Underneath the formatting logic lies the language requirement – the rule of writing for the language. Windows has long been supporting Unicode – another complex standard for global data interchange. Windows supports an increasing number of Unicode script in every single release. The mapping from the input text to the final glyphs in the font requires intricate transformation, which involves parsing of font data and analyzing the language writing pattern. Once the glyph is finalized, it is then rasterized, merged and filtered into the final visual on the display device.

Due to this staging nature, different types of applications require different support from the text system. While a typical application such as the legendary “Hello world” type application may be satisfied with only the ability to get some text out showing to the user. The same level of support is hardly adequate for document preparation system such as Microsoft Word and Adobe InDesign. Some of the more mature application code bases may also have to deal with different graphics systems. This makes it harder in practice for a text system that tie to a particular graphics model to really be widely useful across the wide variety of applications in the Windows ecosystem.

It became obvious to us early on during the planning stage of Windows 7 that text processing is not homogeneous, and different types of applications have different needs and requires different levels of support. The appropriate level of programming access to the text functionality is as important as the functionality itself. The new text system in Windows 7 is assembled into a self-sufficient system called DirectWrite. The API is provided in four layers – the interfaces for font data, rendering support, language processing, and typesetting, each built upon the others with the lower layer makes no requirement to the upper one, and none depends on a specific graphics model. To illustrate the latter point, the figure below shows a sample application that uses the new typesetting interface and language processor while the final rendering happens as an extruded filled 3D geometry from the 2D graphics environment also new to Windows 7 called Direct2D. Both systems were introduced in PDC 2008 as the new graphic foundation in Windows 7.

DirectWrite preserves developer’s investment in existing technologies such as GDI and GDI+ in three important aspects. First, the previously described layering design of DirectWrite allows for the clean separation between the two fundamental processes of placing and rendering of text. It enables applications to use DirectWrite to place text while having it rendered onto traditional graphic surfaces such as GDI and GDI+. The reverse scenario in which the application may use GDI to place text while having it rendered through DirectWrite is also naturally supported. The second aspect of compatibility comes from the fact that DirectWrite also supports all existing methods for placing and rendering text found in GDI. A DirectWrite application can use DirectWrite to place and render text in the same manner as GDI does without actually using GDI. Text placed and rendered under this compatibility mode is indistinguishable from GDI text from the user’s point of view, and as such preserving existing layout of application UI and text document. Lastly, DirectWrite exposes a set of APIs that interoperate with GDI. An application selecting a GDI font object can turn it into a DirectWrite’s font object and vice versa. Since the font system is at the low end of the DirectWrite API layer, it provides a natural interoperability point that is fundamental enough to ensure high degree of data preservation and correctness. Once the application is able to acquire a DirectWrite’s font object, it can in turn use it in any other DirectWrite API requiring a DirectWrite font from that point onward. The conversion from a DirectWrite’s font object back to a GDI font object allows the rest of the GDI-based application to function with no change while still being able to reap the benefit of using DirectWrite’s new and improved font model. As in some real world examples, the XPS print rasterizer in Windows 7 is implemented on top of DirectWrite and utilizes DirectWrite’s interoperability API to convert back to a GDI font as part of the conversion of an XPS-based print job for a non-XPS printer driver. The Windows 7 XPS Viewer also uses DirectWrite alongside the GDI+ graphic rendering for its onscreen display.

There’s a lot more to the details of the API. In the PDC session linked to above, Leonardo Blanco and Kam VedBrat go into the details of DirectWrite and Direct2D and how to develop applications such as this.

The world has changed a lot since the first text APIs of Windows GDI, such as TextOut or ExtTextOut in Windows NT 3.1 (or the subsequent API additions). The evolution of support for text is a critical part of the underpinnings of Windows 7. We continue to improve this most “basic” element of a graphical operating system so that regardless of the language, script, or device used to render text, Windows will offers a great set of tools and APIs for developers and a great experience for end-users.

--Worachai

More Follow up to discussion about High DPI

e7blog — Tue, 16 Sep 2008 10:00:00 GMT

Excellent! What a fun discussion we’ve been having on High DPI. It has been so enriching that Ryan wrote up a summary of even more of the discussion. Thanks so much! --Steven

There have been quite a few comments posted regarding high DPI, along with some lively discussion. Most of what has been said has been good anecdotal examples which are consistent with the data we have collected. For the areas where we didn’t have data, the comments have helped to validate many of our assumptions for this group. It is also clear that there are some areas of this feature which are confusing, and in some cases there is a bit of “myth” around what is ideal, what is possible, and what is there. This follow up post is mostly to summarize what we have heard, and to provide some details around the areas where there has been a bit of confusion.

Here is a list of our top “assumptions” which have been echoed by the comments posted:

Most people adjust the screen resolution either to get larger text, or because it was an accident
There is a core of people who know about high DPI and who use it
Some people prefer more screen real-estate while others people prefer larger UI
Discoverability of the DPI configuration is a concern for some
App compat is a common issue, even a “deal breaker” in some cases
IE Scaling is one of the top issues listed (see IE8 which fixes many of these!)
Lots of complexities/subtleties and it is pretty hard to explain this feature to most people

There have also been a number of areas where there has been a bit of confusion:

Is it true that if everything were vector-based, these problems would all go away?
Shouldn’t this just work without developers having to do anything?
How is this different from per-application scaling like IE zooming?
Should DPI be for calibration or for scaling?

Most of these topics have been covered to some degree in the comments, but since there has been so much interest, we decided to go into a bit more details around a few of the top issues/concerns.

Vector Graphics vs. Raster Graphics

With PCs, there is always the hope of a “silver bullet” technology which solves all problems making life easy for users, developers, and designers across the board. As an example, some of the comments to the original posting suggested that if we just made the OS fully vector based, these scaling problems would go away. It turns out that there are several issues with using vector graphics which are worth explaining.

The first issue is that oftentimes when an icon gets to be small in size, it is better to use an alternate representation so that the meaning is clearer. Notice the icons below. In this case, the designer has chosen a non-perspective view for the icon when it is rendered at it’s smallest size.

This is because the designer felt that the information expressed by the icon was clearer with a straight-on view at the smallest size. Here is another example illustrating this point:

Of course, this means that the designer must create multiple versions of the source image design, so there is additional complexity. The point here is that there is a tradeoff that must be made and the tradeoff is not always clear.

Even when one vector source is used, it is common to have size-dependent tweaking to make sure that the result is true to what the designer had in mind. Imagine a vector graphic which has a 1-pixel line at 128x128 that gets scaled down by 1/2 to 64x64. The display has no way of rendering a perfect 1/2 pixel line! In many cases the answer is that the renderer will “round” to a nearby pixel line. However, doing this inherently changes the layout of the sub-elements of the image. And there is the question of, “which pixel line to round to?” If the designer does not hand tune the source material, it will be up to the rendering engine to make this decision, and that can result in undesirable effects. One could say that this should therefore define rules about what elements should be use in a vector, but that only further restricts what concepts can be represented.

It turns out that even the TrueType fonts which we use in Windows are hand-tuned with size-dependant information in order to make the result as high quality as possible. Most of the TrueType rendering pipeline is based on algorithmic scaling of a vector source, but there are size-dependent, hand-coded “hints” in TrueType which the designer specifies to direct the system how to handle edge cases, such as lines falling between pixel boundaries. Here is a link describing this in more detail: http://blogs.msdn.com/fontblog/archive/2005/10/26/485416.aspx

It is not even true that vector graphics are necessarily smaller in size (especially for small images). Most designers create graphics using an editor which builds up an image using many layers of drawings and effects. With bitmap based graphics, designers will “flatten” the layers before adding it to a piece of software. Most designers today pay little attention to the size of the layers because the flattening process essentially converts it to a fixed size based on the image resolution. With vector graphics, there is no such flattening technique so designers need to carefully consider the tools that they use and the effects that they add to make sure that their icon is not extremely large. I spent some time with one of our designers who had a vector graphic source for one of our bitmaps in Windows and the file was 622k! Of course that file size is fixed regardless of the resulting resolution, but that huge file flattens into this 23k PNG bitmap.

Of course, a hand-tuned vector based representation of this could be probably made smaller if the size constraints were part of the design time process. But that would be an additional constraint put on the designer, and one which they would need to learn how to do well.

How do we help developers?

For applications that need to carefully control the layout and graphics, or scale the fidelity of the images based on the available resolution, having a way of specifying specific pixel locations for items is important to get the best result. This is as true on the Mac as it is on the PC (see http://developer.apple.com/releasenotes/GraphicsImaging/RN-ResolutionIndependentUI/). There is often a belief that if we just provided the right tools or the right framework then all these problems would go away. We all know that each set of tools and each framework have their own set of tradeoffs (whether that is Win 32, .net, or HTML). While there is no silver bullet, there are things we can do to make writing DPI aware applications easier for developers. As an example, there are two important upcoming ecosystem events in which we will be talking in detail about High DPI. We will also have materials which we will be making available to developers which will help educate them on how to convert existing applications to be DPI aware. The first event is Microsoft Professional Developer Conference, where we will talk about High DPI as part of the talk “Writing your Application to Shine on Modern Graphics Hardware (link)”. The second is the Windows Hardware Engineering Conference, in which we will be discussing high DPI as part of the “High Fidelity Graphics and Media” track (link).

Help with App Compat Issues

There have been several posts on app compat and high DPI (for example bluvg’s comment). There have also been comments talking about the complexity and understandability of the High DPI configuration. In some cases the app compat issues can be mitigated by enabling or disabling the automatic scaling feature. This can be changed globally by going to the DPI UI, clicking the button labeled “Custom DPI” and changing the checkbox labeled, “Use Windows XP style DPI scaling”. When this checkbox is unchecked, applications which are not declared to be DPI aware are automatically scaled by the window manager. When it is checked, automatic scaling is disabled globally. It is interesting to note that for DPI settings < 144 DPI, this box is checked by default, and for DPI settings >= 144 it is unchecked by default. In some cases, changing the default settings can result in a better experience depending on the applications that you use and your DPI setting. It is also interesting to note that automatic scaling can be turned off on a per application basis using the Vista Program Compatibility properties. Here is a link for more info on how to do that: http://windowshelp.microsoft.com/Windows/en-US/help/bf416877-c83f-4476-a3da-8ec98dcf5f101033.mspx. (Look at the section for “Disable Display Scaling on high DPI settings”.)

How is DPI scaling different from per-application scaling (like IE Zoom?)

A typical application UI is made up of a content window and a frame UI. The frame UI is where the menu items and toolbar buttons are. The content window is the “document view”. For example, in IE the content window is the actual webpage. It turns out the code required to support high DPI scaling for the content windows is the same code required to do the zooming feature. In fact, for the content window, IE8 simply uses the high DPI setting to configure the default zoom factor (see DPI Scaling and Internet Explorer 8 for more details). However, high DPI has the additional side effect of scaling the size of the frame UI. Since most people use the scaling feature to make text larger to be more readable, it makes sense to scale the frame UI as well, since the text in the menu items and toolbar tooltips will also scale. In a sense if there is per-application scaling that is really about the content area, and applications will support that as developers see the customer need. DPI scaling makes the UI elements of all applications render similarly.

Shouldn’t DPI really be used for calibrating the screen (so “an inch is an inch”)?

Some have suggested that we should just use high DPI as a way to calibrate the screen so that the physical size of an object is the same regardless of the display. This makes a ton of sense from a logical perspective. The idea would be to calibrate the display so “in inch is an inch”. We thought about doing this, but the problem is that it does not solve the end user need of wanting to have a way to configure the size of the text and the UI. If we then had a separate “global scale” variable, this would mean that application developers would need to pay attention to both metrics, which would add complexity to the developer story. Furthermore, if a user feels that the UI is too small, should it be up to the developer or the user to set the preference of how big the UI should be? In other words if the designer wants the button to be an inch, but the user wants the button to be 1.5 inches to make it easier to use, who should decide? The way the feature works today, it is up to the user to set their preference, but it is up to the application developer to make sure that the user preference is honored.

Once again, it is really great to see so much interest in high DPI. We certainly have some challenges ahead of us, but in many ways it seems like the ecosystem is ripe for this change. Hopefully this follow up post helped to consolidate some of feedback which we have heard on the previous post and explain some of the complexities of this feature in more detail.

--Ryan Haveson

Follow-up on High DPI resolution

e7blog — Sat, 13 Sep 2008 10:00:00 GMT

One of the cool results of this dialog is how much interest there is in diving into the details and data behind some of the topics as expressed in the comment and emails. We’re having fun talking in more depth about these questions and observations. This post is a follow-up to the comments about high DPI resolution, application compatibility, and the general problems with readability in many situations. Allow me to introduce a program manager lead on our Desktop Graphics team, Ryan Haveson, who will expand on our discussion of graphics and Windows 7. –Steven

When we started windows 7 planning, we looked at customer data for display hardware, and we found something very interesting (and surprising). We found that roughly half of users were not configuring their PC to use the full native screen resolution. Here is a table representing data we obtained from the Windows Feedback Program which Christina talked about in an earlier post.

We don't have a way of knowing for sure why users adjust their screen resolution down, but many of the comments we’ve seen match our hypothesis that a lot of people do this to because they have difficulty reading default text on high resolutions displays. With that said, some users probably stumble into this configuration by accident; for example due to a mismatched display driver or an application that changed the resolution for some reason but did not change it back. Regardless of why the screen resolution is lower, the result is blurry text that can significantly increase eye fatigue when reading on a PC screen for a long period of time. For LCD displays, much of the blurriness is caused by the fact that they are made up of fixed pixels. In non-native resolution settings, this means that the system must render fractional pixels across fixed units, causing a blurred effect. Another reason for the relative blurriness is that when the display is not set to native resolution, we can’t properly take advantage of our ClearType text rendering technology , which most people (though not all) prefer. It is interesting to note that the loss of fidelity due to changing screen resolution is less pronounced on a CRT display than on an LCD display largely because CRTs don’t have fixed pixels the way that LCDs do. However, because of the advantages in cost and size, and the popularity of the laptop PC, LCD displays are fast gaining market share in the installed base. Another problem with running in a non-native screen resolution is that many users inadvertently configure the display to a non-native aspect ratio as well. This results in an image that is both blurry and skewed! As you can imagine, this further exacerbates the issues with eye strain.

Looking beyond text, in these scenarios the resulting fidelity for media is significantly reduced as well. With the configuration that many users have, even if their hardware is capable, they are not able to see native “high def” 720p or 1080p TV content, which corresponds to 1280x720 and 1920x1080 screen resolutions respectively. The PC monitor has traditionally been the “high definition” display device, but without addressing this problem we would be at risk of trailing the TV industry in this distinction. While it is true that only about 10% of users have a truly 1080p capable PC screen today, as these displays continue to come down in price the installed base is likely to continue to grow. And you can bet that there will be another wave of even higher fidelity content in the future which users will want to take advantage of. As an example, when displays get to 400 DPI they will be almost indistinguishable from looking at printed text on paper. Even the current generation of eBook readers with a DPI of ~170 look very much like a piece of paper behind a piece of glass

From this we see that there is a real end user benefit to tap into here. It turns out that there is existing infrastructure in Windows called “High DPI” which can be used to address this. High DPI is not a new feature for Windows 7, but it was not until Vista that the OS user-interface made significant investments in support for high DPI (beyond the infrastructure present earlier). To try this out in Vista, rt. Click desktop -> personalize and select “Adjust Font Size (DPI)” from the left hand column. Our thinking for Windows 7 was that if we enable high DPI out of the box on capable displays, we will enable users to have a full-fidelity experience and also significantly reduce eye strain for on-screen reading. There is even infrastructure available to us to detect a display’s native DPI so we can do a better job of configuring default settings out of the box. However, doing this will also open up the door to expose some issues with applications which may not be fully compatible with high DPI configurations.

One of the issues is that for GDI applications to be DPI aware, the developer must write code to scale the window frame, text size, graphical buttons, and layout to match the scaling factor specified by the DPI setting. Applications which do not do this may have some issues. Most of these issues are minor, such as mismatched font sizes, or minor layout artifacts, but some applications have major issues when run at high DPI settings.

There are some mitigations that we can do in Windows, such as automatic scaling for applications which are not declared DPI aware (see Greg Schechter’s blog on the subject), but even these mitigations have problems. In the case of automatic scaling, applications which are not DPI aware are automatically scaled by the window manager. The text size matches the user preference, but it also introduces a blurry effect for that application’s window as a result. For people who can’t read the small text without the scaling, this is a necessary feature to make the high DPI configuration useful. However, other customers may only be using applications that scale well at high DPI or may be less impacted by mismatched text sizes and may find the resulting blurry effect caused by automatic scaling to be a worse option. Without a way for the OS to detect whether an application is DPI aware on not, we have to pick a default option. It always comes back to the question of weighing the benefits and looking at the tradeoffs. In the long term, the solution is to make sure that applications know how to be resolution independent and are able to scale to fit the desired user preference, which requires support in both our tools and documentation. The challenge for a platform is to figure out how to get there over time and how to produce the best possible experience during the transition.

Short term vs. long term customer satisfaction

Using the model of high definition TV, we can see that in the long term it is desirable to have a high fidelity experience. The only problem is that even though the high DPI infrastructure has been around for several windows releases (in fact there is an MSDN article dated 2001 on making applications DPI aware), we were not sure how many applications are actually tested in these configurations. So we were faced with an un-quantified potential short term negative customer impact caused by enabling this feature more broadly. The first thing we did is to quantify the exposure. We did this by performing a test pass with over 1,000 applications in our app compat lab to see how they behave at high DPI settings. The results we found are shown below, which shows the distribution of issues for these 1000 applications.

One quick thing, when we say “bug” we mean any time software behaves in a manner inconsistent with expectations—so it can be anything from cosmetic to a crash. We categorize the severity of these bugs on a scale from 1 to 4, where Sev 1 is a really bad issue (such as a crash and/or loss of data or functionality) and Sev 4 is an issue which is quite subtle and/or very difficult to reproduce.

It turns out that most applications perform well at high DPI, and very few applications have major loss of functionality. Of course, it is not the ones that work well which we need to worry about. And if 1% of applications have major issues at high DPI, that could be a significant number. So we took a look at the bugs and put them into categories corresponding to the issue types found. Here is what we came up with:

What we found was that one of the most significant issues was with clipped UI. Looking into this deeper, it became apparent that most of these cases were in configurations where the effective screen resolution would be quite low (800x600 or lower). Based on this, we were able to design the configuration UI in such a way that we minimized the number of cases where users would configure such a low effective resolution. One by one we looked at the categories of issues and when possible, we came up with mitigations for each bucket. Of course, the best mitigation is prevention and so High DPI is a major focus for our developer engagement stories for PDC, WinHEC, and other venues coming up.

Aggregate vs. individual user data

One thing for us to look at is how many users are taking advantage of high DPI today (Vista/XP). Based on the data we have, only a very small percentage of users are currently enabling the high DPI feature. This could easily be interpreted as a clear end user message that they don’t care about this feature or have a need for this feature. An alternate explanation could be that the lack of adoption is largely because XP and Vista had only limited shell support for high DPI, and the version of IE which shipped on those platforms had significant issues with displaying mismatched font sizes and poorly scaled web pages. Also, we do know anecdotally that there are users who love this feature and have used it even before Vista. Once again, we have to make an interpretation of the data and it is not always crystal clear.

Timing: is this the right feature for the market in this point in time?

Fortunately, we don’t have a “chicken and egg” problem. The hardware is already out in the field and in the market, so it is just a matter of the OS taking advantage of it. From a software perspective, most of the top software applications are DPI aware (including browsers with improved zooming, such as IE 8), but there remain a number of applications which may not behave well at high DPI. Another key piece of data is that display resolution for LCD panels is reaching the maximum at standard DPI. For these displays, there is no reason to go beyond 1900x1200 without OS support for high DPI because the text would be too small for anyone to read. Furthermore, this resolution is already capable of playing the highest fidelity video (1080p) as well as 2 megapixel photos. The combination of existing hardware in the field, future opportunity to unlock better experiences, and the fact that the hardware is now blocked on the OS and the software speak to this being the right timing.

Conclusion

Looking at customer data helps us identify ways to improve the Windows experience. In this case, we saw clearly that we had an opportunity to help users easily configure their display such that they would enjoy a high fidelity experience for media as well as crisp text rendered at an appropriate size. With that said, anytime we invest in a feature that can potentially impact the ecosystem of Windows applications we want to be careful about bringing forward your investments in software. We also want to make sure that we engage our community of ISVs early and deeply so they can take advantage of the platform work we have done to seamlessly deliver those benefits to their customers. In the meantime, the internal testing we did and the data that we gathered was critically important to helping us make informed decisions along the way. High DPI is a good example of the need for the whole ecosystem to participate in a solution and how we can use the customer data in the field, along with internal testing, to determine the issues people are seeing and to help us select the best course of action.

--Ryan