These postings are provided "AS IS" with no warranties, and confer no rights. Use of included code samples are subject to the terms specified at Microsoft -
Notes on comments.
Welcome to our blog dedicated to the engineering of Microsoft Windows 7
This is a little bit of a tricky post to write because we’re going to be asking everyone using our Windows 7 Beta to help us out, but doing so is going to take a little time and require a bit of a commitment to helping test the next milestone. This has been a remarkably valuable and beneficial testing cycle for Windows as we have had a tremendous amount of very rigorous testing and usage. We’ve had millions of people install and use the Beta since January and as we’ve talked about, the feedback and telemetry have been of tremendous value as we finalize the product. The effort of Beta testers has contributed immensely to our ability to deliver a high-quality product to hundreds of millions of customers. We continue to follow the plan we have previously outlined and this post is no announcement of any news or change in plans. Since we know many people are running the Beta we want to provide a heads up regarding the behavior of the Release Candidate (RC) as it pertains to upgrades. Of course we are working hard on the RC and following the schedule we have set out for ourselves.
A big part of the beta process is making sure we get as much “real world” coverage of scenarios and experiences as possible and monitor the telemetry of those experience overall. One of the most challenging areas to engineer is the process of upgrading one release of Windows to another. When you think about it, it is the one place where at one time we need to run a ton of code to basically “know” everything about a system before performing the upgrade. During the development of Windows 7 we routinely test hundreds of original OEM images from Windows Vista and upgrade them and then run automated tests validating the upgrade’s success. We also test thousands of applications and many thousands of devices as they too move through the upgrade process.
Many of you installed the Windows 7 beta on a PC running Vista. We received that telemetry and acted on it accordingly. We believe we’ve continued to improve the upgrade experience throughout the release. Similarly, based on our telemetry most of you did clean installations onto new drive partitions. Through this telemetry we learned about the device ecosystem and what drivers were available or missing. We also learned about PC-specific functions that required installing a driver / application (from XP or Vista) to enable support for buttons, connectors, or other hardware components. Together we get great coverage of the setup experience.
We’ve also learned that many of you (millions) are running Windows 7 Beta full time. You’re anxious for a refresh. You’ve installed all your applications. You’ve configured and customized the system. You would love to get the RC and quickly upgrade to it from Beta. The RC, however, is about getting breadth coverage to validate the product in real-world scenarios. As a result, we want to encourage you to revert to a Vista image and upgrade or to do a clean install, rather than upgrade the existing Beta. We know that means reinstalling, recustomizing, reconfiguring, and so on. That is a real pain. The reality is that upgrading from one pre-release build to another is not a scenario we want to focus on because it is not something real-world customers will experience. During development we introduce changes in the product (under the hood) that aren’t always compatible with what we call “build-to-build” upgrade. The supported upgrade scenario is from Windows Vista to Windows 7. Before you go jump to the comment section, we want to say we are going to provide a mechanism for you to use if you absolutely require this upgrade. As an extended member of the development team and a participant in the Beta program that has helped us so much, we want to ask that you experience real-world setup and provide us real-world telemetry.
If you do follow the steps below, you might run across some oddities after upgrade. We experience these internally at Microsoft occasionally but we don’t always track them down and fix them because they take time away from bugs that would not only manifest themselves during this one-time pre-release operation. From time to time we’ve noticed on a few blogs that people are using builds that we have not officially released and complained of “instabilities” after upgrade. Nearly all of these have been these build-to-build issues. We’ve seen people talk about how a messenger client stopped working, a printer or device “disappears”, or start menu shortcuts are duplicated. These are often harmless and worst case often involves reinstalling the software or device.
We’re just trying to be deterministic and engineer the product for the real world. Speaking of the real world, many have asked about upgrading from Windows XP. There's no change here to the plan as has been discussed on many forums. We realized at the start of this project that the “upgrade” from XP would not be an experience we think would yield the best results. There are simply too many changes in how PCs have been configured (applets, hardware support, driver model, etc.) that having all of that support carry forth to Windows 7 would not be nearly as high quality as a clean install. This is something many of you know and already practice. We do provide support for moving files and settings and will prompt at setup time, but applications will need to be reinstalled. We know that for a set of customers this tradeoff seems less than perfect, but we think the upfront time is well worth it.
So when you try to upgrade a pre-RC build you will find that you’re not able to and setup will tell you and you can then exit gracefully. You can install as a clean installation and use the Windows Easy Transfer feature as well (run this from your current installation of course) if you wish to move your accounts, settings, files, and more. To bypass the version check, the instructions below will use a mechanism that is available for enterprise customers (so we are also testing this as well). It is not a simple command line switch. We didn’t make it multi-step on purpose but wanted to stick to using proven, documented and tested mechanisms.
These instructions will be brief. Since everyone reading is a well-versed and experienced beta tester you know ALWAYS BACK UP YOUR MACHINE before running any OS installation and NEVER TEST AN OS ON YOUR ONLY COPY OF ANY DATA. Testing a pre-release product means just that—it is testing and it is pre-release. Even though this is a Release Candidate, we are still testing the product. We have very high confidence but even if an error happens once in 1,000,000 we want to make sure everyone is taking the precautions normal for a pre-release product.
One other related caution is INSTALL ONLY OFFICIALLY RELEASED BUILDS FROM MICROSOFT. It will always be tempting to get the build with the “mod” already done but you really never know what else has been done to the build. There’s a thrill in getting the latest, we know, but that also comes with risks that can’t even be quantified. For the RC we will work to release a hash or some other way to validate the build, but the best way is to always download directly from Microsoft.
Here’s what you can do to bypass the check for pre-release upgrade IF YOU REALLY REALLY NEED TO:
These same steps will be required as we transition from the RC milestone to the RTM milestone.
Again, we know many people (including tens of thousands at Microsoft) are relying on the pre-release builds of Windows 7 for mission critical and daily work, making this step less than convenient. We’re working hard to provide the highest quality release we can and so we’d like to make sure for this final phase of testing we’re supporting the most real world scenarios possible, which incremental build to build upgrades are not. At the same time everyone on the beta has been so great we wanted to make sure we at least offered an opportunity to make your own expert and informed choice about how to handle the upgrade.
We’re always humbled by the excitement around the releases and by the support and enthusiasm from those that choose to run our pre-releases. We’re incredibly appreciative of the time and effort you put into doing so. In return we hope we are providing you with a great release to work with at each stage of the evolution of the product. Our next stop is the RC…see you there!
--Windows 7 Team
PS: At Step 1 above many of you are probably thinking, “hey why don’t you just let me mount the ISO and skip the plastic disc”. We’ve heard this feedback and we deserve the feedback. We don’t have this feature in Windows 7 and we should have. So please don’t fill the comments with this request. There are several third party tools for mounting and if you’ve got a Vista image there’s a good chance your PC came with those tools on it.
One of the areas of any release of Windows that receives a significant amount of testing and scrutiny is the performance of graphics—desktop graphics all the way to the most extreme CAD and game graphics. The amazing breadth of hardware supported for Windows and the broad spectrum of usage scenarios contributes to a vibrant ecosystem with many different goals—from just the basics to the highest frame rates on multiple monitors possible. In engineering Windows 7 we set out to improve the “real world” performance of graphics as well as continue to improve the most extreme elements of graphics. This is work we do in Windows 7 and work our partners do as they work to improve the underlying hardware/software combination through drivers (note: Windows Vista drivers continue to work as they did in Windows Vista, but we've also been working with partners on updated drivers for Windows 7 which many of you have been testing through Windows Update downloads). This post looks at this spectrum of engineering as well as the different ways performance is measured. Ultimately we want to inform you about what we have done in engineering Windows 7, while we leave room for the many forums that will compare and contrast Windows 7 on different hardware and in different scenarios. This post is written by Ameet Chitre, a program manager on our Desktop Graphics feature team. --Steven
If you have gone online to check out or purchase a new PC, you must have noticed that “faster graphics” and “great performance” are always some of the key selling points. People have come to expect faster systems which enable them to edit photos, do email, watch high-definition videos and play the latest 3D games all with greater ease, often shuffling between these tasks seamlessly. Quite a few of these users refer to the enthusiast community blogs and various review sites which run graphics benchmarks and report results evaluating how fast the graphics of new hardware or software performs. Traditionally graphics performance has been measured and analyzed through 3D games but it also impacts what we call “desktop scenarios” - such as when you are using the Windows UI, moving or maximizing windows, or scrolling within Word or IE etc. The performance requirements for these desktop scenarios are quite different from3D games. In fact, this is the reason in Windows Vista Experience Index (WinEI) we give you rating for these two scenarios separately, highlighted in the image below:
Figure 1. WEI sample with Graphics capabilities highlighted.
Graphics performance is usually assessed through many benchmarks. These can be classified into 2 broad categories:
However, there are plenty of things that we all do on our PCs that don’t have benchmarks tracking them that are still quite critical to make fast. In these cases we use the instrumentation within Windows to obtain timing information and then analyze the performance.
This blog entry discusses various aspects of graphics performance - both gaming and desktop graphics performance. It calls out the changes we made in Windows 7 to address user feedback as well as to take advantage of modern hardware to improve graphics performance.
Many have experienced scenarios where an application, or Windows itself, stops responding momentarily. This is type of a performance issue that can be impacted significantly by the performance of graphics in the PC. We categorize these as desktop responsiveness issues. Improving responsiveness, both in real terms and by avoiding non-responsive moments, is one of the key ways that performance is improved in the system. It is also hard to measure.
Measuring desktop responsiveness is a hard problem since a number of issues which affect responsiveness aren’t easily reproducible and there is a great variety of them. They are rarely caught by either kind of benchmark as these issues are dependent on real-world combinations of factors. For Windows 7 we spent a great deal of time looking at these performance glitches using a mechanism in test versions of Windows 7 which has the ability to record key OS events and when they occurred. During real-world testing when we encounter a responsiveness problem, the tester can hit a record key and enter a small description of the issue encountered. The event history with diagnostic information called a “performance trace” is written out to a file and uploaded to a server where a team of performance analysts parse the data to figure out the cause of the responsiveness issue. This process has been successful to the extent that today most responsiveness issues can be quickly tracked down and root-caused.
Using this methodology, we analyzed thousands of desktop responsiveness traces where the tester experienced a frozen desktop anywhere from 100msec to several seconds. The type of issues ranged from an antivirus blocking disk access for all applications while updating itself on the vendor’s website to an application doing network access from a UI thread. In a significant portion of these traces, we found that a GDI application is waiting on another GDI application which is experiencing slowdowns due to excessive paging activity. This was the single-most frequently occurring cause of all desktop responsiveness issues, which without this data we probably would not have assumed. Based on these investigations, we worked to improve the architecture in these two key areas:
These are described in more detail below.
A number of performance traces we investigated in the context of desktop responsiveness pointed us to the design of a key synchronization mechanism in GDI. The performance challenge happens because the design of GDI in Windows Vista allows only a single application to hold a system-wide exclusive global lock. While this seems obvious in hindsight, when this decision was originally made the performance characteristics of different parts of the system made this optimistic implementation perfectly reasonable.
Figure 2. Existing architecture of GDI concurrency.
GDI applications running simultaneously vie for this global lock in order to render on the screen. The application that accesses the global lock prevents other applications from rendering till it releases the global lock. The situation often gets exacerbated when the application that is holding the lock needs to page in a large amount of memory from the disk since moving the memory from the disk to RAM takes a relatively long time. The above picture shows two GDI applications running simultaneously, contending for the global lock. If App X gets hold of the lock, it can render to the screen while App Y is unable to do so and waits for App X to finish.
Figure 3. Windows 7 architecture of GDI concurrency.
The solution to the problem was therefore to reduce the lock contention and improve concurrency by re-architecting the internal synchronization mechanism through which multiple applications can reliably render at the same time. Contention due to the global exclusive lock is avoided by implementing a number of fine-grained locks which are not exclusive but aid parallelism. The increased number of fine-grained locks adds a small overhead for scenarios where only a single application is rendering at a time.
Special attention was paid to GDI application compatibility as changing internal synchronization mechanism in the most widely used API stack could potentially give rise to timing issues such as deadlocks and rendering corruption.
This work also resulted in better rendering performance of concurrent GDI applications on multi-core CPUs. Multi-core Windows PCs benefit from these changes as more than one application can now be rendering at the same time.
After the GDI concurrency work was implemented in the Windows 7 builds leading to the Beta, we saw a large reduction in the number of desktop responsiveness issues reported by our testers which were caused by one application blocking another one due to GDI. To further validate the scalability of our new implementation, we wrote tests that draw 2D GDI primitives and measured the rendering throughput by launching simultaneously multiple such applications. The throughput is measured by adding together the frame rate (FPS) of each application window. Below is a sample of these results on a quad-core CPU system.
Figure 4. GDI Concurrency and Scalability.
Without the Windows 7 GDI concurrency, the rendering throughput of these applications is effectively limited to the performance of a single CPU core. Since only a single application can acquire the global exclusive lock while the others are waiting, this scenario doesn’t benefit from multiple CPU cores. This demonstrates that GDI applications in Windows 7 are now much less dependent on one another. This benefit will not need any new display drivers; it will work on any Vista (WDDM 1.0) and newer display drivers.
Another area which affects system responsiveness is memory usage. Simply put, increased system memory (RAM) usage leads to an increased paging activity which directly leads to reduced system responsiveness. Thus, for the best responsiveness, all applications, processes and OS components need to use as little system memory as possible.
In Windows Vista, the amount of memory required to run multiple windows scales linearly with the number of windows opened on the system. This results in more memory pressure when there are more windows or if the monitors have higher resolution. It gets worse if you have more than one monitor. As part of investigating various means to improve system responsiveness, we saw a great opportunity in reducing the usage of system memory by DWM. In Windows Vista, every GDI application window accounts for two memory allocations which hold identical content – one in video memory and one in system memory. DWM is responsible for composition of the desktop through the graphics hardware. Hence, it requires a copy of the same allocation in video memory, which is easily accessible by the graphics hardware. The duplicate copy present in system memory is required because GDI is being rendered utilizing the CPU completely in the operating system without any assistance or “acceleration” by the graphics hardware. As the CPU performs all the tasks for rendering GDI applications, it requires an easily accessible cacheable copy of memory.
Figure 5. Existing memory allocations.
Windows 7 saves one copy of the memory allocation per application window by getting rid of the system memory copy entirely. Thus, for a GDI application window visible on the desktop, the memory consumed is cut in half.
Figure 6. Windows 7 memory allocations.
We achieved the reduction in system memory by accelerating the common GDI operations through the graphics hardware - the WDDM drivers accelerate these to minimize the performance impact of the CPU read-back of video memory. This was necessary as performing these operations otherwise on the CPU would incur a heavy performance penalty. In order to decide which GDI operations to accelerate, it was important to understand the usage pattern of various GDI applications. We profiled the top 100 GDI applications to learn more about their calling patterns and frequency and nature of the GDI operations.
Figure 7. Calling patterns and frequency of GDI operations for 100 GDI-based applications.
Based on real-world application statistics, a tiny snapshot of which is seen above, we worked with our graphics IHV partners to provide support in their drivers to accelerate the most commonly used GDI operations. Windows 7 systems with these updated drivers, called “WDDM v1.1” will thus benefit from this memory savings work. Please note that WDDM 1.0 drivers continue to function and are fully supported on Windows 7. You might have seen Windows Update providing these 1.1 drivers during the Beta—these drivers are themselves in Beta.
Figure 8. Desktop Window manager memory consumption comparison using WDDM 1.1 v. WDDM 1.0.
The above data shows that the memory savings become more and more pronounced when you have multiple application windows visible on the desktop. Since you save a lot of system memory, the paging activity gets reduced – as a result, your system responsiveness improves for the same workload.
Certain trade-offs had to be made for the desktop responsiveness improvements which benefit a wide range of systems. For example – the elimination of the duplicate system memory copies which “speed up” certain operations introduced slightly reduced performance as the CPU now has to read data back from the video memory. An analysis of real-world application statistics showed that these operations were rare. However, certain GDI micro-benchmarks which issue these operations show some performance degradation. This is something to note if you are running existing benchmarks that stress specific GDI operations repeatedly, which isn’t necessarily a reflection of real-world performance. Our observation has been that these slow-downs do not impact the end-user functionality directly and that the memory savings directly result in Windows 7 being much responsive overall. The improvements overall are definitely noticeable on memory constrained PCs with shared memory graphics.
No article on graphics performance is complete without talking about gaming, which is still the most widely analyzed and discussed aspect of graphics performance. There are a number of popular benchmarks such as 3D Mark as well as in-game benchmarks which are really a mode in which you can run your game where it renders the game scenes and animations without any user interaction. This area has thus been well tracked by the gaming industry through various industry benchmarks, which are pretty realistic and representative of actual games. The different benchmarks and tests are widely discussed and gamers all well-versed in the subtleties of these measurements and translating them into recommendations depending on their hardware, drivers, and gaming expectations.
For Windows 7, we have worked closely with our Graphics IHV partners, helping them improve the WDDM drivers’ gaming performance with specific changes to how Windows 7 works under the hood, while maintaining the same driver model and compatibility. Our continued investments in performance tools has helped us and our IHV partners track down and analyze various gaming performance bottlenecks and fix them in subsequent driver releases. The fundamentals of the Windows Display Driver Model remain unchanged in Windows 7. Some policies around GPU scheduling and memory management were changed to enable better performance in certain scenarios.
Because these benchmarks are very sensitive to the specific hardware, firmware, drivers, and overall system and because these are so widely measured and discussed elsewhere we are going to leave these comparisons to third parties. Like many areas in Windows 7, our commitment is to engineer even better performance across many dimensions. We believe it is better for you to experience these efforts directly. In comparing Windows 7, we would encourage the comparison using Windows Vista SP1 and keep in mind the difference you might see in WDDM 1.0 v. 1.1 and that the 1.1 drivers are still under development.
As you can see, in engineering Windows 7 we have worked hard to improve the architecture for graphics for real-world performance. It benefits both ends of the hardware spectrum – by enabling low physical memory systems to run a leaner and faster Windows and at the same time enabling multi-core PCs render multiple graphics applications much more efficiently.
As mentioned before on this blog (regarding our UAC changes) and on the IE blog (regarding the SmartScreen® filter for malware), we have an increased focus to enable customers to be in control and feel confident about the software that they choose to run on their computers. Folks on this blog have also commented about the concerns they have specifically in the AutoPlay area. This blog entry addresses some of the changes that we have made to increase customer confidence when using their media and devices with Windows. It is authored by Arik Cohen, a program manager on the Core User Experience team. –Steven [Note: There was a technical problem so this post was reposted in its entirety.]
Certain malware, including the Conficker worm, have started making use of the capabilities of AutoRun to provide a seemingly benign task to people – which masquerades as a Trojan Horse to get malware onto the computer. The malware then infects future devices plugged into that computer with the same Trojan Horse. For further information about Conficker please visit http://www.microsoft.com/protect/computer/viruses/worms/conficker.mspx
In the following example for a USB flash drive that has photos, malware registers as the benign task of “Open folders to view files.” If you select the first “Open folders to view files” (circled in red), you would be running malware. However, if you select the second task (circled in green), you would be safe running the Windows task.
Infected USB AutoPlay
People are confused why they have two tasks that appear to do the same thing – and even a knowledgeable person who is careful not to run software from an untrusted source can easily make the mistake of selecting the first task. As a result, people lose confidence and don’t feel in control.
A growing attack
While presenting an AutoRun task in AutoPlay has been available since Windows XP, we have seen a marked increase in the amount of malware that is using AutoRun as a potential method of propagation. According to the Security Intelligence Report, an enterprise study by Forefront Client Security found that the category of malware that can propagate via AutoRun accounted for 17.7% of infections in the second half of 2008 – the largest single category of malware infections.
The chart below shows the increasing amount of detection reports by Microsoft anti-virus software of the class of infections that spread via AutoRun. (Note: The actual method of infection cannot be determined.)
Infection Detections of Malware that Spread via AutoRun
Infection Detections of Malware that Spread via AutoRun
Currently, disabling AutoPlay completely is the only solution for consumers and enterprises to gain confidence with the use of USB flash devices on their computer. Guidance on disabling AutoPlay is available here.
Increasing customer confidence
Windows 7 introduces key changes to AutoPlay that keep you from being exposed inadvertently to malware like Conficker when doing your common scenarios with devices (e.g., get to the files on your USB flash drive, download pictures from an SD card, etc.).
In particular, Windows will no longer display the AutoRun task in the AutoPlay dialog for devices that are not removable optical media (CD/DVD.) because there is no way to identify the origin of these entries. Was it put there by the IHV, a person, or a piece of malware? Removing this AutoRun task will block the current propagation method abused by malware and help customers stay protected. People will still be able to access all of the other AutoPlay tasks that are installed on their computer.
With these changes, if you insert a USB flash drive that has photos and has been infected by malware, you can be confident that the tasks displayed are all from software already on your computer:
Infected USB AutoPlay after AutoPlay changes
Infected USB AutoPlay after AutoPlay changes
On the other hand, if you insert a CD that offers software to install, Windows will still display the AutoRun task provided by the ISV during their media creation process. For example:
AutoPlay for a CD that offers an AutoRun Task
AutoPlay for a CD that offers an AutoRun Task
You will first see this updated AutoRun experience in the Windows 7 RC build, and we will be bringing this change to Vista and XP in the future.
We are working with our ecosystem partners to help mitigate situations where this AutoRun change will have an impact on them.
CDs and DVDs (including CD emulation), where the IHV specified AutoRun task authored during manufacturing, will continue to provide the AutoRun choice allowing customers to run the specified software. IHVs of generic mass storage devices should expect that people will browse the contents of the device to launch any software. The new behavior will allow customers to continue to use AutoPlay (including all Windows and ISV installed tasks) to access their media and devices while not being presented with tasks from malware. Additionally, device classes, such as portable media players and cell phones, now support Device Stage™ on Windows 7. Device Stage offers the IHV a multifunction alternative to AutoPlay where they can present links to software and common tasks, and provides additional features as you use the device.
As you try out the Windows 7 RC, we hope these changes will make you feel more confident and in control when using your media and devices.
There’s a strong community of developers who take advantage of the ink input/TabletPC functionality to develop unique solutions for specific markets (medicine, education, line of business) and create software in Windows that builds on this experience to streamline how these end-users interact with information on their PC (usually with unique form-factors such as slates or wall mounted PCs). Earlier this week I received a great email asking “what’s new for us” from the head of development for one such ISV (medical software) and so we put together an overview of the new functionality. Several Program Managers on the team authored this post.
Also, as you have noticed, the site has had some uptime troubles over the past 10 days or so and I think we’re all back to normal. That’s ok since we’ve also been pretty busy in the Windows 7 hallway :-) --Steven
Hi, my name is Jan-Kristian and I’m a Program Manager on the Core User Experience team for Windows 7. One of my focus areas is pen and touch text input, and I’d like to share some of the exciting things we have been working on.
The Tablet PC Input Panel, what we often called the TIP for short, is the tool to insert text using handwriting into any Windows application. It also has a soft-keyboard you can use for text entry. The Input Panel has been around since the first version of Windows XP Tablet PC Edition, and we’ve made steady improvements to the user experience in each version.
Our goal with the TIP is to make it as light-weight as possible so you can think about what you are writing and not how you are doing it. We received a lot of positive feedback on the improvements we did to the Input Panel in Windows Vista, but there were still areas that caused confusion or took more steps than necessary.
Windows Vista Input Panel – The handwriting recognition results are shown as small text bubbles under the writing surface. To verify recognition you need to look down at the bubbles, if you see an error you then tap on a bubble to bring up a secondary window for correction.
Based on analysis of our telemetry from Vista and usability tests we focused on two significant areas of improvement for Windows 7:
To achieve these goals we needed to make fundamental changes to the writing pad. As we explored different ideas we decided on a model where ink was converted in-place to text as the user was writing. Although this sounds like a straightforward UI model, there were a lot of open questions on what the right behavior should be: when do we convert, how big should the text be, what font should we use… The only way to make sure we created a natural and efficient handwriting experience was to get real user feedback. We utilized the RITE (Rapid Iterative Testing and Evaluation) method. RITE testing is a cycle-based usability method that was developed at Microsoft as part of usability testing of the Age of Empires II game. For each cycle you try to make small improvement to the user experience and then you re-test to see how well it worked. We went through roughly 20 cycles before we had a design that we felt was ready to be documented.
One of the most important things we adjusted during RITE testing was the timing for the automatic ink to text conversion. Converting too early or too late would break the user experience; to get this right we had to do a lot of behind the scenes work. Our final solution is a combination of a distance trigger (automatically adapting to the user’s average word spacing), recognizer-result-based trigger, and a time-based trigger. Another factor was the text size, in the end we use dynamic sizing to closely match the size of the handwriting.
The new text-based UI in the writing surface allows you to get to the text they wanted faster. Having a single representation of the text makes the experience less complex and reduces the height of the Input Panel. Using text instead of ink makes the writing surface much more flexible as we can move the text around as much as we want – inserting a word between two words is now as simple as just starting to write in the space and we will auto-grow the space as much as needed.
With the ink to text conversion working we needed a correspondingly natural way of editing recognized text. Gestures seemed to be the perfect solution for this - we were creating a pen-based UI, so we should use the pen. We limited ourselves to a small number of gestures: delete, split (add space), and join. We collected samples of how people would perform these three actions on paper. Based on our collected data we then created our gestures. To make the gestures discoverable, we added the “gesture panel”, which is an interactive “cheat sheet” in the title bar of the Input Panel.
Let’s take a look at how this all comes together in the new Windows 7 writing surface [Ed. Note, used YouTube with Windows Live Photo Gallery on Windows 7]:
Writing Pad: The new writing pad in action, animation is used to provide meaningful transitions so that the user can easily see the result of their actions.
Our telemetry showed us that corrections were one of the more painful parts of using the TIP in Vista, to correct a word you often had to rewrite all of the characters. In Windows 7, we leveraged work from Microsoft Research to design the Smart Correction feature to make word corrections much faster. Now you just start correcting a word left-to-right and Windows performs a new recognition every time you enter a character. This constrained recognition will almost always give you the desired result within a few character corrections.
Smart Corrections: “worked” is auto-corrected to “wonderful” with just a single character change. All you have to do is start correcting the word from the left and it will keep updating until you get the word you want.
One extra writing pad feature worth mentioning: our instrumentation data showed that the most used applications with the Input Panel are web browsers, and when you are browsing one of the main scenarios is to enter URLs.
Entering URLs: The flexibility of the new writing pad makes entering URLs easy by pre-populating parts of the URL
Notice how the different URL segments are separated and all have alternates that make sense. The alternates are based on what you use most frequently, so if you choose “.net” a lot then that will become the top alternate and set by default in the URL template. The “Insert” button also changed to “Go to” to let the user insert the URL and navigate to it with a single click.
The Input Panel also has a soft-keyboard available which is great for the Pen or Touch. Some of the updates we made might seem like only visual changes, but they were very deliberate and have a big impact on the usability of the touch keyboard. For example, touch screen PCs are often used in mobile situations, we had to be very careful with the color and contrast of the key labels to make sure they are visible on a dimmed screen or in less than optimal light conditions.
The Windows 7 Touch Keyboard
One of the challenges with using a touch based keyboard is the lack of tactile feedback. Coupled with this is the fact that user’s fingers cover keys as they are being tapped. How does the user know that they hit the right key (or even hit a key at all) when they are covering the key with their finger? If a user has to switch focus between the text field and the touch keyboard for every key press they will quickly tire of typing using touch. We wanted to give the user a simple little nudge; “we heard you”, and “yes, you just hit this key”. Our solution was to let the released key have a short glow fade-out effect. This glow feedback gives the user a subtle confirmation that they hit the key they wanted (or not).
The keyboard now supports multitouch so you can press “ctrl+c” or “shift+a” etc. We also enabled key rollover, meaning you can press another key before you finger has lifted off the previous one – this enables a much more natural typing experience. Don’t worry, if you prefer the sticky modifier-key model, where you press “Ctrl” then press “c”, it is still supported.
Hi, my name is Jen and I’m a Program Manager on the Tablet PC Handwriting Recognition team. In our previous post you heard from my co-worker Yvonne about new handwriting recognizers. I’m going to talk about some of the new features that leverage or augment our recognizers including Predictive Text and Personalization as well as our new East Asian Recognizers.
One of our goals on the Tablet PC team is to make it as efficient as possible to enter text when using your keyboard is inconvenient or impractical. To achieve this we’ve made investments in the TIP, we’ve improved overall handwriting recognition accuracy, and we’re leveraging some of these same technologies to deliver Text Prediction on the soft keyboard. Text Prediction offers possible completions for the word being entered, and may offer suggestions for next words or phrases.
Text Prediction – Here I am trying to input the word “Microsoft” using the US English soft keyboard. After entering the first two letters “Mi”, the word “Microsoft” is proposed as the first option. I can then select this option and have the word “Microsoft” inserted into a document.
In Windows 7, we support Text Prediction for English, French, Italian, German, and Spanish using the soft keyboard as well as for Traditional Chinese and Simplified Chinese using handwriting in character by character mode. This section will focus on the Latin based languages; examples for Chinese can be found in the following section.
As we developed Text Prediction, our primary goal was to speed up user input. To do so, we had to make sure predictions were relevant. Our recognizers use a lexicon to improve recognition accuracy. The system lexicon ships with the recognizer and is a fancy name for a word list of the most commonly used words in a given language. Using this lexicon, the out-of-the-box predictions are good, but by including additional user-specific words (your words), we can improve accuracy significantly. This is where Text Harvesting comes in.
In Windows Vista, we shipped Text Harvesting (or Automatic Learning) for US English and UK English to improve handwriting recognition. In Windows 7, this feature will be available for all languages. It allows us to automatically extend the system lexicon based on the words you type when writing email. Text Harvesting is done on a per user basis, so your data is specific to you. From the results of Text Harvesting, we build a new lexicon containing your specific vocabulary and also increase the probability of words already you commonly use, this is use for both handwriting recognition and text prediction. The results are impressive, after augmenting the lexicon with your words and usage patterns, prediction can seem almost magical in its ability to predict which words you are going to enter next!
Windows 7 also includes support for Custom Dictionaries, these are specialized word lists that can be added to the system. Companies can develop domain specific dictionaries, such as for medicine, chemistry etc. and add them to the system – predicting acetaminophen is a lot faster than typing it!
Significant improvements were made to handwriting recognition on the four East Asian languages we support: Traditional Chinese, Simplified Chinese, Korean and Japanese. For many people, handwriting is an efficient input method for these languages due to the large character set.
There are two input modes for East Asian handwriting: character by character mode (or box mode) and freestyle mode (or line mode). In box mode, you input a single character at a time into a fixed width box. In freestyle mode, you write the characters cursively on a line and do not have to concentrate on spacing. Which mode you chose depends on your preference; box mode is slightly more constrained but has text prediction and is more accurate, whereas line mode is closer to natural handwriting.
Traditional Chinese in Line Mode – The top pane contains the user’s writing and the bottom pane contains the recognized text.
In addition to core accuracy across all these languages improvements, we also use personalization to improve the user experience. One method of personalization is to adapt to how you write using the Shape Collector. The Shape Collector is a Wizard that allows you to train handwriting recognition on your individual handwriting style. For the four East Asian Languages, you can use the Shape Collector in a “troubleshooting” mode to improve recognition of a specific character or word, or to add a character or word that is unsupported.
We also learn as you write and correct mistakes. If you write a character and it is initially misrecognized, you can view the alternates list and select the character you intended. We will learn based on this action, and be more likely to provide that as the first choice the next time you write the character.
In Windows 7, Simplified Chinese and Traditional Chinese also support Text Prediction in box mode. For these languages, Text Prediction is especially valuable as writing individual characters can be time consuming. The user writes in character by character mode and is provided with options to complete their word or phrase without having to write the whole thing. In the case below, the user wants to input 中华人民共和国 and only has to input the first two characters (中华) to get the desired text as a prediction. The gray words represent what has been entered and the black shows the predictions.
Notice in this example that Text Prediction is working on both characters together (中华) as well as just the second character (华). As with the other languages, Text Prediction also works with user-specific vocabulary. If a user inputs the same words multiple times, the recognizer will learn this behavior.
We have made significant improvements to Traditional Chinese, Simplified Chinese, Japanese, and Korean handwriting recognition since Windows Vista. These were based on improving our overall customer experience by improving accuracy and throughput. Our goal is to give customers a more natural way to input in these languages and a viable alternative to keyboard use.
Have you ever written a math paper in Word or performed calculations in Mathematica, and spent hours creating equations using a multitude of buttons or a complex linear format, thinking: “Oh, what I wouldn’t give for an easy-to-use input method?” Well, your wishes have just come true, in addition to improving handwriting in Windows 7, we have also invested in recognizing ink drawn math equations.
Hi, my name is Marko and I am the Program Manager for math recognition in Windows. Let me show you the Math Input Panel that provides you with the most natural and efficient way of math input: handwriting recognition. We have taken a completely new approach to this problem and raised math handwriting recognition to a whole new level in terms of functionality, performance and area coverage.
The Math Input Panel (or MIP) is designed to be used with a tablet pen on a Tablet PC, but you can use it with any input device such as a touchscreen, external digitizer or even a mouse. MIP outputs the recognition result via the clipboard in MathML format, a standardized mathematical markup language. Any equation you write and recognize in MIP reaches your destination application in a completely editable form – you can insert and edit the output as you would edit any text.
We spent a lot of time researching and identifying as many areas of math as possible and endless different math notations. The final result is a great coverage of high school and college level math, and of even more advanced areas.
Math Input Panel – Have you heard of the formula recognized in the example above? You haven’t J, well, it’s the Schwarz formula used in complex analysis!
Using MIP is really simple and straightforward. You write the well-formed math expression (this means that will not get recognized, whereas will) just as you do with pencil and paper and the recognizer takes over. The recognized math is shown in the preview area. As no recognizer is perfect, the great power of MIP lies in its ability to provide a fantastic correction experience (let’s be honest, sometimes even a human is not sure what has been written – you should see my handwriting!).
In case your handwritten math is misrecognized, you can select any part of it (symbols or whole sub-structures) and correct it either by selecting an alternate from a drop-down list or by rewriting part of the expression. Usually fixing one part of the equation automatically fixes the rest, in just one step.
Math Input Panel Correction
All you have to do now is tap Insert and you have just painlessly created an equation in your word-processing or computational program.
There are many other cool features like History, moving ink around, and dragging and dropping ink into MIP from other inking applications such as OneNote, all of which you can explore on your own. For software developers, the MIP can be embedded into your applications – check out the documentation on MSDN.