Engineering Windows 7 : Input

Ink Input and Tablet PC

e7blog — Thu, 23 Apr 2009 10:00:00 GMT

There’s a strong community of developers who take advantage of the ink input/TabletPC functionality to develop unique solutions for specific markets (medicine, education, line of business) and create software in Windows that builds on this experience to streamline how these end-users interact with information on their PC (usually with unique form-factors such as slates or wall mounted PCs). Earlier this week I received a great email asking “what’s new for us” from the head of development for one such ISV (medical software) and so we put together an overview of the new functionality. Several Program Managers on the team authored this post.

Also, as you have noticed, the site has had some uptime troubles over the past 10 days or so and I think we’re all back to normal. That’s ok since we’ve also been pretty busy in the Windows 7 hallway :-) --Steven

Tablet PC Input Panel

Hi, my name is Jan-Kristian and I’m a Program Manager on the Core User Experience team for Windows 7. One of my focus areas is pen and touch text input, and I’d like to share some of the exciting things we have been working on.

The Tablet PC Input Panel, what we often called the TIP for short, is the tool to insert text using handwriting into any Windows application. It also has a soft-keyboard you can use for text entry. The Input Panel has been around since the first version of Windows XP Tablet PC Edition, and we’ve made steady improvements to the user experience in each version.

The new Writing Pad

Our goal with the TIP is to make it as light-weight as possible so you can think about what you are writing and not how you are doing it. We received a lot of positive feedback on the improvements we did to the Input Panel in Windows Vista, but there were still areas that caused confusion or took more steps than necessary.

Windows Vista Input Panel – The handwriting recognition results are shown as small text bubbles under the writing surface. To verify recognition you need to look down at the bubbles, if you see an error you then tap on a bubble to bring up a secondary window for correction.

Based on analysis of our telemetry from Vista and usability tests we focused on two significant areas of improvement for Windows 7:

Simplify the experience – Handwriting should be an easy, natural, flowing experience. What we found though, was that using the TIP caused a “high cognitive load”, which means you have to think too much about what you are doing. Your eyes needed to dart back and forth between what you’ve written and the little bubbles down below and corrections meant entering another mode and even then often meant rewriting the whole word. Our goal was to simplify this and make it less taxing.
Add flexibility – We’re all accustomed to the flexibility of using a mouse and keyboard for input. Handwriting with the Windows Vista Input Panel had minimal flexibility because the ink-based model made it hard to edit a sentence once it was written – there was no way to insert more text between words, or to easily replace words. Our goal here was to bring the editing experience more in line with what you are used to with the mouse and keyboard while exploiting the power of the pen.

Creating a new model

To achieve these goals we needed to make fundamental changes to the writing pad. As we explored different ideas we decided on a model where ink was converted in-place to text as the user was writing. Although this sounds like a straightforward UI model, there were a lot of open questions on what the right behavior should be: when do we convert, how big should the text be, what font should we use… The only way to make sure we created a natural and efficient handwriting experience was to get real user feedback. We utilized the RITE (Rapid Iterative Testing and Evaluation) method. RITE testing is a cycle-based usability method that was developed at Microsoft as part of usability testing of the Age of Empires II game. For each cycle you try to make small improvement to the user experience and then you re-test to see how well it worked. We went through roughly 20 cycles before we had a design that we felt was ready to be documented.

One of the most important things we adjusted during RITE testing was the timing for the automatic ink to text conversion. Converting too early or too late would break the user experience; to get this right we had to do a lot of behind the scenes work. Our final solution is a combination of a distance trigger (automatically adapting to the user’s average word spacing), recognizer-result-based trigger, and a time-based trigger. Another factor was the text size, in the end we use dynamic sizing to closely match the size of the handwriting.

The new text-based UI in the writing surface allows you to get to the text they wanted faster. Having a single representation of the text makes the experience less complex and reduces the height of the Input Panel. Using text instead of ink makes the writing surface much more flexible as we can move the text around as much as we want – inserting a word between two words is now as simple as just starting to write in the space and we will auto-grow the space as much as needed.

With the ink to text conversion working we needed a correspondingly natural way of editing recognized text. Gestures seemed to be the perfect solution for this - we were creating a pen-based UI, so we should use the pen. We limited ourselves to a small number of gestures: delete, split (add space), and join. We collected samples of how people would perform these three actions on paper. Based on our collected data we then created our gestures. To make the gestures discoverable, we added the “gesture panel”, which is an interactive “cheat sheet” in the title bar of the Input Panel.

Let’s take a look at how this all comes together in the new Windows 7 writing surface [Ed. Note, used YouTube with Windows Live Photo Gallery on Windows 7]:

<\/param><\/embed><\/object><\/div>";" mce_src="http://blogs.msdn.com/blogfiles/e7/WindowsLiveWriter/InkInputandTablet_12FFC/videof1b202e2ceb0.jpg" galleryimg="no">

Windows 7 - Handwriting overview

Writing Pad: The new writing pad in action, animation is used to provide meaningful transitions so that the user can easily see the result of their actions.

Smart Corrections

Our telemetry showed us that corrections were one of the more painful parts of using the TIP in Vista, to correct a word you often had to rewrite all of the characters. In Windows 7, we leveraged work from Microsoft Research to design the Smart Correction feature to make word corrections much faster. Now you just start correcting a word left-to-right and Windows performs a new recognition every time you enter a character. This constrained recognition will almost always give you the desired result within a few character corrections.

<\/param><\/embed><\/object><\/div>";" mce_src="http://blogs.msdn.com/blogfiles/e7/WindowsLiveWriter/InkInputandTablet_12FFC/videodff1aa808d4c.jpg" galleryimg="no">

Windows 7 - Handwriting Smart Correction

Smart Corrections: “worked” is auto-corrected to “wonderful” with just a single character change. All you have to do is start correcting the word from the left and it will keep updating until you get the word you want.

Entering URLs

One extra writing pad feature worth mentioning: our instrumentation data showed that the most used applications with the Input Panel are web browsers, and when you are browsing one of the main scenarios is to enter URLs.

<\/param><\/embed><\/object><\/div>";" mce_src="http://blogs.msdn.com/blogfiles/e7/WindowsLiveWriter/InkInputandTablet_12FFC/videoa4820face49a.jpg" galleryimg="no">

Windows 7 Handwriting - URL correction

Entering URLs: The flexibility of the new writing pad makes entering URLs easy by pre-populating parts of the URL

Notice how the different URL segments are separated and all have alternates that make sense. The alternates are based on what you use most frequently, so if you choose “.net” a lot then that will become the top alternate and set by default in the URL template. The “Insert” button also changed to “Go to” to let the user insert the URL and navigate to it with a single click.

Touch Keyboard

The Input Panel also has a soft-keyboard available which is great for the Pen or Touch. Some of the updates we made might seem like only visual changes, but they were very deliberate and have a big impact on the usability of the touch keyboard. For example, touch screen PCs are often used in mobile situations, we had to be very careful with the color and contrast of the key labels to make sure they are visible on a dimmed screen or in less than optimal light conditions.

The Windows 7 Touch Keyboard

One of the challenges with using a touch based keyboard is the lack of tactile feedback. Coupled with this is the fact that user’s fingers cover keys as they are being tapped. How does the user know that they hit the right key (or even hit a key at all) when they are covering the key with their finger? If a user has to switch focus between the text field and the touch keyboard for every key press they will quickly tire of typing using touch. We wanted to give the user a simple little nudge; “we heard you”, and “yes, you just hit this key”. Our solution was to let the released key have a short glow fade-out effect. This glow feedback gives the user a subtle confirmation that they hit the key they wanted (or not).

The keyboard now supports multitouch so you can press “ctrl+c” or “shift+a” etc. We also enabled key rollover, meaning you can press another key before you finger has lifted off the previous one – this enables a much more natural typing experience. Don’t worry, if you prefer the sticky modifier-key model, where you press “Ctrl” then press “c”, it is still supported.

Predictive Text and Handwriting Personalization

Hi, my name is Jen and I’m a Program Manager on the Tablet PC Handwriting Recognition team. In our previous post you heard from my co-worker Yvonne about new handwriting recognizers. I’m going to talk about some of the new features that leverage or augment our recognizers including Predictive Text and Personalization as well as our new East Asian Recognizers.

Text Prediction

One of our goals on the Tablet PC team is to make it as efficient as possible to enter text when using your keyboard is inconvenient or impractical. To achieve this we’ve made investments in the TIP, we’ve improved overall handwriting recognition accuracy, and we’re leveraging some of these same technologies to deliver Text Prediction on the soft keyboard. Text Prediction offers possible completions for the word being entered, and may offer suggestions for next words or phrases.

Text Prediction – Here I am trying to input the word “Microsoft” using the US English soft keyboard. After entering the first two letters “Mi”, the word “Microsoft” is proposed as the first option. I can then select this option and have the word “Microsoft” inserted into a document.

In Windows 7, we support Text Prediction for English, French, Italian, German, and Spanish using the soft keyboard as well as for Traditional Chinese and Simplified Chinese using handwriting in character by character mode. This section will focus on the Latin based languages; examples for Chinese can be found in the following section.

As we developed Text Prediction, our primary goal was to speed up user input. To do so, we had to make sure predictions were relevant. Our recognizers use a lexicon to improve recognition accuracy. The system lexicon ships with the recognizer and is a fancy name for a word list of the most commonly used words in a given language. Using this lexicon, the out-of-the-box predictions are good, but by including additional user-specific words (your words), we can improve accuracy significantly. This is where Text Harvesting comes in.

In Windows Vista, we shipped Text Harvesting (or Automatic Learning) for US English and UK English to improve handwriting recognition. In Windows 7, this feature will be available for all languages. It allows us to automatically extend the system lexicon based on the words you type when writing email. Text Harvesting is done on a per user basis, so your data is specific to you. From the results of Text Harvesting, we build a new lexicon containing your specific vocabulary and also increase the probability of words already you commonly use, this is use for both handwriting recognition and text prediction. The results are impressive, after augmenting the lexicon with your words and usage patterns, prediction can seem almost magical in its ability to predict which words you are going to enter next!

Windows 7 also includes support for Custom Dictionaries, these are specialized word lists that can be added to the system. Companies can develop domain specific dictionaries, such as for medicine, chemistry etc. and add them to the system – predicting acetaminophen is a lot faster than typing it!

Improvements in East Asian Handwriting Recognition

Significant improvements were made to handwriting recognition on the four East Asian languages we support: Traditional Chinese, Simplified Chinese, Korean and Japanese. For many people, handwriting is an efficient input method for these languages due to the large character set.

There are two input modes for East Asian handwriting: character by character mode (or box mode) and freestyle mode (or line mode). In box mode, you input a single character at a time into a fixed width box. In freestyle mode, you write the characters cursively on a line and do not have to concentrate on spacing. Which mode you chose depends on your preference; box mode is slightly more constrained but has text prediction and is more accurate, whereas line mode is closer to natural handwriting.

Traditional Chinese in Line Mode – The top pane contains the user’s writing and the bottom pane contains the recognized text.

In addition to core accuracy across all these languages improvements, we also use personalization to improve the user experience. One method of personalization is to adapt to how you write using the Shape Collector. The Shape Collector is a Wizard that allows you to train handwriting recognition on your individual handwriting style. For the four East Asian Languages, you can use the Shape Collector in a “troubleshooting” mode to improve recognition of a specific character or word, or to add a character or word that is unsupported.

We also learn as you write and correct mistakes. If you write a character and it is initially misrecognized, you can view the alternates list and select the character you intended. We will learn based on this action, and be more likely to provide that as the first choice the next time you write the character.

In Windows 7, Simplified Chinese and Traditional Chinese also support Text Prediction in box mode. For these languages, Text Prediction is especially valuable as writing individual characters can be time consuming. The user writes in character by character mode and is provided with options to complete their word or phrase without having to write the whole thing. In the case below, the user wants to input 中华人民共和国 and only has to input the first two characters (中华) to get the desired text as a prediction. The gray words represent what has been entered and the black shows the predictions.

Notice in this example that Text Prediction is working on both characters together (中华) as well as just the second character (华). As with the other languages, Text Prediction also works with user-specific vocabulary. If a user inputs the same words multiple times, the recognizer will learn this behavior.

We have made significant improvements to Traditional Chinese, Simplified Chinese, Japanese, and Korean handwriting recognition since Windows Vista. These were based on improving our overall customer experience by improving accuracy and throughput. Our goal is to give customers a more natural way to input in these languages and a viable alternative to keyboard use.

Math Handwriting Recognition

Have you ever written a math paper in Word or performed calculations in Mathematica, and spent hours creating equations using a multitude of buttons or a complex linear format, thinking: “Oh, what I wouldn’t give for an easy-to-use input method?” Well, your wishes have just come true, in addition to improving handwriting in Windows 7, we have also invested in recognizing ink drawn math equations.

Hi, my name is Marko and I am the Program Manager for math recognition in Windows. Let me show you the Math Input Panel that provides you with the most natural and efficient way of math input: handwriting recognition. We have taken a completely new approach to this problem and raised math handwriting recognition to a whole new level in terms of functionality, performance and area coverage.

The Math Input Panel (or MIP) is designed to be used with a tablet pen on a Tablet PC, but you can use it with any input device such as a touchscreen, external digitizer or even a mouse. MIP outputs the recognition result via the clipboard in MathML format, a standardized mathematical markup language. Any equation you write and recognize in MIP reaches your destination application in a completely editable form – you can insert and edit the output as you would edit any text.

We spent a lot of time researching and identifying as many areas of math as possible and endless different math notations. The final result is a great coverage of high school and college level math, and of even more advanced areas.

Math Input Panel – Have you heard of the formula recognized in the example above? You haven’t J, well, it’s the Schwarz formula used in complex analysis!

Using MIP is really simple and straightforward. You write the well-formed math expression (this means that will not get recognized, whereas will) just as you do with pencil and paper and the recognizer takes over. The recognized math is shown in the preview area. As no recognizer is perfect, the great power of MIP lies in its ability to provide a fantastic correction experience (let’s be honest, sometimes even a human is not sure what has been written – you should see my handwriting!).

In case your handwritten math is misrecognized, you can select any part of it (symbols or whole sub-structures) and correct it either by selecting an alternate from a drop-down list or by rewriting part of the expression. Usually fixing one part of the equation automatically fixes the rest, in just one step.

Math Input Panel Correction

All you have to do now is tap Insert and you have just painlessly created an equation in your word-processing or computational program.

There are many other cool features like History, moving ink around, and dragging and dropping ink into MIP from other inking applications such as OneNote, all of which you can explore on your own. For software developers, the MIP can be embedded into your applications – check out the documentation on MSDN.

Recognizing Improvements in Windows 7 Handwriting

e7blog — Mon, 09 Feb 2009 11:00:00 GMT

Microsoft has been working on handwriting recognition for over 15 years going back to the Pen extensions for Windows 3.0. With the increased integration and broad availability of the handwriting components present in Windows Vista we continue to see increased use of handwriting with Windows PCs. We see many customers using handwriting across a wide variety of applications including schools, hospitals, banking, insurance, government, and more. It is exciting to see this natural form of interaction used in new scenarios. Of course one thing we need to continue to do is improve the quality of recognition as well as the availability of recognizers in more languages around the world. In this post, Yvonne, a Program Manager on our User Interface Platform team, provides a perspective on engineering new recognizers and recognition improvements in Windows 7. --Steven

Hi, my name is Yvonne and I’m a Program Manager on the Tablet PC and Handwriting Recognition team. This post is about the work we’ve done to improve recognition in handwriting for Windows 7.

Microsoft has invested in pen based computing since the early 1990s and with the release of Windows Vista handwriting recognizers are available for 12 languages, including USA, UK, German, French, Spanish, Italian, Dutch, Brazilian Portuguese, and Chinese (Simplified and Traditional), Japanese and Korean. Customers frequently ask us when we plan to ship more languages and why a specific language is not yet supported. We are planning to ship new and improved languages for Windows 7, including Norwegian, Swedish, Finnish, Danish, Russian, and Polish, and the list continues to grow. Let’s explore what it takes to develop new handwriting recognizers.

Windows has true cursive handwriting recognition, you don’t need to learn to write in a special way – in-fact, we’ve taught (or “trained” as we say) Windows the handwriting styles of thousands of people and Windows learns more about your style as you use it. Over the last 16 years we’ve developed powerful engines for recognizing handwriting, we continue to tune these to make them more accurate, faster and to add new capabilities, such as the ability to learn from you in Vista. Supporting a new language is much more than adding new dictionaries – each new language is a major investment. It starts with collecting native handwriting, next we analyze the data and go through iterations of training and tuning, and finally the system gets to you and continues to improve as you use it.

Data Collection

The development of a new handwriting recognizer starts with a huge data collection effort. We collect millions of words and characters of written text from tens of thousands of writers from all around the world.

Before I describe our collection efforts, I would like to answer a question we are frequently asked: “Why can’t you just use an existing recognizer with a new dictionary?” One reason is that some languages have special characters or accents. But the overriding reason is because people in different regions of the world learn to write in different ways, even between countries with the same language like the UK and US. Characters that may look visually very similar to you can actually be quite different to the computer. This is why we need to collect real world data that captures exactly how characters, punctuation marks and other shapes are written.

Setting up a data collection effort is challenging and time consuming because we want to ensure that we collect the “right kind of data”. We carefully choose our collection labs in the respective countries for which we develop recognizers.

Before we start our data collection in the labs, we configure our collection tools, prepare documentation, and compile language scripts that will guide our volunteers through the collection process. Our scripts are carefully prepared by native speakers in the respective language to ensure that we collect only orthographically correct data, data from different writing styles, and data that covers all characters, numbers, symbols and signs that are relevant to a specific language. All of our scripts are proofread and edited before they are blessed to be used at the collection labs.

Once our tools and scripts are ready, we open our labs and start to recruit volunteers to donate their handwriting samples. Our recruitment efforts ensure that we have balanced demographics such as gender, age, left handiness, and educational background that represent the majority of the population for that country.

A supervisor at the lab instructs the volunteers to copy the text as it is displayed in the collection tool in their own writing style. What is important to note is that we want to collect writing samples that accurately represent the person’s natural way of writing. We therefore encourage volunteers to treat “pen and tablet” like “pen and paper”. If one of the volunteers tends to writes in big, curvy strokes, then we want to collect his/her big, curvy strokes during the collection session. High quality data in this context refers to data that was naturally written.

Here is a snapshot of what our collection tool looks like:

Figure 1: Collection Tool

A collection session lasts between 60-90 minutes at which point our volunteer has donated a significant amount of handwritten data without feeling fatigued. The donated data is then uploaded and stored in our database at Microsoft ready for future use. The written samples contain important information like stroke orders, start- and end points, spacing, and other characteristics that are essential to train our new recognizer.

Let’s take a look at some of our samples in our database to illustrate the great variation among ink samples:

Figure 2: Ink samples illustrating different stroke orders.

The screenshot shows how three different volunteers inked the word “black”. The different colors are used to illustrate the exact stroke orders in which the word was written. Our first two volunteers used five strokes to write the word “black”; our third volunteer used four strokes. Please also note how our third volunteer used one stroke only to ink the letters “ck”, while our first volunteer used three strokes for the same combination of letters. All of this information is used to train our recognizers.

Neural Network and Language Model

Once we have collected a sufficient amount of inked data, we split our data into a training set, used by our development team, and a “blind” set, used by our test team. The training set is then employed to train the Neural Network, which is largely responsible for the magic that is taking place during the recognition process. Good, naturally written data is essential in developing a high quality recognizer; the recognizer can’t be any better than its training set. The more high quality data we feed into our Neural Network, the more equipped we are to handle sloppy cursive handwriting.

Our Neural Network is a Time-Delay Neural Network (TDNN) that can handle connected letters of cursive scripts. A TDNN takes ink segments of preceding and following stroke segments into consideration when computing the probabilities of letters, digits and characters for each segment of ink. The output of the TDNN is powerful but not good enough when handwriting is sloppy. In order to come within reach of human recognition accuracy, we have to employ information that goes beyond the shape of the letter: we call this the Language Model context. The majority of this Language Model context comes in form of the lexicon, which is a wordlist of valid spellings for a given language. For many languages, this is the same lexicon that the spellchecker uses. The TDNN and the lexicon work closely together to compute word probabilities and output the top suggestions for the given input.

Training the Neural Network is an involved process that takes time. We often experiment with borrowing data from other languages to increase the size of the training data with the ultimate goal to boost recognition accuracy. Borrowing characters from other languages does not always lead to success. As I mentioned above, stroke order, letter shape, writing styles and letter size can differ significantly from country to country and can have a negative impact on the performance of the TDNN. It often takes us several rounds of training, re-training and tuning before we find “the right formula” that will lead to high recognition accuracy.

How do we know if we are headed in the right direction when we build a new recognizer? This is an important question that the test team and native speakers answer for us. The test team is responsible for generating our recognition accuracy metrics that reflect how good our recognizer is. These accuracy metrics are based on our blind test set which is the collected data that development could not use for training. In addition to our accuracy metrics, we work with native speakers in house and at our world-wide subsidiaries to get feedback and further input.

Improving the recognizers through personalization

In the previous paragraphs I have outlined how we develop high quality recognizers that can handle a wide variety of different writing styles. But there is more as each person can also train the recognizer his/her unique writing style. The training that is done to teach the recognizer a personal writing style is the same training that happens before Microsoft ships the product. The only difference is that we are now collecting unique training data from a specific person (and not that of thousands of people). We call this process “Personalization”.

Figure 3: Personalization Wizard (Sentence module).

As the screenshots of our Personalization wizard illustrates, a person is asked to write the requested sentence to provide his/her ink samples. The more data a person donates during the personalization process, the better the recognizer will become. In addition to providing writing samples based on specified sentences, a person can target specific recognition errors, shapes, and characters that will all be used for training. Our Personalization feature is complex and offers a variety of different modules that enable a person to optimally tune the recognizer. We are proud to announce that Personalization will be available for all Vista languages and all new Windows 7 languages. We encourage you to use this feature to improve your recognition accuracy.

We continue to work on improving our recognizers which also means that we are incorporating our customers feedback through online telemetry (anonymously, privately, voluntary, and opt-in). In Windows Vista we released a new feature called “Report Handwriting Recognition Errors”, which gives people the opportunity to submit those ink samples that the recognizer did not recognize correctly. After the person has corrected a word in the Tablet Input Panel (TIP), we enable a menu that allows a person to send the misrecognized ink together with its corrected version to our team.

Here is a screenshot of what our error reporting tool looks like:

Figure 4: With “Report Handwriting Recognition Errors” people can choose which of the misrecognized ink samples they want to submit.

We receive approximately 2000 error reports per week. Each error report is stored in our database before we analyze it and use it to improve our next generation of recognizers. As you can imagine, real world data is extremely helpful because it is only this type of data that can reveal shortcomings of our recognizers.

We value and appreciate every single error report. Keep sending us your feedback, so that we can use it to improve the magic of our present and future recognizers.

Thank you,

– Yvonne representing the handwriting recognition efforts