Office Natural Language Team Blog

Want to give input on proofing tools? Participate in the Proofing Tools Survey!

Natural Language Group Microsoft Office — Mon, 08 Mar 2010 23:34:00 GMT

We are currently running a survey on how users edit and review content and how they use proofing tools such as the spell checker, the grammar checker or the thesaurus in Microsoft Office. We want to learn more about user preferences so we can improve the user experience in some future version of Office. The survey takes about 15 minutes. If you are interested in participating please go to

http://office.microsoft.com/en-us/word/FX100649251033.aspx

and look for the "Help us make Word Better!" field in the lower section of the page. Click "Start the Survey" in order to participate.

We are looking forward to hearing from you!

Stefanie Schiller

Natural Language Group - Program Manager

Contextual spelling for French in Office 2010

Natural Language Group Microsoft Office — Sat, 18 Jul 2009 00:05:00 GMT

At the Worldwide Partner Conference in New Orleans on 13 July 2009, we announced the launch of the Office 2010 Technical Preview. This technical preview can now be downloaded by thousands of customers. You can discover the innovations on the Office 2010 blog and watch really cool videos on www.microsoft.com/office2010. My colleague Stefanie Schiller wrote a few words about the proofing tools integrated in this Technical Preview and about some of the improvements we have made, specifically with respect to the English thesaurus.

French-speaking users will also be delighted. We have talked on multiple occasions on this blog about the English and Spanish contextual spellers that we launched in Office 2007 (which also includes such a tool for German). We have improved them all in Office 2010 and we are happy to introduce a brand-new contextual speller for French which, when added to the French spellchecker and grammar checker, will greatly improve our French-speaking users’ proofing experience.

The French contextual speller in Office 2010 will detect a lot more mistakes which went so far unnoticed with the traditional proofing tools. Unlike the grammar checker, which is based upon a syntactic parser, our contextual speller is based upon statistical analyses of very large textual corpora and upon “language models” which enable the program to compare the user’s text with huge lists of sequences of words with their frequencies. Words that exist in the language but are used improperly in a given context can then be flagged. A blue squiggly line will appear under mistakes such as the following ones:

Ils on faim. (on à ont)

Elles son malades. (son à sont)

Quand à moi, j’avoue que je sui fier de lui. (Quand à Quant ; sui à suis)

Si je peu me permettre, dans son fort intérieur, elle pense qu’elle a raison. (peu à peux ; fort à for)

Se test montre que le correcteur ne fonctionne pas trop mal. (Se à Ce)

L’installation de la fosse sceptique a pris plus de temps que prévu. (sceptique à septique)

Il arrive cet après midi.(après midi à après-midi)

Mon frère ma dit qu’il ne viendrait pas. (ma à m’a)

Il y a long temps que je l’aime, jamais je ne l’oublierai… (chanson populaire) (long temps à longtemps)

En temps que client de l’hôtel, vous avez gratuitement accès à l’Internet. (temps à tant)

The screenshot below shows this contextual speller in action :

What is a « contextual speller » ? As you know, the traditional Office spellchecker flags the odd typo (omission of a letter, permutation of two letters, etc.) with a red squiggly line. The grammar checker deals with agreement mistakes (such as between a verb and its subject, or agreement in number and gender between a noun and an adjective in French). Mistakes related to words that are pronounced similarly but are spelled differently are very hard to detect, however. Anyone who knows a bit of French knows how frequently people (native and non-native speakers alike) mix up similarly-sounding words like son/sont or on/ont. If I write “Ils on faim” (they are hungry), a grammar checker based upon a syntactic parser has difficulty detecting the mistake (“on” should read “ont”) because the erroneous sentence is made up of a pronoun (Ils), followed by another pronoun (on, instead of the correct verb form ont) and a noun (faim). It is hard to make sense of this structure, since it is not a traditional agreement problem as in “Ils mange du pain” (they eat bread), where “mange” (a singular verb) should read “mangent” (plural form).

Of course, you should not expect this tool to be able to flag any kind of mistake. No existing tool is able to do that and those that would be able to do so would probably create a lot of false flags, or false positives, which tend to irritate the user. I discussed the notion of precision and recall when I blogged about an academic evaluation of our Office 2007 contextual speller. When we developed our tool, we constantly tried to avoid false flags and our tool has a very high precision, which means it rarely makes mistakes when it flags something (in fact, it is right nearly all the time, but there will of course always be mistakes that will not be detected). I tend to feel that this new contextual speller will quickly prove to be an indispensible tool for many an Office 2010 user who writes documents in French. It will certainly very usefully complement the range of proofing tools we make available to them.

Thierry Fontenelle

Microsoft Natural Language Group – Program Manager

Proofing Tools in Office 2010

Natural Language Group Microsoft Office — Fri, 17 Jul 2009 01:45:00 GMT

On Monday this week, Microsoft launched the Office 2010 Technical Preview. Thousands of customers can now download this latest version of Office to try out the cool new features and provide feedback.

Updates and Replacements

The Technical Preview includes proofing tools for English as well as for French and Spanish. We are delighted to announce that these proofing tools include brand-new spell checkers, contextual spellers, hyphenators and thesauri. We are excited about the release of these new proofing tools and eager to receive your feedback.

Improvements in the English Thesaurus

The English thesaurus in particular has seen some changes. You will now be able to find more synonyms in fewer steps. In Office 2007, if you look up the word tables, the only suggestion is table – the same as the lookup word, only in the singular. Next you have to look up the word table, select the synonym you are looking for and manually add an ‘s’ to the word to convert it back to plural. Four steps that include manual edits for one simple lookup. All of this can be done in one simple step in Office 2010. The screenshot below shows the results of the thesaurus in Office 2010 for the word tables:

We call this “inflectional morphology”. Try the same with the word smarter in your Office 2007 version, and you will see why we think that inflectional morphology is a big step forward. We also added additional data to provide more synonyms for your lookups.

Localized Versions and Language Packs

The current Technical Preview includes Proofing Tools for English, French and Spanish. If you install the Technical Preview, you will not be able to run proofing tools for any language other than English, French and Spanish. For this release we have made significant changes in the proofing infrastructure, therefore the Language Packs from previous Office versions including Office 2007 are not compatible with Office 2010. Proofing tools for these languages will become available with the release of the localized versions or with the subsequent release of the Language Packs for Office 2010.

Stefanie Schiller

Microsoft Natural Language Group – Program Manager

Add the Microsoft Translator to the Office Research pane

Natural Language Group Microsoft Office — Mon, 08 Jun 2009 23:24:00 GMT

The Microsoft Translator from the Microsoft Research (MSR) is now available for download in the Microsoft Download Center. Another translation service for Microsoft Office? Do you need it? Well, more choices are a good thing—especially in the area of languages and machine translation. After all, you don’t want to inadvertently say something to your grandmother or to your pen pal in a machine-translated letter that you don’t mean to say.

The Microsoft Translator is free, available in many language pairs with more being added soon, and can be used with Microsoft Office 2003 and Microsoft Office 2007. So you don’t need to worry about having an older version of Microsoft Office. The best feature of this new service is the side-by-side comparison of the original document and the translated document. This allows you to easily compare the translation quality to the original document in a single browser window.

If you discover that you prefer one service for one language and another service for a different language, you can easily change the preferred service for each language pair: in the Research pane, under the Search for text box, selection Translation, and then click Translation options. In the Translations options dialog box, under the Machine Translation section, you can scroll to the right of the relevant language pair and select the service you want from the drop-down menu.

Ultimately, human translation is optimal—so you don’t say something to your grandmother or your pen pal that you need to apologize for later—but machine translation is the next best solution. Having more translation service choices available in the product that you are using to create your letter to your grandmother or your pen pal makes it easier for you to say exactly what you want to say, in the language that you want to say it. You can also quickly understand the gist of the text you have received.

You can find more information about this feature in the “Translate Text” Help asset.

Daj Oberg

Office Content Publishing

Why is chef-d’œuvre my favorite French word?

Natural Language Group Microsoft Office — Thu, 12 Mar 2009 07:49:00 GMT

In this post, I would like to share with you the reasons why I love the French word chef-d’œuvre (=masterpiece). My interest for this word has nothing to do with its meaning. As a program manager working with computational linguists, I find it fascinating because it epitomizes the numerous difficult decisions one has to make when building natural language processing systems like word-breakers, tokenizers, spell-checkers, etc.

One of the most vexing problems in NLP is to decide whether hyphens and apostrophes are breaking characters or not. The identification of word boundaries (tokenization) is indeed essential, as I have argued elsewhere in an attempt to show that word-breaking is not easy. Hyphens frequently separate distinct tokens, as in le match France-Italie (nobody would argue that France-Italie is one word and nobody would expect to find the whole string in a dictionary). In chef-d’œuvre, however, the hyphen is part of the word and everyone will expect the whole string to be granted entry status in a dictionary. It should be considered as one token that has little to do with the word chef (which typically refers to a person), unless one considers the etymology of the word. This means that, in a search scenario, a user would not consider a document containing the words chef and oeuvre used separately as relevant if that user typed the keyword chef-d’œuvre.

The apostrophe in chef-d’œuvre is also interesting. Apostrophes are frequently used in French elided forms when a pronoun or a determiner is followed by a word that starts with a vowel (consider l’école [the school), je l’aime [I love her], j’arrive [I’m coming]]. In such cases, it is normal to consider that the string is made up of two distinct tokens (l’école -> l’ + école). The apostrophe in chef-d’œuvre has a distinct status and is an integral part of the word, very much like in other French words such as aujourd’hui, prud’homme, and a few others.

The word chef-d’œuvre is also interesting because it includes a special character, œ, known as a “ligature” (two or more letters joined together). Many other words in French include ligatures such as œ or æ (œuf [egg], sœur [sister], cœur [heart], cæcum …) and many other languages use characters which are not traditionally found in English (the German β or the Spanish ñ are cases in point). This reminds us that many NLP projects started with applications developed for English initially and subsequently required specific changes to take into account the non-ascii characters found in many other languages. Until very recently, the OpenOffice.org French spell-checker used to flag forms with ligatures like sœur or œuf as incorrect and only verified the incorrect spellings with two distinct characters (soeur, oeuf…). With the advent of Unicode, such problems are probably less frequent today, but it is clear that any multilingual project needs to consider idiosyncrasies such as the use of diacritics and other special characters some languages love so much…

From a morphological point of view, the word chef-d’œuvre is also atypical. While regular nouns typically take a final “s” in the plural in French (singular maison [house] -> plural maisons), the form *chef-d’œuvres is incorrect and should be flagged by a spell-checker, even if œuvres is correct on its own in other contexts. Rather, the plural is formed by adding an “s” at the end of the subtoken chef: des chefs-d’œuvre.

Finally, if you consider how the word is pronounced, it is clear that chef-d’œuvre poses a number of challenges: why is it that the “f” is pronounced in chef [∫εf ], but not in chef-d’œuvre [ ∫εdœvR ]? Interesting problem for my colleagues who create text-to-speech systems…

Chef-d’œuvre is certainly not the only complex word which encapsulates so many difficulties for those of us who create NLP applications. I could probably also have chosen le hors-d’œuvre [appetizer], which begins with an aspirated ‘h’ and does not admit elision, unlike homme [man] -> l’homme. Main-d’œuvre [manpower] would probably have been a nice candidate, too. In fact, there is no dearth of thorny issues linguists need to deal with. Well, languages are difficult, aren’t they? That’s probably what makes my job here so challenging and so interesting …

-- Thierry Fontenelle (Program Manager)

Natural Language Group blog featured in Language Tech News

Natural Language Group Microsoft Office — Tue, 03 Mar 2009 06:59:00 GMT

The post I wrote a few months ago about how users can remove a word from the main dictionary of their Office speller has been reprinted in the latest edition of Language Tech News (vol.2, No.4, February 2009), a publication of the Language Technology Division of the American Translators Association (ATA). Besides some technical details about how to use these “exclude dictionaries”, which the editor of the Newsletter felt would interest his readers, this post was a nice opportunity to show how the new contextual spellers in Office 2007 reduce the need that some users feel to exclude certain words from their speller dictionary.

This issue of Language Tech News also includes interesting articles on Trados’s translation memory technology, terminology management tools, font converters, and statistical machine translation (SMT). It is interesting to read the following comment about the Windows Live Translator, the MT system created by our Microsoft Research colleagues: « Perhaps the most successful MT application in the world today, the Microsoft Knowledge Base, used by hundreds of millions of users across the globe, is mostly a SMT-based effort » (p.17).

Thierry Fontenelle (Program Manager)

Hotfix für die deutsche Silbentrennung

Natural Language Group Microsoft Office — Thu, 22 Jan 2009 19:56:00 GMT

Microsoft hat einen Hotfix für die deutsche Silbentrennung in Office 2007 veröffentlicht. Dieser Hotfix verbessert eine Reihe von inkorrekten Silbentrennungen bezüglich der Trennung von Präfixen, wie z.B. be-fasst, ein-lässt, auf-läuft , um-bringt, und der Trennung von Suffixen, wie z.B. Kennt-nis, sowie der Trennung von zusammengesetzten Wörtern, wie z.B. Haus-tür, Glas-tisch. Betroffen von dem Hotfix ist nur die Spracheinstellung Deutsch (Deutschland), wo die fehlerhaften Silbentrennungen aufgetreten waren. Deutsch (Österreich) und Deutsch (Schweiz) sind nicht betroffen.

Der Hotfix ist unter dem KB-Artikel http://support.microsoft.com/kb/960500/en-us zu finden. Weitere Informationen zu den Installationsanforderungen können Sie im KB-Artikel nachlesen. Der Hotfix wird auch über eines der nächsten Service Packs verfügbar sein.

Stefanie Schiller - Program Manager

A context-sensitive speller for Spanish in Office 2007

Natural Language Group Microsoft Office — Thu, 15 Jan 2009 07:25:00 GMT

A few weeks ago, Microsoft announced an initiative targeting the Hispanic community, with special offers for Microsoft Office 2007 and Microsoft Office 2007 Language Pack in Spanish.

It may be worth pointing out that the Spanish proofing tools in Office 2007 include a brand-new context-sensitive speller in addition to the regular spell-checker, thesaurus, hyphenator and grammar checker. We have discussed the English context-sensitive speller on multiple occasions on this blog and regular readers know that blue squiggly lines now appear under contextual mistakes (real-word errors which traditional spellers cannot flag) such as:

Nobody knows weather he is innocent or guilty. (à whether)

President Bush addresses Untied Nations General Assembly. (à United)

People say your hole life flashes before your eyes before you die. (à whole)

Life insurance plays a very essential part in our every day life. (à everyday)

If you teach Spanish, if you are a translator, or if you write documents in Spanish regularly, you will most certainly understand how difficult it can be to spot the erroneous use of the word echo in a context where hecho should in fact be used. The blue squiggles under contextual mistakes testify to the progress made in this area in Office 2007 (if you right-click on the word echo in the sentence “Lo ha echo muy bien", you will see that the speller correctly suggests hecho, as is shown in the examples below).

Missing accents (as in medico, which should be médico in the sentence “Fuimos al medico ayer”) are very frequent mistakes as well and this new tool should prove useful to identify these blatant errors before you publish your document or send your email.

-- Thierry Fontenelle (Program Manager)

An academic evaluation of the Office 2007 contextual spelling checker

Natural Language Group Microsoft Office — Thu, 08 Jan 2009 08:00:00 GMT

A few days ago, I discovered an analysis of our Office 2007 contextual speller carried out by Prof. Graeme Hirst, from the University of Toronto: An Evaluation of the Contextual Spelling Checker of Microsoft Office Word 2007.

We have discussed this new context-sensitive speller on several occasions on this blog (as well as here) and it is nice to see that it is attracting the attention of researchers in the academic world.

It’s an interesting paper, which provides some food for thought, however, especially with respect to how “aggressive” we should be in our approach to recall.

His conclusion nicely sums up our trade-offs and dilemmas (emphasis mine):

In an evaluation on 1400 examples, it is found to have high precision but low recall — that is, it fails to find most errors, but when it does flag a possible error, it is almost always correct.

The contextual spelling corrector in Microsoft Office Word 2007 is a cautious (low recall) but believable (high precision) system. However, its overall performance, as measured by F, is much poorer than that of the trigram method of Mays et al (1991).

The trade-off between the two systems is a difficult one. In simple terms, better performance is better; but believability is an important attribute for a consumer-level system (“if Word says it’s wrong then it’s wrong”) and could well be considered worth sacrificing performance for. The problem with this, however, is that as users become familiar with the system, their expectations will rise and believability will start to apply also to what Word fails to flag (“If Word says it’s right then it’s right”).

A system that is more visibly error-prone might actually serve users better.

The methodology used by Prof. Hirst and his colleagues to evaluate the system deserves a few comments:

· They automatically induced real-word errors by replacing words by any spelling variation found in the lexicon of the ispell spelling checker. They limit the manipulation to an edit distance of 1 manipulation. So these errors are not natural mistakes.

· They did not consider “malapropisms” (real-word mistakes) involving closed-class words and words formed by the insertion or deletion of an apostrophe or by splitting a word: this means they exclude pairs which we have found to be extremely frequent in real texts (then/than; your/you’re; its/it’s; everyday/every day; to/too; their/there/they’re…). These pairs feature prominently in any analysis of real mistakes, especially in the literature devoted to English as a Second Language. Everyone knows that many native speakers of English have a lot of difficulty mastering these confusables, which is why we decided to specifically target them.

· They did not include phonetic confusables such as cymbal/symbol, principle/principal, pear/pair, there/their which have an edit distance > 1.

The categories they did not include in their tests are precisely those which we focused on because flagging these real and frequent mistakes is very useful for users of Office and Word. So assessing the “performance” of a system by ignoring these may be a bit unfair, at least if one equates “performance” and “usefulness” (will users find the system more useful if we flag “have not lost monkey” (à money), a rare and unnatural mistake, or if we flag “it is to expensive”, a mistake our data shows is very frequent and which we seem to be good at flagging?). Recall would be a lot higher if pairs involving closed-class words and the standard phonetic confusables above were taken into account (our own metrics based on a large corpus of real mistakes shows that our recall is in fact higher than the 20-25% found by Hirst, and is around 40%). The alternative methods which he proposes have even higher recall (50%), but their precision (50%) is way lower than our system’s (96%). Hirst clearly favors a recall-based performance. His assumption is: do people want to use a system like Microsoft’s, which only spots one mistake out of 5 (our metrics show it’s in fact closer to 2 out of 5, i.e. 40%) and is right nearly all the time? Our assumption is: would users really want a system based on the trigram method advocated by Prof. Hirst, which flags 50% of the mistakes but is wrong in 50% of the cases? The feedback we generally get indicates that our users tend to prefer unobtrusive tools and switch off a tool which they consider unreliable.

Interesting debate, isn’t it? I am really grateful to Prof. Hirst for making this discussion possible.

So, what do you think? We are interested in hearing your opinion. Do you prefer a tool which casts the net as wide as possible and catches many mistakes, at the risk of being frequently wrong and of creating many false flags (false positives), or do you prefer a tool which does not catch all possible mistakes, but which you can trust when it does catch one? Do not hesitate to leave your comments below…

Thierry Fontenelle – Program Manager

English Grammar Checker, Fragments, and Settings

Natural Language Group Microsoft Office — Thu, 23 Oct 2008 00:54:00 GMT

A French translator was asking an interesting question the other day on the Word community newsgroup. He wanted to know how he could switch off the grammar rule which flags “Fragments”, i.e. incomplete sentence fragments that the writer is invited to revise. The user did not want to turn off the English spell-checker, which he found very useful, or even the grammar checker, but only that particular rule, which he found particularly annoying when he was translating texts into English.

As can be seen below, the English grammar checker flags sentences that are considered incomplete (consider the fragment “And oranges.”, which is green-squiggled, or a verb-less sentence like “He happy.”). If you right-click on the squiggled string, you will see that the grammar checker advises you to consider revising this fragment.

The grammar checker is aware of the context in which the fragment has been used. A fragment such as “Flight to Paris” will not be flagged if it is not followed by a period (for instance in a title). The grammar checker will not say anything either if the same fragment is used in a bulleted or numbered list, as in the example below. In contexts other than titles, headings, or lists, however, it may seem reasonable to draw the user’s attention to a suspicious fragment. The user is of course free to ignore these flags.

Let us come back to our translator’s original question. It is indeed possible to customize the grammar checker and to prevent this rule from firing if one finds it useless. To do so, go to the Word Options (via the Office button in the top-left corner). Click on Proofing, then on Settings in the section “When correcting spelling and grammar in Word”. You will then see the list of grammar rules used by the English grammar checker, as displayed in the screenshot below. You just need to uncheck the 2^nd rule (Fragments and Run-ons), between the “Capitalization” rule and the “Misused words” rule. Done! The green squiggle will no longer appear under the structure shown in the examples above.

If you are writing a document in multiple languages, the cursor needs to be placed in a text whose language has been set to English. The English grammar checker will only work if the language of the text is set to English. If the text is in French, the rules of the French grammar checker will obviously be very different (the same applies to any other language), and you can customize them using the method described above. The settings of the French grammar checker will then be displayed as follows:

As you can see, users have some freedom to decide how they want to use their grammar checker. As far as I am concerned, I have unchecked the rule “Style – Contractions” of my English grammar checker because I constantly use contracted forms like “I’ll” and “You’ll” in my interactions with my colleagues. I know that, in many businesses, however, users who write very official documents want to spot these contracted forms and replace them with more formal forms like “I will” and “you will”. That is the reason why this gamut of settings is offered to the users of these complex tools.

Do not hesitate to send your feedback or your comments.

-- Thierry Fontenelle (Program Manager)

Can I remove a word from Office’s speller dictionary?

Natural Language Group Microsoft Office — Thu, 11 Sep 2008 08:50:00 GMT

The other day, I was discussing a number of suggestions to improve Office’s spell-checker. A customer was suggesting we should allow users to delete individual items from Word’s spell-checker lexicon. This feature is already available, in fact: if you want to specify a preferred spelling for a word and to exclude a given spelling from the main lexicon used by the Office speller, you need to use an “exclusion dictionary”. Your speller comes with an empty exclusion dictionary and you can add words to it if you want them to be permanently red-squiggled.

You first need to locate your exclusion dictionary, which, if you use Vista and Office 2007, can be found in the following folder:

C:\Users\User Name\AppData\Roaming\Microsoft\UProof\

Each language has a specific dictionary whose name starts with “ExcludeDictionary”, followed by the language code (EN for English, FR for French, SP for Spanish, GE for German…), followed by the LCID (locale identification number). The extension is .lex. For instance:

English (US): ExcludeDictionaryEN0409.lex

English (UK): ExcludeDictionaryEN0809.lex

English (Australia) ExcludeDictionaryEN0c09.lex

English (Canada) ExcludeDictionaryEN1009.lex

French: ExcludeDictionaryFR040c.lex

You can open the file with Notepad or WordPad and add a word which you want the speller to flag as misspelled. Save and close the file. You are done!

You can type “exclude dictionary” or “exclusion dictionary” in the Office help to get more information about this feature.

Of course, caution should be exercised when you decide to remove a word from your Office speller. If you decide to remove the word manger because you frequently type program manger instead of program manager, you should not be surprised when your speller flags manger in a sentence like “Jesus was laid in a manger”. This is why we have introduced a contextual speller, which tries to identify words which exist but are misspelled in a given context (see the post I was referring to, in which I showed how Office 2007 flags some erroneous uses of manger in program manger).

To give another example where contextual spelling might be preferred over exclusion, consider the user who had contacted the Word newsgroup to find out how to exclude the word “ahs” from the main speller lexicon. This user kept typing ahs instead of has. The new context-sensitive speller in Office 2007 flags a number of contexts where "ahs" should not be used, however, which should address this user's problem without having to remove the word altogether from the lexicon. You will see a blue squiggly line under "ahs" if you write something like "He ahs never done it before", for instance. But you will not get any flag under "ahs" if you write "we definitely got oohs and ahs all around when we launched this product".

Thierry Fontenelle – Program Manager

Suggested improvements to Microsoft Word’s spell-checker

Natural Language Group Microsoft Office — Thu, 04 Sep 2008 07:30:00 GMT

A few days ago, James wrote about the articles the Seattle Times published about our Natural Language Group and the Office spell-checkers. One of these articles was encouraging the Seattle Times readers to suggest improvements (What words would you add to the Microsoft spell-checker?). It is very interesting to read what our users consider pain points. The following suggestions from one of the readers (RogerKni) are interesting:

Here are my four suggested improvements to Word's spell-checker:

1. Give the user the option to flag rare words with an orange wavy line. An example would be “manger,” which 99% of the time is a misspelling of “manager.” Or “fro”: usually a misspelling of “for.” Or “whey” for “why.” Allow users to delete or add individual items to the Word-provided list of rarities.

2. Automatically add the apostrophized version of any noun added to a dictionary.

3. Add a “picky-mode” option, which one could turn on to get the preferred spelling of certain words. Currently, if there are two options for spelling a word, Word flags neither. That’s usually what’s wanted—the user doesn’t want to be harassed about a minor issue. But sometimes, as when publishing a book, the user wants to pick the best spelling. (Second-best spellings could be flagged with a wavy purple line.)

4. Give the user the option to tell Word to aggressively correct misspellings with what it thinks is the Best Match. Some would prefer this to Word’s cautious policy of merely flagging such words in red and making the user choose the correction.

In past releases, we looked into solving the problem described in Suggestion (2) for our English users. The solution presupposes an identification of (singular) nouns. We could ask users to do so for us, opting into the ‘s form for words like Palin (Palin’s), and not for the plural noun subproblems (*subproblems’s), the adjective semicontinuous or dermatoglyphics (* dermatoglyphics’s – see my colleague Mari Olsen’s post on possessives and apostrophes) . Of course we could also make guesses based on our knowledge of words that we have (continuous is an adjective; problems is a plural noun), but the computation would have to be weighed against the user benefit.

Suggestion (1) is in fact already covered in a large number of cases in Office 2007 (albeit not with an orange wavy line: we used a blue wavy line to signal contextual mistakes, which have been mentioned on several occasions on this blog). Look at the following screenshots, which show that the contextual speller in Office 2007 is able to flag words like “manger” or “fro” used in the wrong contexts:

Note, too, that users have the possibility of deleting words from the Office speller lexicon. We will come back to this issue in a future post (if you are impatient, type “exclude dictionary” or “exclusion dictionary” in the Help file). You may use that feature to exclude the non-preferred variants (of course, this is an individual decision you need to make: we cannot impose your own preferred spelling onto everyone).

Last but not least, suggestion (4) also exists: it is called internally AutoReplace. This feature “aggressively” corrects misspellings with what it thinks is the best match, as is suggested by RogerKni. Try typing “infomation”, for instance and you will see that Word automatically corrects it to information as you as you hit the space bar. To activate that feature, go to the big Office button in the top-left corner of your Office application, click on Word Options, then on Proofing, then on AutoCorrect Options. At the bottom of the screen, you will see the option “Automatically use suggestions from the spelling checker”. Tick the box as in the screenshot below:

I hope you will find these tips useful.

-- Thierry Fontenelle (Program Manager)

Natural Language Group in the (Local) News

Natural Language Group Microsoft Office — Thu, 28 Aug 2008 21:06:00 GMT

Last week our group was visited by reporters from the Seattle Times, who took some photographs and talked to folks on the team about the challenges of developing software features that keep up with the world's rapidly changing languages. This morning, the front page of the Business & Technology section discusses those challenges, and a sidebar gives a pretty nice summary of how we select words to add or remove from our spellchecking dictionaries, not just for English but for lots and lots of languages. Worth a look, if for no other reason than to get a glimpse of the terribly charming and attractive people who work on your spellcheckers.

-- James Lyle, Tester

Lets Play Two?

Natural Language Group Microsoft Office — Fri, 04 Apr 2008 02:41:00 GMT

For opening day this year, my beloved Chicago Cubs unveiled a statue of the immortal Ernie Banks. On its pedestal was engraved the catchphrase that made famous his enthusiasm for the game of baseball: “Let’s Play Two”. Except that the sculptor forgot one little detail, as reported by the Chicago Tribune: the apostrophe! He should have checked it out in Microsoft Word, first. The contextual spellchecker would have saved him a lot of grief:

-- James Lyle, Tester

The Grammar Checker and Rain Man

Natural Language Group Microsoft Office — Thu, 06 Mar 2008 19:22:00 GMT

Thinking about a couple of recent Language Log posts about grammar checkers, passive sentences, and grammatical Cupertinos, it occurred to me that grammar checkers are a bit like Rain Man. For the benefit of younger readers (no one alive in 1988 could have missed the hype surrounding this movie), Dustin Hoffman plays Raymond (“Rain Man”), an autistic man who is also a mathematical savant. Tom Cruise is his hotshot younger brother who finds himself with the difficult task of caring for Raymond, whose inability to relate to the real world leads them into bizarre situations. Tom takes them to Las Vegas, where Raymond’s uncanny mathematical aptitude makes them big winners until casino security accuses them of cheating.

What does this have to do with grammar checkers? A computer grammar checker is like a grammatical Rain Man—a savant with an encyclopedic knowledge of grammar rules, but no common sense understanding of the real world, which sometimes leads to strange behavior. Take the famous example “Time flies like an arrow”. This sentence is notable because it has a lot of possible but quite strange interpretations, in addition to its straightforward meaning of “time moves fast the way an arrow moves fast”. It could mean, for example, something like: whenever you encounter a fly, time it in the same way you would time an arrow (using a stopwatch, perhaps). We humans naturally disregard such interpretations (they may never even occur to us). But our grammatical Rain Man sees every such possibility at the same time, and has to decide among them. And it doesn’t have our common sense to help it. Instead, it has to figure out the intended meaning of sentence by purely computational means, basically by reasoning only about things that can be counted. Such as, say, the number of times it has seen “time” used as an imperative verb vs. the number of times it’s seen it used as a noun subject, and so on.

Most of the time this all works as advertized, but it can occasionally lead to some odd decisions. Once in a while the grammar checker will make a suggestion that just looks weird, so if you blindly accept all its recommendations, you end up with those cupertinos. Letting Rain Man call the shots may sound like an easy way to get rich, but like Tom at the casino, if you’re not careful you can end up in trouble.

So should you take advice on grammar from a Rain Man? Well, sure, if you need it—after all, it knows all the rules, and does a pretty good job most of the time. But take it only as advice, and look over its shoulder while it counts the cards. Somebody has to be watching the dealer’s eyes. That job is better left to a human.

-- James Lyle, Tester

Spellchecking ain't easy

Natural Language Group Microsoft Office — Sat, 16 Feb 2008 00:15:00 GMT

A customer wrote to us recently with the observation that our English spellchecker doesn’t recognize the word ain’t, a fact which struck this customer as a tad, well, old-fashioned. Pedantic, perhaps. The words “uptight” and “shortsighted” might have been used. Yikes! I’ve been accused of a lot of things, but…

First, let’s admit that this is not a mistake. Yes, we deliberately excluded ain’t. You can tell, because we made sure to get just the right set of words to suggest in its place (isn’t, am not, aren’t) despite these words being pretty far away from ain’t in edit distance. You could say the same about gonna and wanna, which are also excluded, but for which we suggest going to and want to, respectively.

This is one of those tough calls we encounter when building a spellchecker. As a linguist, I have no inherent objection to the word “ain’t” on any moral, intellectual, or even aesthetic grounds. It’s a part of my own spoken idiolect, and I tend to use it unselfconsciously in informal contexts. Ain’t no reason not to, usually. But clearly, it is still universally regarded as nonstandard, and people naturally want to avoid using it in formal writing. That goes for me, too--I doubt I’d want to use it when writing to the boss, even when the boss is a pretty informal kind of guy (Hi Bill!). So from that point of view, flagging this word as an error is a good thing for customers, a lot of whom are using MS Office at work.

On the other hand, people also have lots of reasons to want to write ain’t--to be deliberately jocular, say, or to sound folksy, or just because it’s natural. Or because they’re a Jane Austen character. And in those cases, they’d prefer to spell it right--there’s only one correct way to spell ain’t, after all, and your fingers can stumble on it just as easily as any other word. By not recognizing ain’t, we sure ain’t helping folks in those situations. And that goes for gonna and wanna, too.

So what do you think? Should there be a red squiggle under ain’t, or not?

-- James Lyle, Tester

Microsoft Office 2008 for Mac now supports the French spelling reform

Natural Language Group Microsoft Office — Mon, 21 Jan 2008 02:58:00 GMT

Microsoft Office 2008 for Mac was released this week. Users who are interested in the French language will notice a change in the French spell-checker, which now takes into account the French spelling reform, which is recommended by official bodies such as the Académie Française, the Conseil Supérieur de la langue française, the French Ministry of Education, the Groupe québécois pour la modernisation de la norme du français, the Réseau pour la nouvelle orthographe du français, etc.

The official texts make it abundantly clear that both the traditional (��old’) spelling and the ‘new’ (recommended) spelling are valid. As can be seen below, the Mac Office 2008 speller accepts both forms by default (traditional and new spellings). Users of the Office 2007 French speller are already familiar with the three options which enable them to change this default mode. As of this week, Mac Office users can now also decide whether they want their speller to:

(a) consider the old and new forms as valid (which is the default option)

(b) apply only the traditional (‘old’) spelling (i.e. ‘new’ forms will be squiggled)

If you are interested in a very brief description of the kinds of changes you might notice with this new speller, you can check out this post, which was written when we released the new speller for Office users a while ago.

-- Thierry Fontenelle (Program Manager)

Building corpora with the Live Search API

Natural Language Group Microsoft Office — Sat, 22 Dec 2007 03:01:00 GMT

I just read Building and Exploring Web Corpora, which includes the Proceedings of the 3^rd Web as Corpus Workshop (WAC3-2007) held at the University of Louvain-la-Neuve in September 2007. A number of papers describe how computational linguists have been using Microsoft’s Live Search Application Programming Interface (API) to build and clean corpora to be used in natural language processing.

One of the papers (Leturia et al.) describes the CorpEus tool, which uses the Live Search (LS) API and which the authors designed to create web corpora for the Basque language.

Another very interesting paper, by William Fletcher, describes the various reasons why that API was found to meet the linguists’ requirements to be able to generate concordances for linguistic research. Let me quote Fletcher here:

· Of the Search Engines which provide free APIs to developers, Live Search is the most generous by far: it allows 10,000 queries per application id (AppID) per IP address per day; [TF: Fletcher mentions 10,000 queries per day while Leturia et al. indicate that the API allows 25,000 queries per day. The latter figure is the correct one, in fact, which makes it even more generous]

· LS provides high-quality search results, with relatively few pages from link farms or “scraper sites”, which repeat content from or link to other pages merely for advertising revenue;

· It also supports search by location, i.e. by country or even latitude and longitude;

· Live Search is more responsive to changes on the Web: there is faster turnover in the top hits returned for a given query than with Google or Yahoo!, and documents in the cache tend to be “fresher”, i.e. updated more frequently;

· The LS cache provides quick, reliable access to the original texts. In documents retrieved from the cache, LS generally detects the character set encoding accurately and converts it to UTF-8, thereby eliminating a potential source of variability and errors;

· LS also converts Adobe Acrobat PDF documents to HTML which closely reflects the formatting of the original;

· The Live Search API provides direct links to the cache, and the site responds rapidly and at a high transfer rate, permitting very efficient data collection without delays, redirections or dead links.

Here are the full references of that paper, in case you want to read the whole story:

William H. Fletcher: Implementing a BNC-Compare-able Web Corpus, in Fairon, C., Naets, H., Kilgarriff, A., de Schrijver, G-M (eds): Building and Exploring Web Corpora – Proceedings of the 3^rd Web as Corpus Workshop, Incorporating Cleaneval (WAC3-2007, September 2007), UCL, Presses Universitaires de Louvain, 2007.

It’s cool to see that this search engine is found useful by some members of the computational linguistics communities and not just because it allows 25,000 queries a day. I’m sure more can be done to meet linguists’ needs, but it’s definitely encouraging to read that feedback. If you are interested in this Live Search API, you can check out its “Terms of Use” here: http://dev.live.com/livesearch/.

-- Thierry Fontenelle (Program Manager)

Untied Nations or United Nations?

Natural Language Group Microsoft Office — Thu, 20 Dec 2007 04:14:00 GMT

During my vacation in December 2007, I had a chance to visit a friend of mine who works for the United Nations in Bangkok.

On a Friday evening right before Secretary-General Ban Ki-moon's visit to the UN Bangkok office, I chatted with his colleagues in the UN building over beer and wine.

Many of them said they have a big problem with Office 2003 English spellchecker because it doesn’t correct the most common spelling error in their organization: “the Untied Nations”.

They were fascinated with the idea of upgrading it to Office 2007 which can correctly identify the error and suggest “United” thanks to its Contextual Spellchecker.

-- Kazami Uchida (International Product Engineer)

MSR blog on the Microsoft Research Machine Translation system

Natural Language Group Microsoft Office — Wed, 21 Nov 2007 01:48:00 GMT

Our colleagues from the Microsoft Research (MSR) group have started blogging about the statistical machine translation (MSR-MT) system they are developing. We announced the Windows Live Translator when it was launched in September. Check out their blog if you want to know all the details about this system. For instance, you will discover how to install and use the Windows Live Toolbar Translator Button, which is now available in (and for) 12 languages.

-- Thierry Fontenelle (Program Manager)

New Blog on Enterprise Search

Natural Language Group Microsoft Office — Fri, 16 Nov 2007 20:02:00 GMT

There's a new blog out there from the team that's working on Enterprise Search for MOSS (Microsoft Office SharePoint Server). They've got tips and tricks for administrators and will be posting info on features. It's a team that we work closely with, delivering query spelling suggestions and tokenization with morphological analysis. Some of the most recent news there is around a new release of Search. Check it out. http://blogs.msdn.com/enterprisesearch/

-- Jay Waltmunson (Program Manager)

The French spelling reform in the Canadian press

Natural Language Group Microsoft Office — Sun, 11 Nov 2007 05:38:00 GMT

For readers who are interested in the French spelling reform, two very recent articles published in Canadian newspapers in Montreal a few days ago discuss the penetration of the spelling reform, its slow but increasing adoption by teachers and the press, in Canada, Belgium, Switzerland and France. Both articles, which quote Chantal Contant from the Groupe québécois pour la modernisation de la norme du français, list the reference dictionaries and computerized tools that take the new spelling into account and, in both cases, the Microsoft Office speller is listed as a tool which covers 100% of the new forms.

Chantal Contant fait valoir que certains dictionnaires, tels le Hachette, le Littré et le Bescherelle, ont adopté intégralement les rectifications, tout comme les logiciels de correction Antidote, Myriade, ProLexis, Cordial et le correcteur de Word.

(L'Actualité, no. Vol: 32 No: 16, 15 octobre 2007, p. 70: Débat - Le français frisote)

Du côté des ouvrages de référence, Le Petit Robert et Le Petit Larousse sont plus réticents que le dictionnaire Hachette, le Nouveau Littré, les correcteurs Antidote, ProLexis et Word, ou les grammaires Bescherelle et Grevisse, qui intègrent 100 % des changements.

(Le Devoir, LES ACTUALITÉS, mardi 2 octobre 2007, p. A4 : Rectifications de l'orthographe

Les graphies font peu à peu leur chemin)

I blogged a few months ago about the three options offered to the users of the Office 2007 speller. You can also find a brief description of the differences between the traditional spelling and the “new” spelling (which is now also recommended by the French Ministry of Education in its official curriculum) here.

Thierry Fontenelle – Program Manager

Contextual spelling: US English only?

Natural Language Group Microsoft Office — Wed, 24 Oct 2007 20:48:00 GMT

Laurie asked us via the Email/Contact link:

I was always under the impression that the Contextual Spell Checker only works if your language is set to English (US) rather than English (UK). However, I have recently seen the blue squiggly lines appear for English (UK).

Can you confirm whether this has come about as a result of a specific Office update?

The contextual speller works for all varieties of English (UK, US, Australian, Canadian). This has been the case since the launch of Office 2007 and there has not been any specific update for that version of Office. If you write something like this:

(a) When inserting an Excel chart into a Word document, the chart looses its color when the focus is set to the document.

(b) When inserting an Excel chart into a Word document, the chart looses its colour when the focus is set to the document.

You will see the blue squiggles under looses whether you are in US English or in UK English mode. If you set (b) to UK English to make sure colour is not red-squiggled, looses will nevertheless be flagged as a contextual mistake and the contextual speller will suggest loses.

Thanks for giving us the opportunity to dispel that rumo(u)r, Laurie.

-- Thierry Fontenelle (Program Manager)

When Languages Die

Natural Language Group Microsoft Office — Wed, 17 Oct 2007 03:40:00 GMT

James was talking about endangered languages the other day. I have just finished reading David Harrison’s new book on “When Languages Die – The Extinction of the World’s Languages and the Erosion of Human Knowledge”, which I discovered via Michael Kaplan’s blog. It’s a fascinating account of language disappearance, which takes place because thousands of languages are gradually “crowded out” by bigger languages. Six years ago, there were an estimated 6,900 distinct languages and Harrison points out that by the end of our 21^st century, only about half of these languages may still be spoken because their speakers will have abandoned them to turn to more dominant, more prestigious or more widely known languages. Harrison brilliantly demonstrates what language death or language extinction means for us. He focuses on the vast body of knowledge that will soon be lost and explores various knowledge systems (moon phases, folk taxonomies, knowledge encoded in traditional calendars, topographic naming systems…) to show how cultural knowledge is packaged in languages and cannot be transferred when people stop using their language. I found the discussion about number systems enlightening and captivating. He points out that counting systems provide a window into human cognition and that a lot is lost when the speakers of a language decide to move to the decimal counting system. His demonstration is simply superb. Harrison argues that it is urgent to document languages and to do whatever we can to preserve them and to encourage their speakers to go on using them.

Everyone must play their part there. As a software company, we have a number of initiatives to help linguistic communities (see, for instance, the Microsoft Local Language Program which provides Language Interface Packs (LIPs) in a wide range of languages, or the community glossaries of IT terms which are built by local volunteers with the aim of helping local groups promote and preserve their languages – I also talked recently, in French, about a new Breton speller for Office 2007 which was created by a Breton-speaking volunteer who devotes a lot of time and energy to the preservation of his language). We have talked a lot on this blog about proofing tools and building word lists for spellers and other types of tools such as thesauri or word-breakers is certainly something that needs to be done if one wishes to help communities access technology in their languages. To some extent, I feel that Harrison and a group like ours (and several other groups in the company, of course) share a common passion for languages and a common goal: “what scientists can do is to capture an accurate record in the form of recordings and analyses”, he writes. Our technology can certainly help and I hope we will be able to offer even more in the future to help communities preserve their languages. At the same time, Harrison points out that no one but speakers themselves can preserve languages, since there is no such thing as a living human language without speakers (p.10). My sincere hope is that we’ll manage to create the synergies that are necessary to preserve language diversity and perhaps to prevent some languages from dying. Meanwhile, I definitely encourage you to read David Harrison’s book. You won’t regret it.

Thierry Fontenelle – Program Manager

Fellow linguist blogger in Windows International

Natural Language Group Microsoft Office — Mon, 01 Oct 2007 22:40:00 GMT

Kieran is a fellow linguist on the Windows International team, working closely with the team delivering Windows Desktop Search. She's got some great insight into language and technology on her "Loneliness of the Long Distance Linguist" blog. Check her out here: http://blogs.msdn.com/kierans/. Linguists, we are everywhere! :)

-- Jay Waltmunson (Program Manager)