Loneliness of the long-distance linguist. : ELS

Language detection and system settings, part 1

KieranS — Fri, 19 Dec 2008 04:07:00 GMT

Once upon a time, a customer got in touch with us about a font rendering problem, where instead of seeing the characters he expected to see inside his applications, he saw question marks.

- What’s your system locale? we asked.

- I’m in Cleveland, he said, in my upstairs home office.

Some flavor of this story has been going around our group for years, and I’d call it an urban legend if it weren’t for the fact that I’ve observed several flavors of the story myself. The truth is that we have way too many ways for users to indicate their language (and region) preferences. The proliferation of settings not only confuses users but also makes it near impossible for developers to understand which system setting they should use to determine which aspects of the user experiences that they’re trying to create. Over at Go Global there is some documentation (as well as somewhat older stuff on the old globaldev site) that is designed to help users and developers differentiate between user locale, default user locale, system locale, and input locale, but the fact is that these don’t even constitute the full range of settings available to users on several common installations of Windows. A partial list of the settings that users and/or developers are asked to make sense of:

- User locale

- System locale

- Input locale

- Thread locale

- Input locale

- Default location or geoID

- System UI language

- User UI language

On top of these, users may also encounter the browser Accept Language, language settings in Office or other productivity suite software (sometimes including separate settings UI for every application in the same productivity suite), and language and region preference UI exposed to them from various web services (even multiple times across different web services provided by the same publisher). From the user’s perspective, she’s stuck entering the same information over and over again, in UIs that are different enough to be non-intuitive but similar enough in goal to make her wonder why she’s repeating the same task every time she goes to a new website or opens a new application.

And one of the worst parts about this confusing proliferation of settings is that not one of them is reliable to tell you anything about the particular language that a user cares about at any given point in time; the best they can do is give you a ballpark guess as to a user’s typical intentions or behavior. Throw a little multilingualism into the picture, where a user may regularly interact with a computer in more than one language, and things get even more convoluted.

One of the biggest reasons we introduced ELS language detection is to give developers a way to know what language their user cares about much more scenario-specifically; developers of any Windows application on Windows 7 can now find out the user’s active computing language in text input or reading scenarios simply by passing the text to ELS. This means that in many cases, developers can stop using system settings to make swags at user experience (though the user settings may still end up being used as a fallback for language detection, on which more in a future post).

Next up: Which system setting should you use as a fallback to language detection? More to come.

By popular demand: Transliteration vs. translation.

KieranS — Tue, 16 Dec 2008 08:41:00 GMT

Okay so no, when we say transliteration, we don’t mean translation. But they do have some things in common.

Understanding the difference between language and writing system – which naturally everyone does now, having read the previous blog post – is essential to understanding the difference between translation and transliteration.

As discussed earlier, a script, or a writing system, refers to the way in which a language is written down. To moderately belabor the point:

String	Language	Script
cat	English	Latin
Katze	German	Latin
кошка	Russian	Cyrillic
قطة	Arabic	Arabic
고양이	Korean	Hangul
猫	Chinese	Simplified Chinese
貓	Chinese	Traditional Chinese

There is no 1:1 correlation between the languages of the world and their writing systems. In some cases, a particular writing system may be used to transcribe a single language (e.g. Thai, Malayalam), but in other cases, a writing system may be used to transcribe multiple languages (e.g. Latin, Cyrillic). In a few cases labels confuse people; for instance, the Arabic writing system is used to represent the Arabic language, along with several others (e.g. Urdu, Dari, Persian).

When people talk about translation, they typically refer to the mapping semantic content from one language into another. Each string in the first column above represents a translation of the English word cat into some other language. This is famously difficult to do well – not just for computers, but even sometimes for bilingual humans. It’s all well and good when you limit yourself to translating the word cat, but gets awfully hard when you move anything with syntactic complexity, and unbelievably hard when you move to anything of significant length. Anyone who has ever studied a foreign language probably has a pretty good idea of how hard this is. Even fluently bilingual people tend to have a hard time doing this to their own satisfaction.

Transliteration is another kind of mapping, only instead of mapping from one language to another, transliteration maps from one writing system to another. If you think of a writing system as a purely notational form for designating the spoken language, it’s easy to see where this kind of mapping might come in handy. By transposing strings from an unfamiliar writing system to a familiar one, the reader knows how to pronounce – and in some cases therefore semantically interpret (but more on that below) – the content. Japanese children, for instance, encounter unfamiliar Kanji all the time as they’re learning to read. Oftentimes they may know the word that the Kanji is representing in the spoken language, but they have no way to connect the word they’re used to hearing with the Kanji that they see in front of them. In fact, Japanese has a phonetic notational system – Yomigana – expressly designed to help Japanese speakers (adults as well as children) read unfamiliar Kanji terms, and there is a standard transliteration of Japanese into Latin script that is frequently used by non-native speakers of Japanese to help them pronounce new Japanese words that they encounter.

In many cases, transliteration really refers to some kind of phonetic transcription, helpful either for non-native speaker scenarios as above or in cases where a particular language may be standardly written in multiple writing systems (e.g. Serbian, which may be written in Latin or Cyrillic scripts). However, in one of the most common use cases for transliteration – the mapping between Traditional and Simplified Chinese – sound doesn’t enter into it, as both writing systems are ideographic and not alphabetic or syllabic. This mapping counts as transliteration rather than translation because it is a transposition between two writing systems that represent the same underlying spoken language, rather than a transposition between two distinct spoken languages.

ELS in Windows 7 exposes programmatic support for several transliteration modules that you can use the help shape user experience for your international customers, including:

· Traditional Chinese <> Simplified Chinese: Use this to reduce localization costs for your application, or to extend the reach of your content to new customers

· Cyrillic > Latin normalization, with a focus on Serbian and Russian: This can also be used to reduce localization costs, or in some cases to enable second language learning scenarios

· Indic > Latin normalization (Devanagari, Bengali, Malayalam): These are phonetic transcriptions that are primarily useful in second language learning or communication scenarios

Language detection, script detection, and how they differ.

KieranS — Sat, 22 Nov 2008 04:49:00 GMT

Two of the first services out of the gate for ELS are language detection and script detection, both of which are available in the PDC build of Windows 7. Both of these are already being used by different components in different ways and I thought it might be useful to do a general overview of what they are, how they differ, and how you can use them.

First, let’s start with defining what we’re detecting. A language for ELS refers to the semantic content of a string, and corresponds pretty much to your intuitive notion. English is a language. French is a language. Chinese is a language. A script in this context refers to the writing system that the string is encoded in. The Chinese language, for instance, is written in two scripts: Simplified, which happens to be used in PRC, and Traditional, which happens to be used in Hong Kong and Taiwan. Cyrillic is a script. Devanagari is a script. Arabic is a language, but it’s also the name of a script (which is used to write the Arabic language as well as some other languages, such as Urdu and Pashto). ELS allows you to retrieve both the language and the script of whatever string you pass in. If you pass this paragraph, you’ll discover that it’s the English language, written in the Latin script.

When it comes to language detection, we do better and better the more text you pass us. If you pass this string:

This is an English sentence.

We’ll return you a guess list of languages in order of our confidence. In this case, the sentence is English, so English (represented with its appropriate ISO code, en) will be returned first on the list. If we make the string a bit shorter:

This is an

We have less data to work with, but we can guess that this is an English fragment, and so our guess list will still return en at the top of the list. As we get smaller and smaller:

This is

This

Thi

We’ll return fewer and fewer guesses. We can still handle fragments that are recognizably English, such as the first item on this list, but by the time we work down to the single character or word fragments at the bottom of the list, we’ll start returning an empty string; we’d rather tell you we don’t know than make a bad guess. One of our general goals is to do whatever humans can do in terms of language detection. It’s a good bet that if a human can’t tell what language the single T belongs do, then our language detection can’t either.

However, while a single T doesn’t clearly belong to any particular one language, ELS (and a human) can tell with confidence what script it belongs to, since script identification is a property at the character level. So if you pass a mixed script string to ELS and ask for script detection:

I used to know 望遠鏡で日本人男性.

We’ll identify each script range that we find, telling you which characters are Latin, which are Katakana or Hiragana, and which are Kanji, along with their position in the string. If you pass the same string into language detection, we’ll tell you that we find English and we’ll tell you that we find Japanese, but we don’t break the results into associated ranges. This is arguably something that we should do, but it isn’t implemented this way today (and it’s a non-trivial problem). For this reason, a number of ELS callers are using script and language detection in combination, first passing strings into script detection to range-break, and then passing the resulting subranges back into language detection to up the range accuracy.

One question that people are asking very frequently is how long strings need to be in order to rely on the accuracy of ELS's language detection support. The truth is that it depends somewhat on the string itself. Languages that by virtue of their writing system can be uniquely identified (e.g. Thai) will return accurate results from single-character strings, since those strings cannot be any other language. Where a writing system can be used to represent multiple languages (ELS supports dozens of Latin-script languages, for instance), syntactic fragments (not necessarily conplete sentences) are often required for perfect accuracy. ELS provides script detection for every writing system in Unicode 5.1 and language detection for in the neighborhood of 100 languages. We often end up back to the generalization above; if a human can't tell what language it is, it's a good bet that ELS can't either. And if we're not sure, we'd rather tell you we're not sure than offer you a false positive result.

More usage tips to come!

What's new for you in Windows 7: Extended Linguistic Services

KieranS — Fri, 31 Oct 2008 03:28:00 GMT

Now that you've been able to go check out our PDC talk (online here), I'm excited to introduce you and this blog to ELS, our next big set of globalization support for developers! From our PDC Developer Guide:

Extended Linguistic Services (ELS) is a new feature in Windows 7 that allows developers to use the same small set of APIs to leverage a variety of advanced linguistic functionality. By using ELS APIs in Windows 7, developers can auto-detect the language of any piece of Unicode text and use that information to help make smarter user experience choices for customers around the world. ELS also offers built in transliteration support that converts text from one writing system to another. For instance, developers can now auto-convert text between Simplified and Traditional Chinese to help people communicate with each other across linguistic boundaries. By using ELS APIs, developers will be able to use existing ELS services as well as pick up new services in the future without learning new code.

You can also read more about it on our Go Global website here: http://msdn.microsoft.com/en-us/goglobal/dd156834.aspx

The white paper takes you through the basics of what we're building and some places where we think you might find it helpful, and also offers some code samples that those of you who are planning to play with the Windows 7 beta bits will be able to try out. Some of the first ELS functionality that's coming your way includes:

Transliteration functionality that will let you convert automatically between Simplified and Traditional Chinese, allowing you to bring your Chinese content to an even greater number of customers without any localization costs on your part
Language detection support that handles every language with a supported Windows locale (100+!)
A simple API set that you will be able to learn once and use to call every service that ELS exposes, now or in the future

The first pieces developed by the ELS team are currently online for customers using the Chinese convertor over at the Windows Live Translator site, so if you want a taste of some of the support that ELS APIs will make available in Windows 7, give it a whirl.

I'm pretty excited about it! If you check it out, I'd love to know what you think. The functionality we're adding now is just the beginning of several services that we have in our roadmap. In the coming weeks, I'll be posting more specifically about our API set and services, so stay tuned.