Loneliness of the long-distance linguist.

Language detection and system settings, part 1

KieranS — Fri, 19 Dec 2008 04:07:00 GMT

Once upon a time, a customer got in touch with us about a font rendering problem, where instead of seeing the characters he expected to see inside his applications, he saw question marks.

- What’s your system locale? we asked.

- I’m in Cleveland, he said, in my upstairs home office.

Some flavor of this story has been going around our group for years, and I’d call it an urban legend if it weren’t for the fact that I’ve observed several flavors of the story myself. The truth is that we have way too many ways for users to indicate their language (and region) preferences. The proliferation of settings not only confuses users but also makes it near impossible for developers to understand which system setting they should use to determine which aspects of the user experiences that they’re trying to create. Over at Go Global there is some documentation (as well as somewhat older stuff on the old globaldev site) that is designed to help users and developers differentiate between user locale, default user locale, system locale, and input locale, but the fact is that these don’t even constitute the full range of settings available to users on several common installations of Windows. A partial list of the settings that users and/or developers are asked to make sense of:

- User locale

- System locale

- Input locale

- Thread locale

- Input locale

- Default location or geoID

- System UI language

- User UI language

On top of these, users may also encounter the browser Accept Language, language settings in Office or other productivity suite software (sometimes including separate settings UI for every application in the same productivity suite), and language and region preference UI exposed to them from various web services (even multiple times across different web services provided by the same publisher). From the user’s perspective, she’s stuck entering the same information over and over again, in UIs that are different enough to be non-intuitive but similar enough in goal to make her wonder why she’s repeating the same task every time she goes to a new website or opens a new application.

And one of the worst parts about this confusing proliferation of settings is that not one of them is reliable to tell you anything about the particular language that a user cares about at any given point in time; the best they can do is give you a ballpark guess as to a user’s typical intentions or behavior. Throw a little multilingualism into the picture, where a user may regularly interact with a computer in more than one language, and things get even more convoluted.

One of the biggest reasons we introduced ELS language detection is to give developers a way to know what language their user cares about much more scenario-specifically; developers of any Windows application on Windows 7 can now find out the user’s active computing language in text input or reading scenarios simply by passing the text to ELS. This means that in many cases, developers can stop using system settings to make swags at user experience (though the user settings may still end up being used as a fallback for language detection, on which more in a future post).

Next up: Which system setting should you use as a fallback to language detection? More to come.

By popular demand: Transliteration vs. translation.

KieranS — Tue, 16 Dec 2008 08:41:00 GMT

Okay so no, when we say transliteration, we don’t mean translation. But they do have some things in common.

Understanding the difference between language and writing system – which naturally everyone does now, having read the previous blog post – is essential to understanding the difference between translation and transliteration.

As discussed earlier, a script, or a writing system, refers to the way in which a language is written down. To moderately belabor the point:

String	Language	Script
cat	English	Latin
Katze	German	Latin
кошка	Russian	Cyrillic
قطة	Arabic	Arabic
고양이	Korean	Hangul
猫	Chinese	Simplified Chinese
貓	Chinese	Traditional Chinese

There is no 1:1 correlation between the languages of the world and their writing systems. In some cases, a particular writing system may be used to transcribe a single language (e.g. Thai, Malayalam), but in other cases, a writing system may be used to transcribe multiple languages (e.g. Latin, Cyrillic). In a few cases labels confuse people; for instance, the Arabic writing system is used to represent the Arabic language, along with several others (e.g. Urdu, Dari, Persian).

When people talk about translation, they typically refer to the mapping semantic content from one language into another. Each string in the first column above represents a translation of the English word cat into some other language. This is famously difficult to do well – not just for computers, but even sometimes for bilingual humans. It’s all well and good when you limit yourself to translating the word cat, but gets awfully hard when you move anything with syntactic complexity, and unbelievably hard when you move to anything of significant length. Anyone who has ever studied a foreign language probably has a pretty good idea of how hard this is. Even fluently bilingual people tend to have a hard time doing this to their own satisfaction.

Transliteration is another kind of mapping, only instead of mapping from one language to another, transliteration maps from one writing system to another. If you think of a writing system as a purely notational form for designating the spoken language, it’s easy to see where this kind of mapping might come in handy. By transposing strings from an unfamiliar writing system to a familiar one, the reader knows how to pronounce – and in some cases therefore semantically interpret (but more on that below) – the content. Japanese children, for instance, encounter unfamiliar Kanji all the time as they’re learning to read. Oftentimes they may know the word that the Kanji is representing in the spoken language, but they have no way to connect the word they’re used to hearing with the Kanji that they see in front of them. In fact, Japanese has a phonetic notational system – Yomigana – expressly designed to help Japanese speakers (adults as well as children) read unfamiliar Kanji terms, and there is a standard transliteration of Japanese into Latin script that is frequently used by non-native speakers of Japanese to help them pronounce new Japanese words that they encounter.

In many cases, transliteration really refers to some kind of phonetic transcription, helpful either for non-native speaker scenarios as above or in cases where a particular language may be standardly written in multiple writing systems (e.g. Serbian, which may be written in Latin or Cyrillic scripts). However, in one of the most common use cases for transliteration – the mapping between Traditional and Simplified Chinese – sound doesn’t enter into it, as both writing systems are ideographic and not alphabetic or syllabic. This mapping counts as transliteration rather than translation because it is a transposition between two writing systems that represent the same underlying spoken language, rather than a transposition between two distinct spoken languages.

ELS in Windows 7 exposes programmatic support for several transliteration modules that you can use the help shape user experience for your international customers, including:

· Traditional Chinese <> Simplified Chinese: Use this to reduce localization costs for your application, or to extend the reach of your content to new customers

· Cyrillic > Latin normalization, with a focus on Serbian and Russian: This can also be used to reduce localization costs, or in some cases to enable second language learning scenarios

· Indic > Latin normalization (Devanagari, Bengali, Malayalam): These are phonetic transcriptions that are primarily useful in second language learning or communication scenarios

Language detection, script detection, and how they differ.

KieranS — Sat, 22 Nov 2008 04:49:00 GMT

Two of the first services out of the gate for ELS are language detection and script detection, both of which are available in the PDC build of Windows 7. Both of these are already being used by different components in different ways and I thought it might be useful to do a general overview of what they are, how they differ, and how you can use them.

First, let’s start with defining what we’re detecting. A language for ELS refers to the semantic content of a string, and corresponds pretty much to your intuitive notion. English is a language. French is a language. Chinese is a language. A script in this context refers to the writing system that the string is encoded in. The Chinese language, for instance, is written in two scripts: Simplified, which happens to be used in PRC, and Traditional, which happens to be used in Hong Kong and Taiwan. Cyrillic is a script. Devanagari is a script. Arabic is a language, but it’s also the name of a script (which is used to write the Arabic language as well as some other languages, such as Urdu and Pashto). ELS allows you to retrieve both the language and the script of whatever string you pass in. If you pass this paragraph, you’ll discover that it’s the English language, written in the Latin script.

When it comes to language detection, we do better and better the more text you pass us. If you pass this string:

This is an English sentence.

We’ll return you a guess list of languages in order of our confidence. In this case, the sentence is English, so English (represented with its appropriate ISO code, en) will be returned first on the list. If we make the string a bit shorter:

This is an

We have less data to work with, but we can guess that this is an English fragment, and so our guess list will still return en at the top of the list. As we get smaller and smaller:

This is

This

Thi

We’ll return fewer and fewer guesses. We can still handle fragments that are recognizably English, such as the first item on this list, but by the time we work down to the single character or word fragments at the bottom of the list, we’ll start returning an empty string; we’d rather tell you we don’t know than make a bad guess. One of our general goals is to do whatever humans can do in terms of language detection. It’s a good bet that if a human can’t tell what language the single T belongs do, then our language detection can’t either.

However, while a single T doesn’t clearly belong to any particular one language, ELS (and a human) can tell with confidence what script it belongs to, since script identification is a property at the character level. So if you pass a mixed script string to ELS and ask for script detection:

I used to know 望遠鏡で日本人男性.

We’ll identify each script range that we find, telling you which characters are Latin, which are Katakana or Hiragana, and which are Kanji, along with their position in the string. If you pass the same string into language detection, we’ll tell you that we find English and we’ll tell you that we find Japanese, but we don’t break the results into associated ranges. This is arguably something that we should do, but it isn’t implemented this way today (and it’s a non-trivial problem). For this reason, a number of ELS callers are using script and language detection in combination, first passing strings into script detection to range-break, and then passing the resulting subranges back into language detection to up the range accuracy.

One question that people are asking very frequently is how long strings need to be in order to rely on the accuracy of ELS's language detection support. The truth is that it depends somewhat on the string itself. Languages that by virtue of their writing system can be uniquely identified (e.g. Thai) will return accurate results from single-character strings, since those strings cannot be any other language. Where a writing system can be used to represent multiple languages (ELS supports dozens of Latin-script languages, for instance), syntactic fragments (not necessarily conplete sentences) are often required for perfect accuracy. ELS provides script detection for every writing system in Unicode 5.1 and language detection for in the neighborhood of 100 languages. We often end up back to the generalization above; if a human can't tell what language it is, it's a good bet that ELS can't either. And if we're not sure, we'd rather tell you we're not sure than offer you a false positive result.

More usage tips to come!

To update or not to update?

KieranS — Sat, 08 Nov 2008 06:10:00 GMT

Can't we all just get along?

One of the struggles that we go through every release is in figuring out how to handle updates to the behavior of linguistic sorting. People rely on NLS APIs to sort strings appropriately for display to users. That means that strings need to be in the right order, where the right order is defined as the order that's going to allow a user to locate some particular item in a sorted list. Most of the time the expected order for a user who speaks some particular language doesn't change a whole lot. A lot of the time the expected order reflects something that people learned in school, or see consistently in dictionaries and encyclopedias and phone books. Oftentimes the order of encoding in Unicode is pretty close to what users expect. And sometimes where the above do not apply, we find highly intuitive and aware native speakers who not only have an innate awareness of what to expect in terms of sort order, but can also articulate why.

Then there are all the other times.

And many of the languages for which technology support is more recent? Are the other times.

For at least one language that we added during Vista, there was not only no national standard available, but four published dictionaries used four different (and internally inconsistent) sort orders. The articulate and perceptive native speakers working with us? Did not agree with each other. For many other languages, literacy patterns are emergent enough that native speaker intuitions are quite unclear and often wrapped up in other issues of cultural identity. There are groups pushing to follow sort orders and literacy traditions from other languages with overlapping character sets. There are groups pushing to establish local standards. There are still other languages for which there are no groups at all.

And yet, in many of these places, people are using computers. And the moment people can read and enter text on the screens in front of them, they're gonna be enveloped in sorted data. In the file system. In Excel files and documents. In other application UI. And for all of those users in all of those languages, the lists need to be predictable and linguistically appropriate. When for whatever reason it's not predictable or appropriate, people tell us about it, and we have bugs to fix. Even for languages with long literacy traditions and relatively advanced technology support, every once in a while the group of people who own the language – and here I mean a speech community of native speakers, who may or may not be aligned to a governmental language authority – change how things work. Spelling reforms get all the media play, but dictionary order reforms happen too. And when that happens, sorting behavior that may have been appropriate at the time we started supporting it stops being appropriate for the users it was intended to support. And then there are all the new characters that get added into Unicode, as new scripts are encoded and new code points are added to existing scripts. These characters get encoded because users need them.

So every time we release a version of Windows or the .NET framework, we end up addressing some issues that reflect either changes in local expectations or bugs in things we’ve shipped before. For applications that rely on us for linguistic correctness of sorted strings, this is a good thing. It means that app developers never need to roll their own behavior for linguistic appropriateness – they need to know how to make the right CompareStringEx() call with the right locale information for their user and they’re done.

But then there are the databases. With indexes that can become corrupt and cause huge compatibility headaches for users.

So anyone who needs to persist sorted data, like people building indexes for a database, needs to rely on consistency of their index. When sorting behavior changes, the index can become corrupt, which means that anyone searching on it won’t find the strings that they expect to find. So all the goodness of linguistically appropriate sorting is great for applications that present strings to users for ordered display, but these updates can wreak havoc for databases that aren’t designed correctly. So many people who persist sorted data avoid linguistic sorting entirely, instead using ordinal sorting to ensure consistency of behavior across a given character set. In Windows 2003, we introduced GetNLSVersion(), which allows developers to query the particular version of sorting behavior that is present on the OS to decide whether or not to reindex. If the version number has changed, then some sorting behavior has changed since the last version, which means that databases probably want to reindex to avoid the risk of corruption.

We hope that index builders are sorting ordinally, but what if they’re not? We think that many databases are using SQL or Jet or some other technology that shields them from having to think about the issue, but what if they aren’t? We believe that most databases go offline to reindex at major OS releases, but what if they don’t? We want people to check the NLS version to make informed decisions about reindexing, but what if people don’t know that they can do this? We know that many more people rely on us for linguistic sorting for display than for persisting ordered indexes, but every release we end up stuck between a rock and a hard place, where we need to make updates to serve the needs of international customers, but we need to be cautious about updates… to serve the needs of international customers.

And that about sums up my week. J

What's new for you in Windows 7: Extended Linguistic Services

KieranS — Fri, 31 Oct 2008 03:28:00 GMT

Now that you've been able to go check out our PDC talk (online here), I'm excited to introduce you and this blog to ELS, our next big set of globalization support for developers! From our PDC Developer Guide:

Extended Linguistic Services (ELS) is a new feature in Windows 7 that allows developers to use the same small set of APIs to leverage a variety of advanced linguistic functionality. By using ELS APIs in Windows 7, developers can auto-detect the language of any piece of Unicode text and use that information to help make smarter user experience choices for customers around the world. ELS also offers built in transliteration support that converts text from one writing system to another. For instance, developers can now auto-convert text between Simplified and Traditional Chinese to help people communicate with each other across linguistic boundaries. By using ELS APIs, developers will be able to use existing ELS services as well as pick up new services in the future without learning new code.

You can also read more about it on our Go Global website here: http://msdn.microsoft.com/en-us/goglobal/dd156834.aspx

The white paper takes you through the basics of what we're building and some places where we think you might find it helpful, and also offers some code samples that those of you who are planning to play with the Windows 7 beta bits will be able to try out. Some of the first ELS functionality that's coming your way includes:

Transliteration functionality that will let you convert automatically between Simplified and Traditional Chinese, allowing you to bring your Chinese content to an even greater number of customers without any localization costs on your part
Language detection support that handles every language with a supported Windows locale (100+!)
A simple API set that you will be able to learn once and use to call every service that ELS exposes, now or in the future

The first pieces developed by the ELS team are currently online for customers using the Chinese convertor over at the Windows Live Translator site, so if you want a taste of some of the support that ELS APIs will make available in Windows 7, give it a whirl.

I'm pretty excited about it! If you check it out, I'd love to know what you think. The functionality we're adding now is just the beginning of several services that we have in our roadmap. In the coming weeks, I'll be posting more specifically about our API set and services, so stay tuned.

PDC -- Windows 7: Writing World-Ready Applications

KieranS — Tue, 28 Oct 2008 00:58:00 GMT

If you're at the PDC this week, be sure to check out Yaniv Feinberg and Erik Fortune's talk, where they will introduce you to many of our new globalization features for developers in Windows 7. After their talk tomorrow (1:45pm-3pm Pacific Time), I'll be following up back here with a link to a white paper we've written to describe many of new features, complete with code samples that you'll be able to start playing with as soon as you have Windows 7 bits.

Hello (again) world! Also: The Hiring Chronicles.

KieranS — Fri, 22 Aug 2008 05:39:00 GMT

Hello world!

It's been a while, but I'm coming out of hibernation to mention a couple of things --

We've been working super hard on a bunch of new things and we're getting excited to show you some of what we've been doing at the PDC this October. Expect a pick-up in the number of blog posts related to some of the ideas that we're currently playing with, as well as some guidance on how to use some of the features that you've probably discovering in Vista. It's been a while since I've blogged, but that means I have a bunch of topics stored up, so get ready to ask questions, give feedback or suggestions, and add me back to or remove me from your RSS feed. :)

I also wanted to mention for all aspiring PMs out there that we're hiring. If you consider yourself an engineering-oriented business thinker, or a business-oriented engineer, we want to hear from you. We have a few new projects and we're trying to find smart and passionate people to add to our team. If you're that person at the party saying "If I only worked for Microsoft, I'd fix Windows by doing [x]," this is your chance to make [x] a reality. We work on core Windows and .NET features, so empathy with software developers is a must. We're thinking about all new ways to enrich the set of developer features across a variety of programming models, and we need creative, smart engineers to help us make that a reality.

You can take a look at a couple of job listings here and here. You can also email me with any questions that you have. I'd love to hear from you.

More to follow soon!

Shameless self-promotion.

KieranS — Fri, 12 Oct 2007 23:47:00 GMT

And now, some shameless advertising!

Our team is looking for a good tester to work on our next set of projects. We have some pretty cool stuff in the works for the next generation of globalization support in Windows and .NET, including more comprehensive linguistic functionality and new language coverage as well as additional support in the traditional NLS areas of locales, sorting, and calendars. We work on the core globalization APIs in Windows, .NET, and Silverlight, so there is a lot of opportunity to help shape the developer experience across platforms. If you have developer empathy and an interest in international software, please read the full job description and let us know if you're interested!

Vosotros? Ustedes? Who, me?

KieranS — Tue, 02 Oct 2007 01:10:00 GMT

For various reasons I've been doing statistical analyses on word frequencies in a number of languages lately, and I found a pattern that differentiates different dialects of Spanish that struck me as curious enough that I asked Gerardo, my test lead from Mexico, about it. I was working with Spanish corpora from Spain and from Mexico, and I noticed statistically significant patterns that differentiate the usage of vosotros and Ustedes across the two varieties of Spanish. The frequency of vosotros in the Spanish corpus was actually about two hundred times greater than it was in the Mexican corpus, for corpora of the same genre (i.e. corpus type difference does not account for greater/lesser presence of second person writing).

I had heard anecdotally that people don't use vosotros in Mexico, and come to think of it I noticed this when we traveled there last spring, kinda -- I mean, pretty much everyone talked to us formally anyway, so I didn't have a lot of occasion to pay attention. Still, I had no idea that the generalization of Usted/es to informal speech patterns was so far along in Mexico. So now I'm wondering whether there is anything pragmatic associated with the use of vosotros for Mexican speakers of Spanish, apart from the obvious not-from-around-these-parts marking. I don't have time to follow up on the corpus analysis on this point in detail, but I'd be interested in reading about it if someone else has.

A to-do list.

KieranS — Sat, 29 Sep 2007 00:21:00 GMT

The last several months have been packed with planning our next set of features in the international space -- talking to customers about what they need and what they want, looking at the teams we have and seeing where our skills are up to the challenges ahead and where we need improvement, and looking at the initial Vista feedback as it begins to roll in. We're seeing some good stuff and some bad stuff. A few themes:

People want to use strings: We've been pleased at the initial adoption of our name-based NLS APIs both in Windows and in .NET. Most existing components still rely on LCID-based support, but new components are increasingly starting to rely on names, and even several existing components are starting to plan for a migration to strings. This is crucial as the rate of custom locale adoption will only increase over the next few years, and in order to benefit from the full range of locales that a user has installed, it will be vital that applications ask for international support using strings rather than LCIDs. Legacy LCIDs aren't going away any time soon, but there will come a time when new locales are identified with strings and strings only.

But strings have to be reliable: As we have seen more than a few times in the last several years, sometimes there are changes in the world situation that require changes to software. A geography that is united as one jurisdiction today may vote to become distinct nations in a month or a year or ten years. We have to make it easy for developers to call us for reliable globalization support for customers even when borders shift and the customers' country of residence is updated. We have done some things well here and other things not so well, so figuring out how to handle these situations at an infrastructure level is one of our key focuses.

Locale Builder is great, so where can I get it? We've gotten a lot of great questions about Locale Builder and we're seeing a number of customers interested in custom locale technology. We're seeing questions from customersacoss the spectrum: from OEMs, from application developers who need to deploy custom locales with their applications, and from users whose languages and regions do not have existing locale support. The initial beta of LB expired, but you can get the current version on the Microsoft Locale Builder download page. Please let us know what you think.

The globalization support that people have today is not enough. People are using what Globalization Services provides today, but all the time we're getting requests for more. In particular, people want more core linguistic support in the OS. This includes updates to our sorting functionality -- we have to follow through on the versioning story that we've begun building -- and also requirements for linguistic functionality that developers cannot get from the operating system at all today.

We have an ambitious list of stuff that we need to work on, but it's also been exciting to hear from customers what is and is not working today. All our work for improvements to existing functionality, or the introduction of new functionality, has come directly from the feedback that we've received. We want to know how you're using our stuff and ways that the experience could be better. We're also interested in whether we have any gaps in existing support that are blocking you in some way.