Loneliness of the long-distance linguist. : community solutions

A to-do list.

KieranS — Sat, 29 Sep 2007 00:21:00 GMT

The last several months have been packed with planning our next set of features in the international space -- talking to customers about what they need and what they want, looking at the teams we have and seeing where our skills are up to the challenges ahead and where we need improvement, and looking at the initial Vista feedback as it begins to roll in. We're seeing some good stuff and some bad stuff. A few themes:

People want to use strings: We've been pleased at the initial adoption of our name-based NLS APIs both in Windows and in .NET. Most existing components still rely on LCID-based support, but new components are increasingly starting to rely on names, and even several existing components are starting to plan for a migration to strings. This is crucial as the rate of custom locale adoption will only increase over the next few years, and in order to benefit from the full range of locales that a user has installed, it will be vital that applications ask for international support using strings rather than LCIDs. Legacy LCIDs aren't going away any time soon, but there will come a time when new locales are identified with strings and strings only.

But strings have to be reliable: As we have seen more than a few times in the last several years, sometimes there are changes in the world situation that require changes to software. A geography that is united as one jurisdiction today may vote to become distinct nations in a month or a year or ten years. We have to make it easy for developers to call us for reliable globalization support for customers even when borders shift and the customers' country of residence is updated. We have done some things well here and other things not so well, so figuring out how to handle these situations at an infrastructure level is one of our key focuses.

Locale Builder is great, so where can I get it? We've gotten a lot of great questions about Locale Builder and we're seeing a number of customers interested in custom locale technology. We're seeing questions from customersacoss the spectrum: from OEMs, from application developers who need to deploy custom locales with their applications, and from users whose languages and regions do not have existing locale support. The initial beta of LB expired, but you can get the current version on the Microsoft Locale Builder download page. Please let us know what you think.

The globalization support that people have today is not enough. People are using what Globalization Services provides today, but all the time we're getting requests for more. In particular, people want more core linguistic support in the OS. This includes updates to our sorting functionality -- we have to follow through on the versioning story that we've begun building -- and also requirements for linguistic functionality that developers cannot get from the operating system at all today.

We have an ambitious list of stuff that we need to work on, but it's also been exciting to hear from customers what is and is not working today. All our work for improvements to existing functionality, or the introduction of new functionality, has come directly from the feedback that we've received. We want to know how you're using our stuff and ways that the experience could be better. We're also interested in whether we have any gaps in existing support that are blocking you in some way.

Information central.

KieranS — Mon, 05 Feb 2007 21:24:00 GMT

Just in time for the Vista street date, the Vista International Support Portal is live at www.microsoft.com/globaldev/vista/vistahome.mspx! It contains lots of information about the locales, language packs, LIPs, fonts, and keyboards that we ship, and it makes a handy reference to figure out which languages and regions have support in Vista. You can also visit the portal to find free downloads of several of the internationalization tools that we provide, including the recently updated Microsoft Keyboard Layout Creator, the LocaleBuilder, the Microsoft Transliteration Utility, and the RC1 version of the Microsoft Phonetic Input Tool. Check it out and let us know what you think!

What comes around goes around.

KieranS — Fri, 17 Nov 2006 22:20:00 GMT

Yes, it's been a while. Since I've last blogged, we've shipped Vista and I've gone on a long vacation to Hawaii. Personally I find both of these things very exciting. It's understandable if you care more about the first than the second. :)

But I'm back from vacation and into planning mode for our next set of requirements and projects, so if as you're using Vista you find stuff about our international story that's missing/broken/just plain wrong and you haven't already told us, please let us know.

You might also like to check out the article that Shawn Steele and I wrote for MSDN magazine, Locale Hero: Enable Global Apps With Locale Builder And Windows Vista, now available in the December issue. The article gives best practices and common pitfalls to watch out for in designing custom locales, and it also provides some guidance on how to develop globalized apps that are equipped to take advantage of the full range and variety of locale data that you could see. If you have comments or questions regarding the material in the article, Shawn and I would love to hear what you think.

Lexical diffusion, software style.

KieranS — Thu, 17 Aug 2006 10:53:00 GMT

The Microsoft Community Glossary - Linguistic Partnership Project allows members of a speech community to work together to create and standardize technical vocabulary for their language. Community Glossaries can be an important step on the road to software localization for customers in regions that are newer to technology. Thus far there are online Community Glossary projects for languages all around the world, including Greenlandic, Kannada, Māori, Thai, Vietnamese, and several others. Interested participants work with a local moderator to decide on the appropriate terminology to be used in the software space.

So does it work? That's an interesting question.

New vocabulary is introduced into a language in several ways. Sometimes, rarely, someone well-known or otherwise influential coins a word or phrase that happens to stick. This can happen on many scales, ranging from the popular kid in a group of friends all the way up to celebrities who get media attention. Sometimes a new technology is introduced and some brand name associated with it becomes the generic term, as with kleenex, xerox, and more recently and much to their apparent chagrin, google.

More often, though, new vocabulary gets introduced when speakers of one language come into contact with speakers of another. English has borrowed vocabulary from virtually every language with which it has come into contact. No matter how government language authorities try to impose standards around vocabulary in advertising, media, and school curricula, such efforts have not generally been successful in achieving results in the real speech of real people. Mexican kids still speak Spanish during Little League games and the French still end the work week with le weekend. It's just how language works.

That's not to say that all standards efforts are ineffective. When a particular term catches on and its usage is popularized, it becomes a de facto standard for the speech community that uses it, and its very popularity both propels it forward and also catalyzes its eventual decline. Vocabulary choice is one way that speakers define themselves either as part of or against the speech communities they inhabit.

What is true for colloquial vocabulary is also true for terminology in the technology space. Terminology persists when people use it. Which brings us back to the Community Glossary efforts.

If efforts to identify and standardize technical vocabulary are driven solely by official language authorities, I don't think they will be successful in the long run. A word is only as present as the community that uses it. It is only by involving the target user community that standardization efforts can hope to identify relevant and enduring terminology, and the most successful products will be the ones that create a user experience with that terminology in appropriate ways. I'm very excited about the Community Glossary project because it lets real speakers identify real vocabulary that actually works for them, and it creates a lexical resource that can be valuable in language preservation efforts for endangered languages.

Anyone who believes that a Community Glossary effort would be beneficial for their language can work with their local Microsoft subsidiary office to request one and get the process started.

Not all custom locales are created equal.

KieranS — Wed, 09 Aug 2006 20:12:00 GMT

If you're going to use the Locale Builder tool to create a custom locale, you have a couple of options: You can create a brand new locale for a language-region combination that is not currently supported by Microsoft, or you can replace a locale that exists on your machine with appropriately customized data. If you choose to add support for a language-region combination that's entirely new, you'll be creating a supplemental locale, designed to supplement the existing Windows list. If you choose to replace data for an existing locale, you'll be creating a replacement locale, designed to replace some locale that is already supported by Windows.

So how do you know which you really want?

A replacement locale is a good choice if you want to continue identifying the locale with its Windows name and LCID; the crucial property of a replacement locale that cannot be changed is its identifier (e.g. en-US, ii-CN, ja-JP, etc). This means that applications will be able to call our APIs with existing Windows identifiers and on your machine they will return the data that you have provided rather than the data that shipped with Windows. There are a few other properties that cannot be changed if you choose to create a replacement locale. One of the most crucial of these is code page assignments. We know that you wouldn't dream of developing using anything but Unicode :), but just in case you're still maintaining or running ANSI applications, we don't want those applications to break because code page assignments have changed.

Internally, we also try not to change sorting behavior for existing locales between major versions -- and if we discover that there is customer need for providing custom sorting behavior in the future, we'll have to think very hard about what to do here for replacement locales. When we change sorting behavior for existing locales -- say, if we discover that our collation isn't linguistically accurate -- applications that rely on consistent sorting results need to make changes to accomodate our updates (e.g. database applications that need to reindex whenever sorting behavior changes). So we want to keep this kind of update to a minimum. (We are introducing APIs that will let applications know which version of our sorting behavior is on a particular machine, which will help mitigate this, but that's fodder for another post.)

Supplemental locales require you to choose identifier names that do not overlap with the names of locales already on your machine. We recommend that you choose identifiers that conform as much as possible to the IETF standard; one reason that our names generally follow this model is that people recognize and use them. If you want your custom locale to be readily accessible to applications, using a standard identifier name is a good start. Once you've changed the name, you can customize code page assignments or anything else, since we don't run the same kind of a risk of breaking applications that are relying on long-standing settings.

In the end, whether you choose to create a supplemental locale or a replacement locale depends on your needs; it is just as easy to create, share, and install either of them.

Lights, camera, action!

KieranS — Fri, 04 Aug 2006 01:44:00 GMT

For everything you ever wanted to know about Windows globalization and localization, or at least for one or two things, you might like to check out this Channel 9 interview. It features Michael Kaplan, whose blog you may know; Jeff Allen, our director of localization services; Alp Turkmen, who is driving our business direction in the Windows international space; Claus Juhl, one of our localization engineers; and me. Michael and I speak about globalization issues, with a heavy focus on the Locale Builder and associated ways that we'd like to open up our infrastructure to accomodate all kinds of community solutions, and we're about 45 minutes in. But don't skip! There's great information throughout.

If you watch, please let us know what you think! And yes -- there are closer to 7,000 languages in the world than 2,000, for those of you playing along at home.

Why wikipedia doesn't work?

KieranS — Thu, 27 Jul 2006 08:14:00 GMT

My last post produced a real onslaught of mail. It turns out that when it comes to community solutions in general and to wikipedia in paticular, people have no shortage of opinions. Curiously enough, it is the people who identified themselves as wikipedia contributors who were most likely to contend that the overall quality of information is subpar.

That's why I found CarlosT's comment interesting:

I've made some contributions to wikipedia on topics I really care about, and the motive has always been to improve the quality of those entries. I care about those topics enough that I want to make sure that the best, most accurate information is available. It's not so much for the joy in sharing information, but more that if a source of information is available, then I want the things that I care about to be represented and represented correctly.

I suspect that the motivations are similar for other contributors.

Maybe it's the people who are most likely to scrutinize who are also the most likely to make contributions. Or the other way: the people who make contributions are the ones watching, waiting to see if their contributions stick.

So bringing it back to something relevant to our projects, then. If there's one area where we know people have a hard time reaching consensus, it's around language use and the standards that govern it. Forget sort of nuanced stuff like pronunciation or vocabulary choice. We don't even have to dig that deep to find disagreements, disagreements that have ramifications all across the software space and beyond. There are regions of the world where there's regular flip-flop around such basic standards of language use as which writing system is used to record the language. Latin? Cyrillic? Arabic? In some areas of Central Asia it's all three.

I'm wondering how our customers will choose to manifest these variables in the solutions that they use our tools to create, and what we'll be able to learn from those solutions that in turn are actually used once they've been created. And I can't wait to see how the community editorial process is likely to work as those communities iterate on the solutions that they choose to adopt.

Why does wikipedia work?

KieranS — Wed, 26 Jul 2006 02:43:00 GMT

I had an interesting conversation this morning with a colleague about community approaches to large-scale projects. We were mainly thinking about software, and in particular, the pieces of software that are most susceptible to a modular approach using a set of tools and formats that are intuitive and standard. Things like the Locale Builder and Keyboard Layout Creator, for instance -- but we were really interested in thinking about how that approach might or might not scale with projects of increasingly ambitious scope, software or otherwise. It's an interesting question.

So I started thinking about wikipedia, which I'd call a resounding success for a community-driven and -owned knowledge base. I don't know how you feel about wikipedia coverage for your own areas of expertise. However, I am routinely impressed with the coverage in the areas that I feel qualified to assess, and the quality of the content in the areas that I can evaluate makes me more likely to take the content that is newer to me at face value. I used to work with someone who spent a lot of his free time contributing to wikipedia, and at the time it never occurred to me to ask him why.

Well, now I'm asking why.

It can't be for personal visibility for the author(s), because outside the wiki community the authors aren't really known -- and if they are known, it's for their knowledge in the area among other subject matter experts, and their wikipedia work doesn't add to that. I'm pretty sure that it isn't for the money, and I'd be hard-pressed to say that it's to learn, in that the authors are typically in content-generating or -sharing mode rather than anything else.

As I list and discard possible motivations, I'm left with really only one: joy in sharing one's own knowledge, as an end in itself. I was going to say that anyone who has ever enjoyed teaching or writing a book can probably understand this, but on second thought that isn't necessarily true. A teacher or a conventionally published author gets to experience lots of other things, including every motivation listed in the paragraph above (visibility, money, learning), plus a few more (the fulfillment that comes from connecting with people and seeing them grow, to name one). The anonymous author of wikipedia articles gets none of these things.

Yet the enterprise flourishes. It's something to think about.

Languages, regions, locales, oh my!

KieranS — Thu, 20 Jul 2006 23:12:00 GMT

Michael Kaplan has blogged quite a bit about our release of the Microsoft Locale Builder Beta, the final version of which is planned for use on Vista when it ships. We're pretty excited about the Locale Builder here for a whole bunch of reasons. Like The Microsoft Keyboard Layout Creator before it, the Locale Builder makes it possible for customers to create and share solutions faster and to ensure that the support available in Windows (support for formatting times, dates, currencies, and text, in the case of the Locale Builder) provides the best possible user experience across Windows applications. Straight up, you speak your language better than we do.

One of the pieces that I have been involved with since joining Microsoft is the investigation and maintainance of the NLS locale data that are used in Windows and .NET framework locales. Collecting data in an accurate, useful way poses many challenges. A whole array of things can impact language and technology standards in a region, including many factors whose rapid rate of change can make things interesting. Some things we need to keep in mind:

Changes in politics. Governments change at every scale. New leaders are elected; new representatives are appointed; new curricula are developed for school systems; official language legislation is passed and then revoked and then passed again.

Changes in standards. As technology evolves, new standards get created. As standards organizations change, new standards get created.

Changes in language or language use. Some languages are written in multiple writing systems. Vocabulary is created. A change in one part of the linguistic system can have ripple effects everywhere else in the system.

People move around. Just in my building, I work with native speakers of Spanish, French, Hebrew, Arabic, Chinese, Catalan, Hindi, Kannada, German, Portguese, Italian, Russian, Ukrainian, Japanese, Czech, Persian, and that's just off the top of my head. But they all work here, in Redmond. As people move around more and more, the usual pairings of language and region information inside a locale become less and less relevant. It's just not possible to predict the particular combinations that any individual user will want.

Even in a static universe, one size rarely fits all. I'm an en-US user, but I like a 24-hour clock. I'm weird like that.

The fact is that correct NLS data can be a moving target, and we're hoping that the Locale Builder can help address this for our customers. We're giving users the ability to change the locale data that we've shipped, not only for their own machines, but also for others, since they'll be able to share custom locales that they create. We're also giving users the ability to add new locales entirely, since we recognize that we provide just a small set of the possible languages and regions that our customers care about.

One thing I'm really interested in learning is which of these options will be more popular for our customers. Are people going to use the Locale Builder to modify existing Windows locales? In which case, which locales are people changing, and which pieces of data are they changing within those locales? Or are people going to use the tool to create new locales entirely? It's important to me that we understand what we're getting wrong today and what users feel that they need to fix.