Loneliness of the long-distance linguist. : sorting

To update or not to update?

KieranS — Sat, 08 Nov 2008 06:10:00 GMT

Can't we all just get along?

One of the struggles that we go through every release is in figuring out how to handle updates to the behavior of linguistic sorting. People rely on NLS APIs to sort strings appropriately for display to users. That means that strings need to be in the right order, where the right order is defined as the order that's going to allow a user to locate some particular item in a sorted list. Most of the time the expected order for a user who speaks some particular language doesn't change a whole lot. A lot of the time the expected order reflects something that people learned in school, or see consistently in dictionaries and encyclopedias and phone books. Oftentimes the order of encoding in Unicode is pretty close to what users expect. And sometimes where the above do not apply, we find highly intuitive and aware native speakers who not only have an innate awareness of what to expect in terms of sort order, but can also articulate why.

Then there are all the other times.

And many of the languages for which technology support is more recent? Are the other times.

For at least one language that we added during Vista, there was not only no national standard available, but four published dictionaries used four different (and internally inconsistent) sort orders. The articulate and perceptive native speakers working with us? Did not agree with each other. For many other languages, literacy patterns are emergent enough that native speaker intuitions are quite unclear and often wrapped up in other issues of cultural identity. There are groups pushing to follow sort orders and literacy traditions from other languages with overlapping character sets. There are groups pushing to establish local standards. There are still other languages for which there are no groups at all.

And yet, in many of these places, people are using computers. And the moment people can read and enter text on the screens in front of them, they're gonna be enveloped in sorted data. In the file system. In Excel files and documents. In other application UI. And for all of those users in all of those languages, the lists need to be predictable and linguistically appropriate. When for whatever reason it's not predictable or appropriate, people tell us about it, and we have bugs to fix. Even for languages with long literacy traditions and relatively advanced technology support, every once in a while the group of people who own the language – and here I mean a speech community of native speakers, who may or may not be aligned to a governmental language authority – change how things work. Spelling reforms get all the media play, but dictionary order reforms happen too. And when that happens, sorting behavior that may have been appropriate at the time we started supporting it stops being appropriate for the users it was intended to support. And then there are all the new characters that get added into Unicode, as new scripts are encoded and new code points are added to existing scripts. These characters get encoded because users need them.

So every time we release a version of Windows or the .NET framework, we end up addressing some issues that reflect either changes in local expectations or bugs in things we’ve shipped before. For applications that rely on us for linguistic correctness of sorted strings, this is a good thing. It means that app developers never need to roll their own behavior for linguistic appropriateness – they need to know how to make the right CompareStringEx() call with the right locale information for their user and they’re done.

But then there are the databases. With indexes that can become corrupt and cause huge compatibility headaches for users.

So anyone who needs to persist sorted data, like people building indexes for a database, needs to rely on consistency of their index. When sorting behavior changes, the index can become corrupt, which means that anyone searching on it won’t find the strings that they expect to find. So all the goodness of linguistically appropriate sorting is great for applications that present strings to users for ordered display, but these updates can wreak havoc for databases that aren’t designed correctly. So many people who persist sorted data avoid linguistic sorting entirely, instead using ordinal sorting to ensure consistency of behavior across a given character set. In Windows 2003, we introduced GetNLSVersion(), which allows developers to query the particular version of sorting behavior that is present on the OS to decide whether or not to reindex. If the version number has changed, then some sorting behavior has changed since the last version, which means that databases probably want to reindex to avoid the risk of corruption.

We hope that index builders are sorting ordinally, but what if they’re not? We think that many databases are using SQL or Jet or some other technology that shields them from having to think about the issue, but what if they aren’t? We believe that most databases go offline to reindex at major OS releases, but what if they don’t? We want people to check the NLS version to make informed decisions about reindexing, but what if people don’t know that they can do this? We know that many more people rely on us for linguistic sorting for display than for persisting ordered indexes, but every release we end up stuck between a rock and a hard place, where we need to make updates to serve the needs of international customers, but we need to be cautious about updates… to serve the needs of international customers.

And that about sums up my week. J

Oh, the enumeration? Part two.

KieranS — Tue, 03 Oct 2006 23:56:00 GMT

A few days ago, in Oh, the enumeration!, I wrote about some changes to the enumeration of our Spanish Traditional sort locale that would be new for Vista. What I wrote is that callers to EnumSystemLocales would have to pass the LCID_ALTERNATE_SORTS flag to return 040a (Spanish Traditional sort) in Vista. Well, based on customer feedback, we have decided to revert that change. That is, EnumSystemLocales will continue to behave as it behaved in XP, returning 040a even without the LCID_ALTERNATE_SORTS flag.

However, the name-based API counterpart, EnumSystemLocalesEx, will require the LCID_ALTERNATE_SORTS flag in order to return es-ES_tradnl, which is the locale name identifier for Spanish traditional sort (040a). That means that the LCID-based API and the name-based API are not strictly in parity, with EnumSystemLocales respecting previous behavior and EnumSystemLocalesEx introducing the desired behavior -- in which Spanish Traditional sort is properly treated as an alternate sort locale -- instead.

Oh, the enumeration!

KieranS — Tue, 26 Sep 2006 03:37:00 GMT

Note (10/3/06): This post contains outdated content. For the correction, please see the update in Oh, the enumeration? Part two.

If you use EnumSystemLocales(), one thing you may have noticed in Vista is that the Spanish Traditional sort locale (040a) is no longer included in our enumeration. The locale still exists, so you can ask for it specifically by LCID. But it is no longer enumerated in Vista. So if your code has some dependency on returning the locale in enumeration -- say, if you're enumerating all the locales on the system and relying on the returns from XP pushing forward onto Vista -- you'll want to be mindful of the change.

Note that you can still return this locale in our Vista enumeration function by passing the LCID_ALTERNATE_SORTS flag. If you pass this flag, you will return the Traditional sort Spanish locale as well as all other alternate sort locales available in Vista.

Why did we make this change? Good question.

When we first introduced the Traditional sort locale for Spanish, we made it identical to the locale for Spanish - Spain in all respects but two. First, the two locales used different sorting behavior, as their descriptions would indicate. At the time, we did not realize that we would find other locales that might require the support of multiple sorting standards or behaviors, so we introduced another key difference between the Spanish - Spain locale and the Traditional sort Spanish locale: the two locales share a primary lang ID, but they have different LCIDs.

In retrospect, in light of the many other alternate sort locales for which we have added support, this was not the right design call. Our model for other alternate sort locales is that they actually share all locale data programmatically with the locales on which they are based; the only difference is found in the sorting behavior itself.

Because the Traditional sort Spanish locale is more properly considered an alternate sort variant of the Spanish - Spain locale rather than a full-fledged locale with its own independently defined data, moving forward we would like to treat it like the other alternate sort locales that we support. We wanted to minimize the breaking nature of the change, so we are committed to continuing to support the locale if it is specifically asked for by LCID. However, we also wanted our Vista enumeration behavior to accurately reflect the true nature of the locale. Hence it is available when people enumerate asking for alternate sort locales, but it is not returned by a call to EnumSystemLocales without the LCID_ALTERNATE_SORTS flag.

Not all custom locales are created equal.

KieranS — Wed, 09 Aug 2006 20:12:00 GMT

If you're going to use the Locale Builder tool to create a custom locale, you have a couple of options: You can create a brand new locale for a language-region combination that is not currently supported by Microsoft, or you can replace a locale that exists on your machine with appropriately customized data. If you choose to add support for a language-region combination that's entirely new, you'll be creating a supplemental locale, designed to supplement the existing Windows list. If you choose to replace data for an existing locale, you'll be creating a replacement locale, designed to replace some locale that is already supported by Windows.

So how do you know which you really want?

A replacement locale is a good choice if you want to continue identifying the locale with its Windows name and LCID; the crucial property of a replacement locale that cannot be changed is its identifier (e.g. en-US, ii-CN, ja-JP, etc). This means that applications will be able to call our APIs with existing Windows identifiers and on your machine they will return the data that you have provided rather than the data that shipped with Windows. There are a few other properties that cannot be changed if you choose to create a replacement locale. One of the most crucial of these is code page assignments. We know that you wouldn't dream of developing using anything but Unicode :), but just in case you're still maintaining or running ANSI applications, we don't want those applications to break because code page assignments have changed.

Internally, we also try not to change sorting behavior for existing locales between major versions -- and if we discover that there is customer need for providing custom sorting behavior in the future, we'll have to think very hard about what to do here for replacement locales. When we change sorting behavior for existing locales -- say, if we discover that our collation isn't linguistically accurate -- applications that rely on consistent sorting results need to make changes to accomodate our updates (e.g. database applications that need to reindex whenever sorting behavior changes). So we want to keep this kind of update to a minimum. (We are introducing APIs that will let applications know which version of our sorting behavior is on a particular machine, which will help mitigate this, but that's fodder for another post.)

Supplemental locales require you to choose identifier names that do not overlap with the names of locales already on your machine. We recommend that you choose identifiers that conform as much as possible to the IETF standard; one reason that our names generally follow this model is that people recognize and use them. If you want your custom locale to be readily accessible to applications, using a standard identifier name is a good start. Once you've changed the name, you can customize code page assignments or anything else, since we don't run the same kind of a risk of breaking applications that are relying on long-standing settings.

In the end, whether you choose to create a supplemental locale or a replacement locale depends on your needs; it is just as easy to create, share, and install either of them.

So I'm ready to test-drive Windows Vista. What next?

KieranS — Tue, 01 Aug 2006 05:40:00 GMT

Many of our Windows Vista beta testers contact us excited to be test-driving the beta but wondering what to look for. I thought it might be nice to share here the test cases that we give interested beta testers. If you try these test cases and you have feedback for us on any aspect of our basic NLS/language enablement support, please let me know either in the comments section here or by sending me email directly.

***

NLS data testing for Beta partners

This is a collection of usage scenarios that beta testers might like to try in order to verify that their locale data is correct.

Fonts

If you do one thing: Look at sample representative strings and longer pieces of text in Notepad in Vista (or in XP with ClearType turned on) at 8pt (Office UI) and 9pt (Vista UI). Are the strings readable?

If you do more: For Latin, Cyrillic, and Greek, the font to look at is Segoe UI. Look at all four styles (the italic, bold and bold italic as well as the regular). Identify any bad character shapes, or unacceptable clipping of diacritics. If you can do more, also check out strings using Microsoft Sans Serif and MS UI Gothic.

In Vista, one good way to test the UI font is by creating/renaming folders with strings set in the relevant languages.

Keyboards

If you do one thing: Make sure that you can select the keyboards that are important for your locale. Type using the keyboard you have selected and make sure that the output matches what you expect from the layout.

If you do more: Verify that the keyboards for your locale match the standards that you expect to find.

Regional Options

If you do one thing: Open up Regional Options and change your user locale. Is your locale on the list? Is it in the form that you expect? Are the sample data displayed what you expect?

If you do more: Click customize in Regional Options. Are the default values what you expect?

Are you able to edit and save these values?

Change your user locale (Standards and Formats) setting and save settings. Does your formatting change as you expect it to (e.g. in Windows calendar)?

Are the date formats that you expect available? Are there any errors in month or day day names?

If you speak a language that uses genitive forms for month names, are the month names displayed correctly in date strings?

Open up your Windows calendar. Does it look like you expect it to?

Sorting

If you do one thing: Create several filenames using characters from your language. Do they sort properly in Explorer?

If you do more: Create a list of words using characters from your language in Word or Excel. Sort the list. Does it sort properly?

***

If you try any of this stuff and have feedback for us on what we're doing right and not right, please let me know!