I'm not a Klingon ( ) : Custom Cultures / Locales / CultureInfo

SetLocaleInfo() is horrid, don't use it!

shawnste — Fri, 06 Nov 2009 17:54:00 GMT

I just ran into a bug with SetLocaleInfo() use, and it pretty much reminded me that SetLocaleInfo() stinks. Michael said it years ago and it's still true.

The only thing it's useful for is a "Regional and Language Options" type app, and there's already one of those in the control panel :) Some apps try to call SetLocaleInfo() to set some behavior, and then call the date/time APIs or whatever. One app changed the currency symbol while it ran reports. Problem is that SetLocaleInfo() changes the user preferences for every app in the user account. Now the poor user's going to be running some report or something and they're records will be formatted wrong.

Some apps try to "fix" the setting, and change it only momentarily, then change it back. Now the other app will have a couple odd results in the middle of mostly-correct results, and it'll be nearly impossible for them to figure out why it doesn't work. SetLocaleInfo() also makes things slow because apps have to reload their settings all the time. The app that's using SetLocaleInfo() needs to seriously wonder what happens if another app did the same thing and called SetLocaleInfo() right after you did. Obviously that wouldn't be what you'd expect, so please don't do it to everyone else.

Anyway, don't use SetLocaleInfo(). If there's some limitation you're trying to work around by calling SetLocaleInfo(), let me know.

-Shawn

Missing International Setting Registry Key?

shawnste — Tue, 29 Sep 2009 21:41:00 GMT

Some Zune users ran into a strange problem http://forums.zune.net/2/3/518796/ShowPost.aspx and it seems like the Nation Value in HKEY_CURRENT_USER\Control Panel\International\Geo isn't there on some Windows XP machines. That seems really wierd to me because if the International key is missing I'd expect odd behavior from the machine. We get the user locale from there, so that's not particularly good. For Zune it seems like the missing key causes a log-on problem with a "Zune can't sign in: this functionality is not available in your Region" because GetUserGeoId() ended up failing.

Anyway, odd user problems because of a missing HKEY_CURRENT_USER\Control Panel\International key can be fixed by adding HKEY_CURRENT_USER\Control Panel\International\Geo through regedit. (Or if it's a permissions problem, delete the key and readd it). Of course editing the registry can mess up your machine and all that. If you run the Regional and Language Options Control Panel you can reset your settings to the correct values and apply (I usually change to something else and them back to make sure it sees a change).

How did the key get missing anyway? That's a really good question, if you know please tell me! In at least one case it seems the OEM image may not have had the key, which is kind of bizarre. Maybe the out-of-box-experience got skipped or something. (The one that asks you where you live). Maybe a registry "cleaner" mangled the key or maybe something else did. We've seen cases where an enterprise installation messes with these keys, but usually those are pretty rare. Generally it is not a good idea to change these keys yourself, intl.cpl provides a way to change it, and it supports xml config files too: http://blogs.msdn.com/shawnste/archive/2007/04/12/configuring-international-settings-from-the-command-line.aspx In the future we'd like to remove our dependency on some of the stuff in this key, so don't be suprised if it changes in future versions.

- Shawn

What is Title Case?

shawnste — Tue, 18 Aug 2009 20:30:00 GMT

Disclaimer: I'm not an English teacher (that's my mom), so I'm sure my description of title casing in English probably has exceptions/variations.

Title casing has an interesting history in computer programming. Programmers like to use CamelCase to make variable names more readable, and, particularly amongst developers native to some languages, there's an idea that title casing is interesting, such as in String.ToTitleCase(), and in Windows 7, LCMapString(LCMAP_TITLECASE). Most title casing algorythms are linguistically bad, even in English. For other languages it's worse.

ToTitleCase() takes a very simple approach to title casing. Maybe in the future it'll be smarter, but for now it just uppercases the first letter in a group of letters, and tries to pay attention to non-letters and word breaks. It also tries to keep acronyms all upper-case.

Even in English this is a simplistic approach. The title of this post is "What is Title Case?" Is is supposed to be lower case, but ToTitleCase() would mess it up. Additionally unexpected word breaks or punctuation could trick the algorithm. Even the acronym test isn't complete since it just expects all-upper case and sometimes acronyms keep the lower case of the full title. Also it messess up names like DiSilva or McConnell. Contractions can also be messed up.

Outside of English, ToTitleCase() rapidly gets silly. In English we capitalize everything except articles, short prepositions and some other short words. In German it's just like a normal sentence, with only nouns getting capitalized, so the English slightly over-eager capitilization behavior becomes very over-eager. Other languages also can have letters before the main word, eg: l'État, so the ToTitleCase rules can mess up those words as well.

And then there're scripts/languages that don't even have an upper/lower case distinction, so ToTitleCase gets pointless.

Anyway, use care when using ToTitleCase(). It might work in some cases, but don't expect it to work linguistically, particularly globally, particularly in non-English cases. Also maybe we'll get smarter and figure out a more correct way to do it in the future.

-Shawn

Locale Builder and Two Letter ISO name and Three Letter Windows Language Name

shawnste — Fri, 15 May 2009 03:22:00 GMT

When you use the Microsoft Locale Builder tool to build a custom locale, it asks for a lot of fields. Two may not be obvious:

The Two Letter ISO Language name is permitted to be 3 letters for locales that don't have a 2 letter code (eg: haw for Hawaiian).

The Three Letter Windows Language Name is mostly used for in-box locales for things having to do with our build process, so you can pretty much pick anything. Mostly I just use the ISO code, but note that Windows tries to keep this value unique. Note: Do NOT use this 3 letter windows code in your application, instead use the ISO standard codes.

Cheating to UNinstall Custom Cultures / Locales

shawnste — Fri, 15 May 2009 01:15:00 GMT

In Cheating To Install Custom Cultures, I mentioned how to add the custom cultures without using CultureAndRegionInfoBuilder.Register(). Should you have any problems with a custom culture / locale and want to uninstall it but are having difficulty with an uninstaller or whatnot, this is how to get rid of it:

{Warning this edits system stuff and could mess up your computer if you aren't careful, or if the custom culture was required by some application.}

Warning, if your locale was installed with the Microsoft Locale Builder installer or another installer, you'll still have to run that uninstaller to make the system happy if you want to reinstall it that way. In other words, don't use this if it came through an installer.

1) Run intl.cpl (Regional and Language Options) and change to some other locale.

2) Get rid of the custom culture file.

a) Open an elevated command window (eg: press windows key, then type "cmd", then CTRL+SHIFT+ENTER, or right click on cmd and choose "Run as Administrator"

b) "dir %windir%\globalization\*.nlp" to see installed custom locales

c) "rename %windir%\globalization\fj-FJ.nlp fj-FJ.disabled" to disable a custom culture named "fj-FJ" (Fijiian (Fiji)).

d) After rebooting, if desirable you can then "del %windir%\globalization\fj-FJ.disabled". Often you can't just delete the .nlp file at first because it may be in use.

3) You can also clean the registry key though this isn't necessary, it won't work if it can't find the file:

a) Run regedit (warning: improper use can mess up your computer, etc.)

b) Expand all the + arrow thingies to get to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CustomLocale (each \ is a new level)

c) Select the value (eg: fj-FJ) you want to delete, then press the delete key.

Hope someone finds this helpful.

Shawn

Changing the currency symbol (Euro, etc) in Windows XP & Vista & Server

shawnste — Mon, 29 Sep 2008 19:27:00 GMT

Countries sometimes change which currency symbol they're using. This is most obvious for countries using the Euro (wikipedia currently says those are currently: Austria, Belgium, Cyprus, Finland, France, Germany, Greece, Ireland, Italy, Luxembourg, Malta, Netherlands, Portugal, Slovenia, Spain, Mayotte, Monaco, Saint Pierre and Miquelon, San Marino, Vatican City, Akrotiri and Dhekelia, Andorra, Kosovo, Montenegro, Saint Barthélemy, and Saint Martin)

Other countries have changed their currency symbol as well, either because of a political shift, currency devaluation or other causes. In the future Slovakia, Lithuania, Estonia, Bulgaria, Czech Republic, Hungary, Latvia, Poland and Romania are expected to adopt a Euro.

So what happens if you use a locale that changes currency? How do you get that set as your currency symbol in Windows or .Net?

The easiest solution is to use the Regional Options control panel (Windows Key + r then type intl.cpl and OK opens the control panel, or you can select it from the control panel). From intl.cpl you can use the advanced settings to change the format of the currency symbol and enter the Euro symbol. This change only impacts the current locale for the current user though, and has to be reset for each user or if the user changes their locale to something else and then back.

Another option for Vista, Server 2008 & .Net 2.0+ is to create a custom culture with the desired symbol. Ironically anyone with the locale already set isn't going to see the update because the old symbol is set in their currency. The advantage of the custom locale solution is that it provides the ability to update the currency name (and other data) as well as the symbol, and that it persists and impacts non-current locales.

So a complete solution is probably to create a custom locale and to also change the user override. I have a link to the custom locale tool at http://blogs.msdn.com/shawnste/pages/custom-cultures-vista-custom-locales.aspx or directly from http://www.microsoft.com/downloads/details.aspx?FamilyID=e4588c5e-8f21-45cc-b862-38df8d9bd528&displaylang=en or search for "Locale Builder" on msdn.

Silverlight Time Zone World Clock (Very Beta) Demo

shawnste — Wed, 10 Sep 2008 23:40:00 GMT

For my presentation of globalization of Silverlight at the Unicode Conference I wanted to make a quick Silverlight demo application that would show at least a little bit of globalization and not be too hard to write. My first choice was to find an existing app, and thought I was close when I found a pretty application, but it was always stuck in English and didn't respect the user settings :(.

Then I thought about making a world clock in Silverlight. I knew the Olsen tz database would provide the data, but I needed a map, so I did a live search for some maps. Most seemed out of date, I didn't know if I could use them, and I'd have to map latitude/longitude to the image. I sort of had a "duh" moment when I found VIEWS at http://www.codeplex.com/views. VIEWS is a silverlight wrapper for the Virtual Earth control. Virtual Earth (http://www.microsoft.com/virtualearth/) is really cool but, better yet, gives me latitude & longitude when you click. Serious overkill for a world clock, but oh, well.

It took me about an hour to figure out how to make a silverlight app that used VIEWS. Ironically this is the first time I've used the Visual Studio IDE to make a silverlight app. Most of the silverlight code I write is low-level, so I use a console based test tool and don't make "real" silverlight apps normally. After getting the flashy stuff done really quickly it took me a bit more effort to get the timezone database into a format I could read and use in the application.

My demo works for the most part, but has some serious bugs. I didn't worry about getting the daylight savings transitions to behave, so the demo can be off by an hour for a few weeks around the transition times (I only enabled checking the month, not the day rules). Also the tz database only has cities, not bounderies, so it can be hard to find the right data point. I added Seattle by hand so that it wouldn't show Vancouver, BC when I did the demo, but many places can be a bit unexpected. Clicking on Disney World in Florida (I just got back from vacation) will happily show you times for Havana, which probably isn't expected. You have to go all the way "up" to New York to get Eastern Time.

I called the demo "SilverTime" and stuck it on CodePlex at http://www.codeplex.com/SilverTime. Its kind of cool, so I'm hoping that other people will participate in the open project and fix some of the bugs or extend its features. There's some interesting potential in the app, and my bugs, although serious, aren't really that hard to fix. (I was just running out of time before my vacation :-)

Have Fun,

Shawn

How Come My "ț" (or Another Character) Doesn't Work in Code Page XXX?

shawnste — Wed, 04 Jun 2008 20:30:00 GMT

First of all, as I always suggest, Use Unicode when practical :) Then you don't run into these kinds of problems.

The "thing" to remember about code pages in general is that they were an early way to get characters to display in a readable way on CRT MS-DOS displays, or, before that, for teletypes and such. ASCII is a common representation, but most software developers realized that one of the bits wasn't being used, and extended ASCII in several standard, and not so standard ways. Usually those extensions were for characters that someone thought were useful, but then other users discovered that some characters didn't "work" for their language and invented a variation of a code page, after all, all you had to do was change a bitmap font. Often times subtle distinctions between characters were lost, or users "made do" with the closest code page.

Sometimes the behaviors were pretty much a "hack". Some code pages represented right-to-left text, like Arabic, in a left-to-right fashion since their computer systems didn't really understand the concept of left-to-right text. On the CRT, in addition to using the 8th bit, MS-DOS reused the 1-31 code points for "symbols" since the ASCII values were invisible concepts like "bell" and "return". That hack allowed for card suits and console card games. Commodore did something similar with their PET fonts.

So what's this got to do with a Romanian ț (U+021B "Latin small letter t with comma below")? Well, code page 1250, "Eastern Europe" has a code point "0xfe" for ţ (U+0163 "Latin small letter t with cedilla"). I'm not sure of all of the history behind these characters, however they are different in appearance. I don't know if the 1250 ţ was originally intended for use with Romanian, however a cedilla isn't a comma below, and this distinction caused a seperate Unicode code point to be created. 1250 however still has its original meaning and U+021B isn't in it.

So, for Romanian, if you really want to use the "correct" U+021B character, then 1250 won't work (nor will any other non-Unicode code page). Those kind of subtle (to non-Romanian) issues are why its best to have Unicode applications and data stores. We can't really change the behavior of code page 1250, or else someone else's usage (even a Romanian application making do with the cedilla) will break.

MSDN Code Gallery Custom Locales

shawnste — Thu, 06 Mar 2008 05:12:00 GMT

MSDN has this new nifty Code Gallery place for samples and the like, so I stuck my custom locale examples there at http://code.msdn.microsoft.com/CustomLocales I added a few more since posting them to my blog here.

http://www.languagegeek.com/

shawnste — Thu, 20 Dec 2007 05:04:00 GMT

http://www.languagegeek.com/ says it is dedicated to the promotion of Native North American languages. A coworker ran into this site while she was trying to learn more about the Lakota language (and she made a Lakota custom locale too! :) I don't know much about the site, but apparently there are keyboards and other things that might be helpful to those of you trying to get keyboards or fonts or whatnot. I don't see custom locale files there yet, but maybe in the future ;-)

Cantonese and Manderin language tagging.

shawnste — Fri, 14 Dec 2007 21:49:00 GMT

The IETF "Language Tag Registry Update" working group has noted that lots of data is tagged as "zh-Hant", regardless of whether or not it is pronounced as Cantonese or Manderin. For video and audio however, this doesn't allow a fine enough distinction, and so the LTRU is working on revising RFC 4646/4647 and the registry to allow for new tags to distiguish Cantonese and Manderin from the "macrolanguage" of Chinese.

So in the future we should expect to see "cmn" and "yue" tags instead of zh. The LTRU is still a bit in flux about the details, but it is clear that in the future newly tagged data will use "cmn" and "yue". This is going to cause "an interesting time" since lots of legacy data, resources and systems will continue to use the zh tags.

User configurations may need to change, such as allowing both "cmn and zh" in a web browser's language configuration. Applications and systems may also need to change to provide "cmn" resources if "zh" was asked for, or vice versa. Content providers may also need to retag existing data to distinguish between Cantonese and Manderin.

With these types of changes, the adoption rate is usually quite varied, so expect some applications and content to shift rapidly to using the new recommended names once that new standard is created. Other data and systems will probably remain unchanged for a very long time, leading to very interesting scenarios when those environments communicate with each other.

zh-Hans, zh-Hant and the "old" zh-CHS, zh-CHT

shawnste — Thu, 13 Dec 2007 21:37:00 GMT

With Windows Vista and Microsoft .Net 2.0 (MS07-040 security patch) and 3.0+, we've started to use the IETF standard "zh-Hans", and "zh-Hant" names for Chinese simplified and traditional. In windows the zh-CHS/zh-CHT names were never used because the named APIs are new to Vista. Also in Silverlight we don't use the old names since Silverlight is new.

However in .Net 2.0/3.x we still recognize zh-CHS & zh-CHT for backwards compatibility. Additionally we prefer the "old" zh-CHS and zh-CHT names when enumerating or returning the name of a CultureInfo created by LCID (0x0004 or 0x7c04) The other oddity is that to recognize more resources, we made the parent of zh-CHS/zh-CHT be zh-Hans/zh-Hant.

This allows resources labeled zh-Hans or zh-Hant (the preferred name) to be loaded by systems that used the older zh-CHS/zh-CHT names. Unfortunately they can't be parents of each other, so this resource fallback only works one direction. You cannot normally find zh-CHS resources if you start with a zh-Hans locale. So the recommendation is to use zh-Hans/zh-Hant when creating resources.

In the future (like v4+, we're not sure when) zh-CHS/zh-CHT will no longer be recognized by default. Users will still be able to create zh-CHS/zh-CHT custom cutlures if necessary to workaround legacy naming related issues, similar to the way you can make an az-AZ-Latn now to work around that name change.

How do you make your regional and language options apply to new user accounts?

shawnste — Thu, 04 Oct 2007 19:30:00 GMT

In general its a good idea to allow users to choose appropriate settings, but being able to adjust the default user account settings to provide users with an appropriate default locale is often helpful. Also one cannot easily change the system account's settings.

The Regional and Language Options Control Panel (intl.cpl) has an administrative tab that you can then choose "Copy to reserved accounts" from. This copies the settings to the system and default accounts so that new users get those settings. I'd recommend testing on the local account to make sure that these settings are what you want. The label was slightly different in XP, but it does the same thing.

To do this in an automated manner see "Windows Vista Command Line Configuration of International Settings" is on-line on MSDN. The MSDN article http://www.microsoft.com/globaldev/vista/vista_tools/vista_command_line_international_configuration.mspx discusses changing options using an xml configuration file which you create.

What's a genitive month name anyway?

shawnste — Fri, 03 Aug 2007 21:34:00 GMT

I’m not a linguistic expert, so I’ll probably get this a bit wrong, but basically a genitive month name is used when there’s a number next to the month name. This doesn’t happen in English, but I think of it sort of like instead of saying “1 April 2008”, using “1 of April 2008”, where the “of April” is the “genitive month name”.

For most cultures the “genitive month names” and the “month names” are identical, and in .Net 2.0 DateTimeFormatInfo the “genitive” forms are internally delay created from the month name. Unfortunately what this means is that you cannot reliably set the DateTimeFormatInfo.MonthNames property unless you also set the DateTimeFormatInfo.MonthGenitiveNames property.

The example code demonstrates this by changing the month names and not the genitive names, then changing those as well.

MMMM dd format: April 01   
System.String[]   
w/o Genitive:   April 01   
with Genitive:  prilAy 01   

using System;   
using System.Globalization;   
public class Test   

{
     static void Main()
     {
         // Pick a date and show what it looks like normally
         DateTimeFormatInfo dtfi = new DateTimeFormatInfo();
         DateTime dt = new DateTime(2008,4,1,12,30,15);
         Console.WriteLine("MMMM dd format: " + dt.ToString("M", dtfi));

         // Force the delay create to happen in some versions of .Net
         Console.WriteLine(dtfi.MonthGenitiveNames);

         // Try to make the month names pig-latn (hey, its an example)
         dtfi.MonthNames = new string[]
             { "anuaryJay", "ebruaryJay", "archMay", "prilAy", "aMay", "uneJay",
               "ulyJay", "ugustJay", "eptemberSay", "ctoberOay", "ovemberNay", "ecemberDay", "" };
         // It doesn't work quite as expected
         Console.WriteLine("w/o Genitive:   " + dt.ToString("M", dtfi));

         // We also have to set the genitive names
         dtfi.MonthGenitiveNames = dtfi.MonthNames;
         Console.WriteLine("with Genitive:  " + dt.ToString("M", dtfi));
     }
}

"Windows Vista Command Line Configuration of International Settings" is on-line on MSDN

shawnste — Wed, 01 Aug 2007 21:07:00 GMT

Generally its a good idea to let the users figure out their international settings, but sometimes they need to be managed in a command-line manner. Windows Vista Command Line Configuration of International Settings at http://www.microsoft.com/globaldev/vista/vista_tools/vista_command_line_international_configuration.mspx describes how to manage the international settings from the command line