I'm not a Klingon ( )

Unicode, IDN (IDNA), EAI (IMA) and Homograph Security

I wrote about IDN & Security before http://blogs.msdn.com/shawnste/archive/2005/03/03/384692.aspx but thought I'd share some of my more updated views about security of URLs/IDN/Unicode/Email addresses.

People haven't really bothered much with DNS or character based security when it was limited to ASCII.  I'm not sure if this because people just didn't think about it, or if they thought there wasn't a problem or whatever.  What security attacks happen have been regarded more as "oh, that's curious" rather than a real concern.  Basically there seems to be a presumption that a script, like the ASCII subset of Latin, are inherintly secure.  Therefore it would seem reasonable that if ASCII Latin can be secure, then other scripts, or mixed script environments have homographs, then those scenarios must be insecure and are therefore broken.

Latin and ASCII aren't Secure

The problem with that logic is that it's flawed.  Homographs exist in Latin/ASCII, however http://rnicrosoft.com tends to be regarded as "quaint and amusing" rather than a security problem.  (There used to be a web page there, dunno what happened).  Similarly g00gle or MlCROSOFT or whatnot can all happen in ASCII.  Some things can be done to ASCII to limit the risk, such as choosing fonts or making things lowercase, but that's not always possible. 

Strings are Typed and Read by Humans

Even if the scripts themselves are perfect, the strings we use with the scripts are not.  For example, users have to type them in, and they may or may not use upper or lower case (in cased scripts).  I heard one computer expert indicate that users should just figure out how to enter URLs in lower case, in Unicode Normalization Form C.  (Instead of addressing the problem we should educate all the users).  I wish he were joking.

Depending on the context, there are things you can do to ASCII only strings that can confuse users.  For example http://microsoft.secure.com isn't going to necessarily go to a Microsoft site.  http://secure.com/microsoft.com is a similar trick.

DNS isn't the only subject of these problems.  I get mail all the time in the form company@mail-servicing.com where "company" is a legitimate company and "mail-servicing" is the people they've contracted to send their bulk mail.  So it's impossible for me to determine if that's actually a good address for the company.  Even worse is when the mail contains a link.  "Provide feedback about your recent warrenty support to http://feedback-surveys.com/OEMsupport"

Strings aren't Even Strings

Sometimes what we click on isn't even related to where we end up going.  We've all seen phishing attacks that are look like mybank.com but go to an IP address that no one can tell if it's real or not.

Strings aren't Always Specific

In some environments strings often aren't even very specific.  I'm pretty certain that if I want a live.com account that I won't get shawn or shawns or even shawnsteele.  Instead I'll be shawn7935 or something.  There's another Shawn here at work that gets some of my mail from simple typos, let alone malicious intent.  There's a pretty good chance that Fred8374 could pass himself off as Fred8347 if he really wanted to.  

We've even been trained that strings don't even have to be close.  If I buy something on eBay from "JoesBestStuff", it takes some faith for me to pay SallySewing7@live.com (apologies if those are real accounts).  I've been quite amused at the varation betwee "seller's name" and the email sometimes.

Even when we expect them to be the same, there are many spellings for some words.  "Mohammed" is often transliterated differently to Latin.  Unless you deal with one quite often, you're likely to assume most spellings are the same.

Globalization of Strings

Now we've figured out that strings aren't secure, and we'll get tricked even if they were secure.  How does that change in a global environment, such as with IDNA or EAI/IMA strings?  Not much.

Sticking to Latin, you suddenly gain a bunch of look-alikes (homographs) by allowing non-ASCII values.  Strings like mícrosoft, mïcrosoft and mıcrosoft are all “close enough” to be convused, particularly at a quick glance, even more so if the user is conditioned to expect the "real" string.  E.g:  "Important security update for windows, go download it from Mícrosoft.com"  We're already expecting to see microsoft, so the few different pixels are easily missed.

For other scripts the problem can be much more severe.  Complex scripts can have simliar appearing strings, and many include numerous characters.  Chinese for example has enough characters available that it can be fairly easy in some cases to find a rare character that is similar in appearance to a common character which people have been preconditioned to expect.

"I Solved Homographs"

This leads to a typical problem for developers, particularly "Western" Latin-script based developers.  We tend to expect that if we solve script mixing so that we can't mix up Cyrillic and Latin, that we've solved the homograph problem.  Instead, we've barely scratched the surface and effectively buried our heads in the sand.

In some cases the "solution" can be worse than the problem.  For example, some browsers decide that I don't understand Cyrillic since my user locale is en-US (or Klingon), and then prints out punycode.  That's mildly useful to me as a warning, however it does the same thing for Chinese.  It's very unlikely that I'm going to confuse Chinese with Latin, but I'll get Punycode in the address bar anyawy.  Now I have no chance of finding out what the actual URL is supposed to look like.  Punycode is all gibberish, but I could probably decipher a Chinese glyph enough to see if it looked similar to what I expected.  With any punicode strings, I don't even need homographs to confuse me, any Chinese would look the same.  For that matter I could be expecting Chinese, but it could actually be Japanese or Korean, or Cyrillic for that matter.  I'm not trying to say that the browsers' approach is "wrong", just that while this approach may address some problems, it can also cause new ones.

Most of the "solutions" to Homographs that I've seen are similar in my opinion.  They may address a specific issue, but don't solve the entire problem globally.  I also think some approaches are unnecessarily limiting.  Mitigations that reduce the surface area for an attack are useful, however developers should recognize the limitations of those approaches and make sure they aren't spending tons of effort "shutting the window, but leaving the front door wide open."  That only provides a false sense of security, which can be far worse than the original problem.

Comprehensive Solutions

So instead of thinking that strings like URLs are inherintly secure somehow if they're ASCII, and focusing on the differences from ASCII, like Cyrillic homographs, we should rather assume that ANY URL might not take us to a place we want to go.  Even an ASCII one.

A much better solution to URL security is one that addresses the entire system rather than focusing on Homographs.  IE, for example, detects malicious web sites (I don't know exactly how it works, but I gather there's blacklisting and bad behavior detection, kinda like virus checking for web sites).  This is far more effective than preventing mixed scripts, and has the advantage of working with ASCII only URLs.  It also does a good job against homographs, pretty much making the punicode-in-the-address-bar irrelevent.  It also works with many forms of attack, even non-obvious ones. 

My opinion is that if you do a "good job" of detecting any phishing/spoofing type web site, even ASCII-only, then the need for Homograph detection is much reduced.  And if you can't do that, then the attackers will merely add an extra label or something to get around your homograph detection.

Mitigation by Protocol

For things like IDN, it is interesting to consider how the protocol itself approaches security.  Some things are "obvious" as not being interesting for a name.  Compatibility characters, control characters, etc. could somewhat readily be excluded.  Some things are generally considered technically "obvious" to some users, but may frustrate others.  It is generally considered that lower casing the DNS name causes less confusing (can't mix up lower case l with capital I), but I doubt that AAA.com prefers lower casing.  Similarly IDNA2003 allows unicode "symbols," which are widely regarded as being useless, particularly since they're hard to type, but I suspect that someone would like I♥NY.  So there's a gray area that gets a bit confusing.

Consideration for other protocols is similar.  EAI (email) is interesting because it basically defers "correctness" to the registrar (whoever runs the mail server).  IDN provides some restriction by protocol and more at the registrar level.

One problem with restricting valid characters at the protocol level is that it works OK in a small set, but once you get to a global audiance the rules get very complicated.  Domain names allowed (most) English names when they were restricted to ASCII, but German and French had difficulties.  With IDN additional languages are supported, but perhaps the needs of an English registrar and a German one differ.  A complete set of rules applicable world-wide for all strings in all languages may not be possible (eg: turkish i), but even if they were, they would be very complex and difficult to implement for every application adopting a protocol.

Mitigation by Registrar

Restriction at the registrar can be more effective, though perhaps less consistent.  A registrar could be like a domain name registrar, but for these purposes you could also think of the person that assigns user accounts at a business, or email address registration from your ISP.

Registrars can restrict languages to those used in the country they support.  They can bundle or block homographs or alternate spellings (like Traditional and Simplified Chinese spellings of the same word.)  In a business they could have certain rules.  First name, last initial, or first initial, last name is common for user accounts in many companies, at least until they get too many employees).

IDN has some restrictions by protocol, but allows much tighter restriction at the registrar level.  Ironically, a label at a lower level could then have different "rules" than at the higher level.  EAI allows the local part to be determined entirely by the provider/registrar rather than the protocol.

Rules at the "registrar" level can still be very complex for a complete set of rules, however cases with conceptual differences can still be adopted as applicable for the registrar's environment, whereas a protocol level rule has to either be too flexible, or disallow one registrar's legitimate scenario.  Rules at the registrar level can also be adjusted more readily than at the protocol level.

Mitigation by Application

An application can also decide to be more comprehensive than the protocol.  An application may also have more information, such as blacklists or user settings.  They can make choices for some users like "they only read English, so don't bother with Cyrillic then," and a different choice for a different user.  Applications can also potentially be grayer in their behavior.  Instead of "allowing" and "disallowing" strings, they can say "gee, I'm not so sure, you really want to do this?", or flag it and continue.  They can also be dynamic, such as when you add a sender to a junk mail filter.

IDN vs EAI/IMA vs Unicode

Pretty much this entire "strings aren't secure" concept applies to any Unicode (or for that matter any other code page) string.  That could be an IDN domain name, an EAI mail address, a user account name, etc.  Some environments may be more ameniable to certain solutions than others, but the types of attacks that impact a Unicode IDN label could also succeed with the local (user name) part of a Unicode EAI email address.  The general concepts are portable.

I used IDN heavily as an example, but the same things happen to EAI addresses, user account names, logon credentials, etc.  Anything that uses Unicode, or strings, needs to realize that strings can't be expected to be inherintly "secure."

There's more info on some thinking about Unicode Security in Unicode TR#39 http://www.unicode.org/draft/reports/tr39/tr39.html.  TR39 addresses the appropriate use of Unicode characters and homographs, but this is at best a mitigation of the more general security concerns of identifier strings.  Phishing and spoofing would still happen even in plain ASCII.

Hope this was helpful, or at least interesting,

Shawn

 

A helpful reader pointed out I don't really know Klingon.

PS. I just checked out your blog (very nice by the way, lots of stuff I need to read) and I noticed along the top of the page you have  (jItlhInganbe') for "I'm not a Klingon". The translation your looking for would be   (tlhIngan jIHbe' - I am not a Klingon).

Thanks, fixed it :)  As I said, I'm not a Klingon, and had a terrible time finding something to work for "am" in The Klingon Dictionary or one of the on-line lessons I found.

Although that's kind of like one of those odd things that always strikes me as funny when traveling. 

Me: "Can you tell me where a good restaurant is?"
Native: "Sorry, I don't speak English". (Sometimes even without an accent)

Hmm.  I'll continue to have to rely on others for Klingon translations :)

Email Address Internationalization / Internationalized eMail Addresses (EAI/IMA)

With the IDN work for Internationalized Domain Names using characters beyond ASCII, it is only natural to tackle the problem of Internationalized Internet eMail.

Some smart people have been working on an IETF working group to figure out how non-ASCII email would work, and I encourage people to take a look: http://www.ietf.org/html.charters/eai-charter.html.  That page has the charter, a list of drafts and RFCs that have already been produced, and links to the IMA working group mailing list.

Assuming you're an ASCII/Latin character user, imagine having to type all your URL's in Chinese, or Cyrillic (or if you know those, imagine typing everything in Klingon, eg:  )  In many cultures, that's what it's like to use the web.  Some users may not be literate in Latin letters, or may have to do a lot of hunt-n-pecking.  EAI should help address that problem.

How EAI/IMA Works

The basic idea of the EAI working group is to stick email in UTF-8 instead of ASCII.  UTF-8 works pretty well in many systems, and many mailers already handle 8 bit encodings, so this is a pretty "simple" solution.  Unfortunately email touches a lot of places, so there're a lot of protocols that need updates (eg: STMP, POP, mailto:, etc.)  Additionally everyone knows that UTF-8 email can't happen instantly, so there needs to be a system for existing servers to talk to UTF-8 aware ones, which leads to a few more RFCs.

UTF8SMTP allows the servers to make decisions about the "local" part of the email address, which allows for groups to fit their own needs.  The backwards compatibility means that users also need ASCII addresses, as they do today.  The server would alias from one address to another so mail to @microsoft.com could map to my normal mailbox, and I'd only have one mail.  Unfortunately that simple concept means that places that didn't have to worry about aliasing before may now have to consider aliases and fallback addresses.  Contact lists may need to have both forms, etc.

Current Status of EAI/IMA

Currently there are several experimental RFCs, and several people have created interoperating systems that work with each other to demonstrate the feasibility of UTF8SMTP.   The next step is to move towards a standards track process, which could happen "reasonably quickly".  I'm optimistic that the standards will move quickly, but sometimes these things take a while.

So Who's Gonna Use It?

There are a lot of markets where ASCII doesn't work very well for various reasons.  Even when people have ASCII aliases, it may seem artificial, and there may be a desire for an email that reflects them or their country.  There are many ISPs in countries like Korea, China, & Japan that are very eager to be able to send email in a native script.  Some governments like Russia and China are weighing in on the importance of being able to send mail and use the Internet in their script. 

What's IMA Mean To Me As a Software Developer? (who cares?)

If you are a developer, then you may run into IMA addresses.  Even if your app doesn't explicitly deal with mail, there may be a place for email to sneak into your app.  For example, IDN and domain names don't really have much to do with Word or PowerPoint, yet they often show up in documents and presentations.  I could imagine an author address in metadata, such as a photographer contact in a photo's metadata.  Many apps probably will run into IMA addresses whether they realize it or not.

Anyway, I have been thinking about this space for a while and thought I'd share my observations.  It's worth considering what impact IMA will have on your application (while you're at it, how's IDN behave?)

 -Shawn

Writing "fields" of data to an encoded file.

The moral here is "Use Unicode," so you can skip the details below if you want :)

A common problem when storing string data in various fields is how to encode it.  Obviously you can store the Unicode as Unicode, which is a good choice for an XML file or text file.  However, sometimes data gets mixed with other non-string data or stored in a record, like a database record.  There are several ways to do that, but some common formats are delimited fields, fixed width fields, counted fields.  I'm going to ignore more robust protocols like XML for this problem.

A delimited field would be a character between fields that indicated that one field ended an another started.  Common delimiters are null (0), comma, and tab.  Using delimited fields, a list of names would look something like "Joe,Mary,Sally,Fred".

A fixed width field would be a field of a known size regardless of the input data size.  Generally data that is too short is padded with a space or null, and data that is too long is clipped.  If our "names" field was of fixed size four, then the previous list could look something like "Joe_MarySallFred".  Note the _ to pad the 3 character name, that Sally is clipped, and that the other names are "run together".

A counted field would indicate the field size for each piece of data before outputting the data.  The advantage is that it doesn't have the size restriction/clipping of fixed width fields, nor does it have to waste space with unnecessary padding.  (It could still be clipped for large strings as the count is likely restricted so some # of bits).  Similarly delimiters aren't a problem.  Generally the count is binary, but I'll show an example using numbers "3Joe4Mary5Sally4Fred"

A somewhat obvious way to store and read Unicode char or Unicode string data in the above formats is to write it in Unicode.  Counted fields can just count the Unicode code points to be read in.  Fixed width fields can similarly check for the space available and use Unicode character counts.   Delimited fields can also use Unicode.

When the desired output isn't Unicode (UTF-16) however, then you start running into some interesting problems.  Encodings (code pages) don't have a 1:1 relationship with UTF-16 code points, so you have to be careful.  Additionally some encodings shift modes and maintain state through shift or escape sequences.

For all of the fixed, counted, delimited techniques shift states cause an additional problem in that either the writer has to terminate the sequence, or persist the state until the next field.  Consider 2 fields where field 1 has some ASCII data that looks like "Joe" followed by shift sequence, then a Japanese character, and field 2 has "Kelly" in what looks like ASCII.  If the decoder retains the state between reading the 2 fields, it may accidentally read in "Kelly" as Japanese and presumably corrupt the output.  Alternatively if "Kelly" was really intended to read in "japanese" mode, then any application starting to read at field 2 gets confused since it didn't see the shift at the end of field 1. 

For that reason I like to make sure the fields are "complete", flushing the encoder at the end of each field (this is different than writing a pure-text document like XML).  So then field 1 above would have a shift-back-to-ASCII sequence at the end.

For fixed fields this could introduce another problem because the shift-back-to-ASCII sequence may exceed the allowed field size.  In that case the string would have to be made smaller before encoding to allow enough room for flushing.

For delimited fields there's an additional problem in that the delimiter could accidentally look like part of an encoded sequence.  Delimiters should only be tested on the decoded data.

For counted fields you start having trouble if the count isn't in encoded bytes.  If you counted the Unicode code points, then encode those code points, you don't know how many bytes to read back in when decoding.  It isn't possible to "just guess" when to stop reading data because there may or may not be some state changing data that you are expected to either ignore or read.  For example "Joe++" where ++ is a Japanese character could look like:

4<shift-to-ascii>Joe<shift-to-Japanese><+><+>, or
4<shift-to-ascii>Joe<shift-to-Japanese><+><+><shift-to-ascii>, or
4<shift-to-ascii>Joe<shift-to-Japanese><+><+><shift-to-mode-q><shift-to-mode-z><shift-to-mode-x>

where "4" represents the count, <+> represents the encoded character, and <shift...> indicates some sort of state change that doesn't cause output directly by itself.

Since the application doesn't know whether to expect the trailing <shift> sequence(s), it may not read enough data, and then may try to use <shift-to-ascii> as the count of the next field.  Similarly if it does see a <shift-to-ascii> and tries to read it in, then maybe it'll be confused if that was actually the count of the next field that just happened to look like a mode change.

So the moral is: Use UTF-16 because that's what the strings look like so they're less likely to get shifty about their sizes. 

• Use Unicode.  Either UTF-16, or maybe use UTF-8, though it still can change size and you have to be careful, but at least each code point represents a Unicode code point. 
• If you must count, try to count the actual encoded data size, not the unencoded form since that'll be confusing when decoding.
• Be good and flush your encoder if you must encode, so that the state gets back into a known state (usually ASCII) and then the decoding application doesn't get confused if they don't reset their decoder.
• Make sure you say which encoding you used.

Of course you may be talking to a GPS or something where you don't get to define the standard.  In that case you can just watch out for these caveats.  Should you be designing such a protocol however, make sure to use Unicode.  If that cannot happen, at least make sure to pay attention to the impact of encoding and decoding the data when the protocol's used.

-Shawn

Locale Builder and Two Letter ISO name and Three Letter Windows Language Name

When you use the Microsoft Locale Builder tool to build a custom locale, it asks for a lot of fields.  Two may not be obvious:

The Two Letter ISO Language name is permitted to be 3 letters for locales that don't have a 2 letter code (eg: haw for Hawaiian). 

The Three Letter Windows Language Name is mostly used for in-box locales for things having to do with our build process, so you can pretty much pick anything.  Mostly I just use the ISO code, but note that Windows tries to keep this value unique.  Note:  Do NOT use this 3 letter windows code in your application, instead use the ISO standard codes.

Cheating to UNinstall Custom Cultures / Locales

In Cheating To Install Custom Cultures, I mentioned how to add the custom cultures without using CultureAndRegionInfoBuilder.Register().  Should you have any problems with a custom culture / locale and want to uninstall it but are having difficulty with an uninstaller or whatnot, this is how to get rid of it:

{Warning this edits system stuff and could mess up your computer if you aren't careful, or if the custom culture was required by some application.}

Warning, if your locale was installed with the Microsoft Locale Builder installer or another installer, you'll still have to run that uninstaller to make the system happy if you want to reinstall it that way.  In other words, don't use this if it came through an installer.

1) Run intl.cpl (Regional and Language Options) and change to some other locale.

2) Get rid of the custom culture file.

a) Open an elevated command window (eg: press windows key, then type "cmd", then CTRL+SHIFT+ENTER, or right click on cmd and choose "Run as Administrator"

b) "dir %windir%\globalization\*.nlp" to see installed custom locales

c) "rename %windir%\globalization\fj-FJ.nlp fj-FJ.disabled" to disable a custom culture named "fj-FJ" (Fijiian (Fiji)).

d) After rebooting, if desirable you can then "del %windir%\globalization\fj-FJ.disabled".  Often you can't just delete the .nlp file at first because it may be in use.

3) You can also clean the registry key though this isn't necessary, it won't work if it can't find the file:

a) Run regedit (warning: improper use can mess up your computer, etc.)

b) Expand all the + arrow thingies to get to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CustomLocale (each \ is a new level)

c) Select the value (eg: fj-FJ) you want to delete, then press the delete key.

Hope someone finds this helpful.

Shawn

A Pet Peeve of Mine

One of my pet peeves is software that is too restrictive about installing.  The #1 compatibility thing I find is applications that refuse to install on a newer OS for no good reason.  Generally if you can get them to install OK then they run OK.

I feel better now,

Shawn

Don't use MB_COMPOSITE, MB_PRECOMPOSED or WC_COMPOSITECHECK

This pretty much demonstrates another reason to Use Unicode, but if you do need to use some non-Unicode encoding until you can convert to Unicode, please don't use these flags. 

MultiByteToWideChar() and WideCharToMultiByte() provide some interesting sounding flags that are actually useless, slow, badly broken, or far worse.  All of these flags would be expected to behave like Unicode Normalization, so you should instead use NormalizeString() to handle the desired behavior, either Form C for composed strings or Form D for decomposed strings.

MB_PRECOMPOSED is the simplest to address:  Basically this flag doesn't really do anything.  Nominally it would put data into something like Normalization Form C, however most code pages are already in a composed form, so there's little real impact.  Just to make sure, the flag's ignored internally :)

MB_COMPOSITE is my most hated of these flags.  First of all, it nominally pretends to put the data into something like Normalization Form D, decomposed into a base character and combining characters.  To me that's the opposite of "Composite".  Indeed, I've seen numerous code examples that seem to be passing MB_COMPOSITE expecting Form C data, and pretty much zero examples expecting Form D data.  Windows leans towards Form C internally (though you may use Form D or mixed data), so this flag probably isn't that helpful.  If you really want to decompose your data, then use NormalizeString with Form D instead of this flag.

MB_COMPOSITE also is very slow because it does a lookup in some data tables.  NormalizeString with Form D is probably faster.

MB_COMPOSITE also has some horrible behavior for many code points:

• Several code points will not round trip if this flag is set, even if WC_COMPOSITECHECK is used when converting back to the code page.
• Additionally its data tables are incomplete and inconsistent with the normalization
• Worse, some characters are decomposed into nonsensical sequences.
• Lastly some sequences decompose to strange choices, breaking some text.  Japanese is particularly impacted.

WC_COMPOSITECHECK basically has all of the problems of MB_COMPOSITE (its used in the other direction).  Its name isn't as annoying to me though.  Nominally WC_COMPOSITECHECK puts the data into Normalization Form C before encoding.  Since most code pages are in a composed form Normalization Form C isn't a bad idea, however please use NormalizeString with Form C instead of this flag.

WC_COMPOSITECHECK is also very slow because of the way it does lookup.  NormalizeString with Form C is probably faster.

WC_COMPOSITECHECK also has horrible behavior for many code points:

• It will convert sensible sequences into a form that, when round tripped by MB_COMPOSITE will end up in nonsensical forms.
• Sequences of 3 code points created by MB_COMPOSITE aren't correctly decoded by WC_COMPOSITECHECK back into their single code point form, resulting in extra ? when round tripping data.
• Several sequences map to a single code point, which MB_COMPOSITE will map back to a single form, so they won't round trip.  If you really need similar behavior try Normalization Form C, or KC if you really need the multiple mappings.  KC causes data to not round trip, so it might not be appropriate for all applications.  (Of course converting to the code page will also likely cause data to be lost so that may not matter so much).
• Again some sequences are composed in a strange form based on appearance rather than linguistics.  This could cause some unexpected behavior.
• Some scripts, like Japanese, are particularly impacted.

Hopefully I've terrified you and you'll stop using these flags, perhaps using NormalizeString() if you really need similar behavior.  Most applications don't even really need that though.  Of course you always have the option of Using Unicode!

'til next time,
Shawn

Front page uses windows-1252, shouldn't it be iso-8859-1?

I received this question:

I use Frontpage for my webpage design and FP automatically inserts the meta tag "<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">".
 
Should I have reference to ISO-8859-1 ?

I'm not a front page expert, and I can't answer all questions like this, however this is an common confusion.  Windows-1252 is very similar to ISO-8859-1, but they aren't identical.  Web sites and browsers have historically often treated these as equivilent, but they aren't, which is a great reason to use unicode for your encoding.  (No, I don't know how to make front page use UTF-8, but that'd be the best solution).  Looking on search.live.com (of course) for iso-8859-1 and windows-1252 will find some discussion of the differences.  Wikipedia has some articles (they change so I won't quote them directly, but their encoding related articles are usually informative and often accurate.)

Changing the currency symbol (Euro, etc) in Windows XP & Vista & Server

Countries sometimes change which currency symbol they're using.  This is most obvious for countries using the Euro (wikipedia currently says those are currently: Austria, Belgium, Cyprus, Finland, France, Germany, Greece, Ireland, Italy, Luxembourg, Malta, Netherlands, Portugal, Slovenia, Spain, Mayotte, Monaco, Saint Pierre and Miquelon, San Marino, Vatican City, Akrotiri and Dhekelia, Andorra, Kosovo, Montenegro, Saint Barthélemy, and Saint Martin)

Other countries have changed their currency symbol as well, either because of a political shift, currency devaluation or other causes.  In the future Slovakia, Lithuania, Estonia, Bulgaria, Czech Republic, Hungary, Latvia, Poland and Romania are expected to adopt a Euro.

So what happens if you use a locale that changes currency?  How do you get that set as your currency symbol in Windows or .Net?

The easiest solution is to use the Regional Options control panel (Windows Key + r then type intl.cpl and OK opens the control panel, or you can select it from the control panel).  From intl.cpl you can use the advanced settings to change the format of the currency symbol and enter the Euro symbol.  This change only impacts the current locale for the current user though, and has to be reset for each user or if the user changes their locale to something else and then back.

Another option for Vista, Server 2008 & .Net 2.0+ is to create a custom culture with the desired symbol.  Ironically anyone with the locale already set isn't going to see the update because the old symbol is set in their currency.  The advantage of the custom locale solution is that it provides the ability to update the currency name (and other data) as well as the symbol, and that it persists and impacts non-current locales.

So a complete solution is probably to create a custom locale and to also change the user override.  I have a link to the custom locale tool at http://blogs.msdn.com/shawnste/pages/custom-cultures-vista-custom-locales.aspx or directly from http://www.microsoft.com/downloads/details.aspx?FamilyID=e4588c5e-8f21-45cc-b862-38df8d9bd528&displaylang=en or search for "Locale Builder" on msdn.

Our Unicode Globalization Windows Language Support Presentation

Recently Peter Constable and I gave a presentation on Windows Language Support, which I've attached here.

Poornima Priyadarshini and I also gave a presentation in Globalization in Silverlight, which I've attached in the previous post.

Ironically this is too big to attach as PDF, and the previous post is too big to attach as a pptx.

Our Unicode & Globalization Silverlight Presentation

Recently Peter Constable and I gave a presentation on Windows Language Support, which I've attached in the next post.

Poornima Priyadarshini and I also gave a presentation in Globalization in Silverlight, which I've attached here.

How come Substring(0, xxx) matches something, but StartsWith returns false?

I was asked how a string can match a substring of another string, yet StartsWith can return false?  For example:

returns this:

So if you test the first 2 characters with the search string, you'll see that they match, yet StartsWith() returns false, and IndexOf can't find it.  This is because the 0308 diacritic is considered part of the u that it is modifying, so it won't be found.  In many languages diacritics like this are really different letters.  Since you don't expect a == z, then you wouldn't expect u == ü. 

Doing the substring effectively "breaks" the character, changing its meaning.  Substring can even create illegal Unicode if it chops off part of a surrogate pair (eg: U+D800, U+DC00).

A similar oddity would be characters with no weight like U+FFFD.  So if I have str = "A\xFFFD\xFFFD\xFFFD", then all of str.Substring(0,1) == str.Substring(0,2) == str.Substring(0,3) == str.Substring(0,4) == "A".  And in this case str.StartsWith("A") would be true.

Another perhaps unexpected behavior would be unweighted characters (or ignored by a flag) at the beginning of hte string.  So if str="\xFFFD" + "A", then str.IndexOf("A") can return 1, yet str.StartsWith() will return true (even though IndexOf didn't return 0).

Similar behaviors can be seen with LastIndexOf() and EndsWith(), and with the native Vista API FindNlsString and its variations.  In addition with the FindNlsString() API, the found substrings may be unexpected.

World Wide Telescope is really cool

This isn't really related to anything I talk about, but I thought that Microsoft Research's World Wide Telescope is pretty fun:  http://worldwidetelescope.org.

This is a free program you can install and then it lets you zoom in on celestial objects like a virtual planetarium.  It zooms pretty well and expands to lots of hubble and other photos, so you can drill down pretty far on parts of the sky.

Silverlight Time Zone World Clock (Very Beta) Demo

For my presentation of globalization of Silverlight at the Unicode Conference I wanted to make a quick Silverlight demo application that would show at least a little bit of globalization and not be too hard to write.  My first choice was to find an existing app, and thought I was close when I found a pretty application, but it was always stuck in English and didn't respect the user settings :(.

Then I thought about making a world clock in Silverlight.  I knew the Olsen tz database would provide the data, but I needed a map, so I did a live search for some maps.  Most seemed out of date, I didn't know if I could use them, and I'd have to map latitude/longitude to the image.  I sort of had a "duh" moment when I found VIEWS at http://www.codeplex.com/views.  VIEWS is a silverlight wrapper for the Virtual Earth control.  Virtual Earth (http://www.microsoft.com/virtualearth/) is really cool but, better yet, gives me latitude & longitude when you click.  Serious overkill for a world clock, but oh, well.

It took me about an hour to figure out how to make a silverlight app that used VIEWS.  Ironically this is the first time I've used the Visual Studio IDE to make a silverlight app.  Most of the silverlight code I write is low-level, so I use a console based test tool and don't make "real" silverlight apps normally.  After getting the flashy stuff done really quickly it took me a bit more effort to get the timezone database into a format I could read and use in the application.

My demo works for the most part, but has some serious bugs.  I didn't worry about getting the daylight savings transitions to behave, so the demo can be off by an hour for a few weeks around the transition times (I only enabled checking the month, not the day rules).  Also the tz database only has cities, not bounderies, so it can be hard to find the right data point.  I added Seattle by hand so that it wouldn't show Vancouver, BC when I did the demo, but many places can be a bit unexpected.  Clicking on Disney World in Florida (I just got back from vacation) will happily show you times for Havana, which probably isn't expected.  You have to go all the way "up" to New York to get Eastern Time.

I called the demo "SilverTime" and stuck it on CodePlex at http://www.codeplex.com/SilverTime. Its kind of cool, so I'm hoping that other people will participate in the open project and fix some of the bugs or extend its features.  There's some interesting potential in the app, and my bugs, although serious, aren't really that hard to fix.  (I was just running out of time before my vacation :-)

Have Fun,

Shawn