I'm not a Klingon ( ) : Unicode and Code Pages/Encodings

Alternate encoding names recognized by .Net / IE

shawnste — Tue, 18 Aug 2009 20:07:00 GMT

If you run the sample from http://msdn.microsoft.com/en-us/library/system.text.encoding.getencodings.aspx then you can get a list of what Microsoft .Net thinks each Encoding/Code Page's name is. (WebName is more consistent to what's used in charset). eg:


using System;
using System.Text;

public class SamplesEncoding
{
   public static void Main()
   {
      // For every encoding, get the property values.
      foreach( EncodingInfo ei in Encoding.GetEncodings() )
      {
         Encoding e = ei.GetEncoding();
         Console.Write( "{0,-6} {1,-25} ", ei.CodePage, ei.Name );
      }
   }
}

There are several other names that are recognized by Encoding.GetEncoding() however, similar to what IE would recognize in a charset tag. I'm not sure if there's a way to get at the full list of aliases programatically, but this is what you'd get for these input strings:

Label	Code Page
"437"	437
"ANSI_X3.4-1968"	20127
"ANSI_X3.4-1986"	20127
"arabic"	28596
"ascii"	20127
"ASMO-708"	708
"Big5"	950
"Big5-HKSCS"	950
"CCSID00858"	858
"CCSID00924"	20924
"CCSID01140"	1140
"CCSID01141"	1141
"CCSID01142"	1142
"CCSID01143"	1143
"CCSID01144"	1144
"CCSID01145"	1145
"CCSID01146"	1146
"CCSID01147"	1147
"CCSID01148"	1148
"CCSID01149"	1149
"chinese"	936
"cn-big5"	950
"CN-GB"	936
"CP00858"	858
"CP00924"	20924
"CP01140"	1140
"CP01141"	1141
"CP01142"	1142
"CP01143"	1143
"CP01144"	1144
"CP01145"	1145
"CP01146"	1146
"CP01147"	1147
"CP01148"	1148
"CP01149"	1149
"cp037"	37
"cp1025"	21025
"CP1026"	1026
"cp1256"	1256
"CP273"	20273
"CP278"	20278
"CP280"	20280
"CP284"	20284
"CP285"	20285
"cp290"	20290
"cp297"	20297
"cp367"	20127
"cp420"	20420
"cp423"	20423
"cp424"	20424
"cp437"	437
"CP500"	500
"cp50227"	50227
"cp819"	28591
"cp850"	850
"cp852"	852
"cp855"	855
"cp857"	857
"cp858"	858
"cp860"	860
"cp861"	861
"cp862"	862
"cp863"	863
"cp864"	864
"cp865"	865
"cp866"	866
"cp869"	869
"CP870"	870
"CP871"	20871
"cp875"	875
"cp880"	20880
"CP905"	20905
"csASCII"	20127
"csbig5"	950
"csEUCKR"	51949
"csEUCPkdFmtJapanese"	51932
"csGB2312"	936
"csGB231280"	936
"csIBM037"	37
"csIBM1026"	1026
"csIBM273"	20273
"csIBM277"	20277
"csIBM278"	20278
"csIBM280"	20280
"csIBM284"	20284
"csIBM285"	20285
"csIBM290"	20290
"csIBM297"	20297
"csIBM420"	20420
"csIBM423"	20423
"csIBM424"	20424
"csIBM500"	500
"csIBM870"	870
"csIBM871"	20871
"csIBM880"	20880
"csIBM905"	20905
"csIBMThai"	20838
"csISO2022JP"	50221
"csISO2022KR"	50225
"csISO58GB231280"	936
"csISOLatin1"	28591
"csISOLatin2"	28592
"csISOLatin3"	28593
"csISOLatin4"	28594
"csISOLatin5"	28599
"csISOLatin9"	28605
"csISOLatinArabic"	28596
"csISOLatinCyrillic"	28595
"csISOLatinGreek"	28597
"csISOLatinHebrew"	28598
"csKOI8R"	20866
"csKSC56011987"	949
"csPC8CodePage437"	437
"csShiftJIS"	932
"csUnicode11UTF7"	65000
"csWindows31J"	932
"cyrillic"	28595
"DIN_66003"	20106
"DOS-720"	720
"DOS-862"	862
"DOS-874"	874
"ebcdic-cp-ar1"	20420
"ebcdic-cp-be"	500
"ebcdic-cp-ca"	37
"ebcdic-cp-ch"	500
"EBCDIC-CP-DK"	20277
"ebcdic-cp-es"	20284
"ebcdic-cp-fi"	20278
"ebcdic-cp-fr"	20297
"ebcdic-cp-gb"	20285
"ebcdic-cp-gr"	20423
"ebcdic-cp-he"	20424
"ebcdic-cp-is"	20871
"ebcdic-cp-it"	20280
"ebcdic-cp-nl"	37
"EBCDIC-CP-NO"	20277
"ebcdic-cp-roece"	870
"ebcdic-cp-se"	20278
"ebcdic-cp-tr"	20905
"ebcdic-cp-us"	37
"ebcdic-cp-wt"	37
"ebcdic-cp-yu"	870
"EBCDIC-Cyrillic"	20880
"ebcdic-de-273+euro"	1141
"ebcdic-dk-277+euro"	1142
"ebcdic-es-284+euro"	1145
"ebcdic-fi-278+euro"	1143
"ebcdic-fr-297+euro"	1147
"ebcdic-gb-285+euro"	1146
"ebcdic-international-500+euro"	1148
"ebcdic-is-871+euro"	1149
"ebcdic-it-280+euro"	1144
"EBCDIC-JP-kana"	20290
"ebcdic-Latin9--euro"	20924
"ebcdic-no-277+euro"	1142
"ebcdic-se-278+euro"	1143
"ebcdic-us-37+euro"	1140
"ECMA-114"	28596
"ECMA-118"	28597
"ELOT_928"	28597
"euc-cn"	51936
"euc-jp"	51932
"euc-kr"	51949
"Extended_UNIX_Code_Packed_Format_for_Japanese"	51932
"GB18030"	54936
"GB2312"	936
"GB2312-80"	936
"GB231280"	936
"GBK"	936
"GB_2312-80"	936
"German"	20106
"greek"	28597
"greek8"	28597
"hebrew"	28598
"hz-gb-2312"	52936
"IBM-Thai"	20838
"IBM00858"	858
"IBM00924"	20924
"IBM01047"	1047
"IBM01140"	1140
"IBM01141"	1141
"IBM01142"	1142
"IBM01143"	1143
"IBM01144"	1144
"IBM01145"	1145
"IBM01146"	1146
"IBM01147"	1147
"IBM01148"	1148
"IBM01149"	1149
"IBM037"	37
"IBM1026"	1026
"IBM273"	20273
"IBM277"	20277
"IBM278"	20278
"IBM280"	20280
"IBM284"	20284
"IBM285"	20285
"IBM290"	20290
"IBM297"	20297
"IBM367"	20127
"IBM420"	20420
"IBM423"	20423
"IBM424"	20424
"IBM437"	437
"IBM500"	500
"ibm737"	737
"ibm775"	775
"ibm819"	28591
"IBM850"	850
"IBM852"	852
"IBM855"	855
"IBM857"	857
"IBM860"	860
"IBM861"	861
"IBM862"	862
"IBM863"	863
"IBM864"	864
"IBM865"	865
"IBM866"	866
"IBM869"	869
"IBM870"	870
"IBM871"	20871
"IBM880"	20880
"IBM905"	20905
"irv"	20105
"ISO-10646-UCS-2"	1200
"iso-2022-jp"	50220
"iso-2022-jpeuc"	51932
"iso-2022-kr"	50225
"iso-2022-kr-7"	50225
"iso-2022-kr-7bit"	50225
"iso-2022-kr-8"	51949
"iso-2022-kr-8bit"	51949
"iso-8859-1"	28591
"iso-8859-11"	874
"iso-8859-13"	28603
"iso-8859-15"	28605
"iso-8859-2"	28592
"iso-8859-3"	28593
"iso-8859-4"	28594
"iso-8859-5"	28595
"iso-8859-6"	28596
"iso-8859-7"	28597
"iso-8859-8"	28598
"ISO-8859-8 Visual"	28598
"iso-8859-8-i"	38598
"iso-8859-9"	28599
"iso-ir-100"	28591
"iso-ir-101"	28592
"iso-ir-109"	28593
"iso-ir-110"	28594
"iso-ir-126"	28597
"iso-ir-127"	28596
"iso-ir-138"	28598
"iso-ir-144"	28595
"iso-ir-148"	28599
"iso-ir-149"	949
"iso-ir-58"	936
"iso-ir-6"	20127
"ISO646-US"	20127
"iso8859-1"	28591
"iso8859-2"	28592
"ISO_646.irv:1991"	20127
"iso_8859-1"	28591
"ISO_8859-15"	28605
"iso_8859-1:1987"	28591
"iso_8859-2"	28592
"iso_8859-2:1987"	28592
"ISO_8859-3"	28593
"ISO_8859-3:1988"	28593
"ISO_8859-4"	28594
"ISO_8859-4:1988"	28594
"ISO_8859-5"	28595
"ISO_8859-5:1988"	28595
"ISO_8859-6"	28596
"ISO_8859-6:1987"	28596
"ISO_8859-7"	28597
"ISO_8859-7:1987"	28597
"ISO_8859-8"	28598
"ISO_8859-8:1988"	28598
"ISO_8859-9"	28599
"ISO_8859-9:1989"	28599
"Johab"	1361
"koi"	20866
"koi8"	20866
"koi8-r"	20866
"koi8-ru"	21866
"koi8-u"	21866
"koi8r"	20866
"korean"	949
"ks-c-5601"	949
"ks-c5601"	949
"KSC5601"	949
"KSC_5601"	949
"ks_c_5601"	949
"ks_c_5601-1987"	949
"ks_c_5601-1989"	949
"ks_c_5601_1987"	949
"l1"	28591
"l2"	28592
"l3"	28593
"l4"	28594
"l5"	28599
"l9"	28605
"latin1"	28591
"latin2"	28592
"latin3"	28593
"latin4"	28594
"latin5"	28599
"latin9"	28605
"logical"	28598
"macintosh"	10000
"ms_Kanji"	932
"Norwegian"	20108
"NS_4551-1"	20108
"PC-Multilingual-850+euro"	858
"SEN_850200_B"	20107
"shift-jis"	932
"shift_jis"	932
"sjis"	932
"Swedish"	20107
"TIS-620"	874
"ucs-2"	1200
"unicode"	1200
"unicode-1-1-utf-7"	65000
"unicode-1-1-utf-8"	65001
"unicode-2-0-utf-7"	65000
"unicode-2-0-utf-8"	65001
"unicodeFFFE"	1201
"us"	20127
"us-ascii"	20127
"utf-16"	1200
"UTF-16BE"	1201
"UTF-16LE"	1200
"utf-32"	12000
"UTF-32BE"	12001
"UTF-32LE"	12000
"utf-7"	65000
"utf-8"	65001
"visual"	28598
"windows-1250"	1250
"windows-1251"	1251
"windows-1252"	1252
"windows-1253"	1253
"Windows-1254"	1254
"windows-1255"	1255
"windows-1256"	1256
"windows-1257"	1257
"windows-1258"	1258
"windows-874"	874
"x-ansi"	1252
"x-Chinese-CNS"	20000
"x-Chinese-Eten"	20002
"x-cp1250"	1250
"x-cp1251"	1251
"x-cp20001"	20001
"x-cp20003"	20003
"x-cp20004"	20004
"x-cp20005"	20005
"x-cp20261"	20261
"x-cp20269"	20269
"x-cp20936"	20936
"x-cp20949"	20949
"x-cp50227"	50227
"X-EBCDIC-KoreanExtended"	20833
"x-euc"	51932
"x-euc-cn"	51936
"x-euc-jp"	51932
"x-Europa"	29001
"x-IA5"	20105
"x-IA5-German"	20106
"x-IA5-Norwegian"	20108
"x-IA5-Swedish"	20107
"x-iscii-as"	57006
"x-iscii-be"	57003
"x-iscii-de"	57002
"x-iscii-gu"	57010
"x-iscii-ka"	57008
"x-iscii-ma"	57009
"x-iscii-or"	57007
"x-iscii-pa"	57011
"x-iscii-ta"	57004
"x-iscii-te"	57005
"x-mac-arabic"	10004
"x-mac-ce"	10029
"x-mac-chinesesimp"	10008
"x-mac-chinesetrad"	10002
"x-mac-croatian"	10082
"x-mac-cyrillic"	10007
"x-mac-greek"	10006
"x-mac-hebrew"	10005
"x-mac-icelandic"	10079
"x-mac-japanese"	10001
"x-mac-korean"	10003
"x-mac-romanian"	10010
"x-mac-thai"	10021
"x-mac-turkish"	10081
"x-mac-ukrainian"	10017
"x-ms-cp932"	932
"x-sjis"	932
"x-unicode-1-1-utf-7"	65000
"x-unicode-1-1-utf-8"	65001
"x-unicode-2-0-utf-7"	65000
"x-unicode-2-0-utf-8"	65001
"x-x-big5"	950

There's one really egregious name here. UnicodeFFFE is actually Big Endian UTF-16. It's like the byte order mark (BOM) for UTF-16BE written in little endian order. Try to use UTF-16BE instead :)

Note that historically lots of data on the web has been mis-tagged, or isn't tagged at all. For data from windows machines that data is often in the windows system code page, such as windows-1252. So sometimes browsers may attempt to use the current system code page, or try to guess (with varying degrees of success), the actual code page. Additionally there are differences between different vendor's code page behavior causing further ambiguity.

Unicode, IDN (IDNA), EAI (IMA) and Homograph Security

shawnste — Tue, 07 Jul 2009 22:33:00 GMT

I wrote about IDN & Security before http://blogs.msdn.com/shawnste/archive/2005/03/03/384692.aspx but thought I'd share some of my more updated views about security of URLs/IDN/Unicode/Email addresses.

People haven't really bothered much with DNS or character based security when it was limited to ASCII. I'm not sure if this because people just didn't think about it, or if they thought there wasn't a problem or whatever. What security attacks happen have been regarded more as "oh, that's curious" rather than a real concern. Basically there seems to be a presumption that a script, like the ASCII subset of Latin, are inherintly secure. Therefore it would seem reasonable that if ASCII Latin can be secure, then other scripts, or mixed script environments have homographs, then those scenarios must be insecure and are therefore broken.

Latin and ASCII aren't Secure

The problem with that logic is that it's flawed. Homographs exist in Latin/ASCII, however http://rnicrosoft.com tends to be regarded as "quaint and amusing" rather than a security problem. (There used to be a web page there, dunno what happened). Similarly g00gle or MlCROSOFT or whatnot can all happen in ASCII. Some things can be done to ASCII to limit the risk, such as choosing fonts or making things lowercase, but that's not always possible.

Strings are Typed and Read by Humans

Even if the scripts themselves are perfect, the strings we use with the scripts are not. For example, users have to type them in, and they may or may not use upper or lower case (in cased scripts). I heard one computer expert indicate that users should just figure out how to enter URLs in lower case, in Unicode Normalization Form C. (Instead of addressing the problem we should educate all the users). I wish he were joking.

Depending on the context, there are things you can do to ASCII only strings that can confuse users. For example http://microsoft.secure.com isn't going to necessarily go to a Microsoft site. http://secure.com/microsoft.com is a similar trick.

DNS isn't the only subject of these problems. I get mail all the time in the form company@mail-servicing.com where "company" is a legitimate company and "mail-servicing" is the people they've contracted to send their bulk mail. So it's impossible for me to determine if that's actually a good address for the company. Even worse is when the mail contains a link. "Provide feedback about your recent warrenty support to http://feedback-surveys.com/OEMsupport"

Strings aren't Even Strings

Sometimes what we click on isn't even related to where we end up going. We've all seen phishing attacks that are look like mybank.com but go to an IP address that no one can tell if it's real or not.

Strings aren't Always Specific

In some environments strings often aren't even very specific. I'm pretty certain that if I want a live.com account that I won't get shawn or shawns or even shawnsteele. Instead I'll be shawn7935 or something. There's another Shawn here at work that gets some of my mail from simple typos, let alone malicious intent. There's a pretty good chance that Fred8374 could pass himself off as Fred8347 if he really wanted to.

We've even been trained that strings don't even have to be close. If I buy something on eBay from "JoesBestStuff", it takes some faith for me to pay SallySewing7@live.com (apologies if those are real accounts). I've been quite amused at the varation betwee "seller's name" and the email sometimes.

Even when we expect them to be the same, there are many spellings for some words. "Mohammed" is often transliterated differently to Latin. Unless you deal with one quite often, you're likely to assume most spellings are the same.

Globalization of Strings

Now we've figured out that strings aren't secure, and we'll get tricked even if they were secure. How does that change in a global environment, such as with IDNA or EAI/IMA strings? Not much.

Sticking to Latin, you suddenly gain a bunch of look-alikes (homographs) by allowing non-ASCII values. Strings like mícrosoft, mïcrosoft and mıcrosoft are all “close enough” to be convused, particularly at a quick glance, even more so if the user is conditioned to expect the "real" string. E.g: "Important security update for windows, go download it from Mícrosoft.com" We're already expecting to see microsoft, so the few different pixels are easily missed.

For other scripts the problem can be much more severe. Complex scripts can have simliar appearing strings, and many include numerous characters. Chinese for example has enough characters available that it can be fairly easy in some cases to find a rare character that is similar in appearance to a common character which people have been preconditioned to expect.

"I Solved Homographs"

This leads to a typical problem for developers, particularly "Western" Latin-script based developers. We tend to expect that if we solve script mixing so that we can't mix up Cyrillic and Latin, that we've solved the homograph problem. Instead, we've barely scratched the surface and effectively buried our heads in the sand.

In some cases the "solution" can be worse than the problem. For example, some browsers decide that I don't understand Cyrillic since my user locale is en-US (or Klingon), and then prints out punycode. That's mildly useful to me as a warning, however it does the same thing for Chinese. It's very unlikely that I'm going to confuse Chinese with Latin, but I'll get Punycode in the address bar anyawy. Now I have no chance of finding out what the actual URL is supposed to look like. Punycode is all gibberish, but I could probably decipher a Chinese glyph enough to see if it looked similar to what I expected. With any punicode strings, I don't even need homographs to confuse me, any Chinese would look the same. For that matter I could be expecting Chinese, but it could actually be Japanese or Korean, or Cyrillic for that matter. I'm not trying to say that the browsers' approach is "wrong", just that while this approach may address some problems, it can also cause new ones.

Most of the "solutions" to Homographs that I've seen are similar in my opinion. They may address a specific issue, but don't solve the entire problem globally. I also think some approaches are unnecessarily limiting. Mitigations that reduce the surface area for an attack are useful, however developers should recognize the limitations of those approaches and make sure they aren't spending tons of effort "shutting the window, but leaving the front door wide open." That only provides a false sense of security, which can be far worse than the original problem.

Comprehensive Solutions

So instead of thinking that strings like URLs are inherintly secure somehow if they're ASCII, and focusing on the differences from ASCII, like Cyrillic homographs, we should rather assume that ANY URL might not take us to a place we want to go. Even an ASCII one.

A much better solution to URL security is one that addresses the entire system rather than focusing on Homographs. IE, for example, detects malicious web sites (I don't know exactly how it works, but I gather there's blacklisting and bad behavior detection, kinda like virus checking for web sites). This is far more effective than preventing mixed scripts, and has the advantage of working with ASCII only URLs. It also does a good job against homographs, pretty much making the punicode-in-the-address-bar irrelevent. It also works with many forms of attack, even non-obvious ones.

My opinion is that if you do a "good job" of detecting any phishing/spoofing type web site, even ASCII-only, then the need for Homograph detection is much reduced. And if you can't do that, then the attackers will merely add an extra label or something to get around your homograph detection.

Mitigation by Protocol

For things like IDN, it is interesting to consider how the protocol itself approaches security. Some things are "obvious" as not being interesting for a name. Compatibility characters, control characters, etc. could somewhat readily be excluded. Some things are generally considered technically "obvious" to some users, but may frustrate others. It is generally considered that lower casing the DNS name causes less confusing (can't mix up lower case l with capital I), but I doubt that AAA.com prefers lower casing. Similarly IDNA2003 allows unicode "symbols," which are widely regarded as being useless, particularly since they're hard to type, but I suspect that someone would like I♥NY. So there's a gray area that gets a bit confusing.

Consideration for other protocols is similar. EAI (email) is interesting because it basically defers "correctness" to the registrar (whoever runs the mail server). IDN provides some restriction by protocol and more at the registrar level.

One problem with restricting valid characters at the protocol level is that it works OK in a small set, but once you get to a global audiance the rules get very complicated. Domain names allowed (most) English names when they were restricted to ASCII, but German and French had difficulties. With IDN additional languages are supported, but perhaps the needs of an English registrar and a German one differ. A complete set of rules applicable world-wide for all strings in all languages may not be possible (eg: turkish i), but even if they were, they would be very complex and difficult to implement for every application adopting a protocol.

Mitigation by Registrar

Restriction at the registrar can be more effective, though perhaps less consistent. A registrar could be like a domain name registrar, but for these purposes you could also think of the person that assigns user accounts at a business, or email address registration from your ISP.

Registrars can restrict languages to those used in the country they support. They can bundle or block homographs or alternate spellings (like Traditional and Simplified Chinese spellings of the same word.) In a business they could have certain rules. First name, last initial, or first initial, last name is common for user accounts in many companies, at least until they get too many employees).

IDN has some restrictions by protocol, but allows much tighter restriction at the registrar level. Ironically, a label at a lower level could then have different "rules" than at the higher level. EAI allows the local part to be determined entirely by the provider/registrar rather than the protocol.

Rules at the "registrar" level can still be very complex for a complete set of rules, however cases with conceptual differences can still be adopted as applicable for the registrar's environment, whereas a protocol level rule has to either be too flexible, or disallow one registrar's legitimate scenario. Rules at the registrar level can also be adjusted more readily than at the protocol level.

Mitigation by Application

An application can also decide to be more comprehensive than the protocol. An application may also have more information, such as blacklists or user settings. They can make choices for some users like "they only read English, so don't bother with Cyrillic then," and a different choice for a different user. Applications can also potentially be grayer in their behavior. Instead of "allowing" and "disallowing" strings, they can say "gee, I'm not so sure, you really want to do this?", or flag it and continue. They can also be dynamic, such as when you add a sender to a junk mail filter.

IDN vs EAI/IMA vs Unicode

Pretty much this entire "strings aren't secure" concept applies to any Unicode (or for that matter any other code page) string. That could be an IDN domain name, an EAI mail address, a user account name, etc. Some environments may be more ameniable to certain solutions than others, but the types of attacks that impact a Unicode IDN label could also succeed with the local (user name) part of a Unicode EAI email address. The general concepts are portable.

I used IDN heavily as an example, but the same things happen to EAI addresses, user account names, logon credentials, etc. Anything that uses Unicode, or strings, needs to realize that strings can't be expected to be inherintly "secure."

There's more info on some thinking about Unicode Security in Unicode TR#39 http://www.unicode.org/draft/reports/tr39/tr39.html. TR39 addresses the appropriate use of Unicode characters and homographs, but this is at best a mitigation of the more general security concerns of identifier strings. Phishing and spoofing would still happen even in plain ASCII.

Hope this was helpful, or at least interesting,

Shawn

Writing "fields" of data to an encoded file.

shawnste — Tue, 02 Jun 2009 03:38:00 GMT

The moral here is "Use Unicode," so you can skip the details below if you want :)

A common problem when storing string data in various fields is how to encode it. Obviously you can store the Unicode as Unicode, which is a good choice for an XML file or text file. However, sometimes data gets mixed with other non-string data or stored in a record, like a database record. There are several ways to do that, but some common formats are delimited fields, fixed width fields, counted fields. I'm going to ignore more robust protocols like XML for this problem.

A delimited field would be a character between fields that indicated that one field ended an another started. Common delimiters are null (0), comma, and tab. Using delimited fields, a list of names would look something like "Joe,Mary,Sally,Fred".

A fixed width field would be a field of a known size regardless of the input data size. Generally data that is too short is padded with a space or null, and data that is too long is clipped. If our "names" field was of fixed size four, then the previous list could look something like "Joe_MarySallFred". Note the _ to pad the 3 character name, that Sally is clipped, and that the other names are "run together".

A counted field would indicate the field size for each piece of data before outputting the data. The advantage is that it doesn't have the size restriction/clipping of fixed width fields, nor does it have to waste space with unnecessary padding. (It could still be clipped for large strings as the count is likely restricted so some # of bits). Similarly delimiters aren't a problem. Generally the count is binary, but I'll show an example using numbers "3Joe4Mary5Sally4Fred"

A somewhat obvious way to store and read Unicode char or Unicode string data in the above formats is to write it in Unicode. Counted fields can just count the Unicode code points to be read in. Fixed width fields can similarly check for the space available and use Unicode character counts. Delimited fields can also use Unicode.

When the desired output isn't Unicode (UTF-16) however, then you start running into some interesting problems. Encodings (code pages) don't have a 1:1 relationship with UTF-16 code points, so you have to be careful. Additionally some encodings shift modes and maintain state through shift or escape sequences.

For all of the fixed, counted, delimited techniques shift states cause an additional problem in that either the writer has to terminate the sequence, or persist the state until the next field. Consider 2 fields where field 1 has some ASCII data that looks like "Joe" followed by shift sequence, then a Japanese character, and field 2 has "Kelly" in what looks like ASCII. If the decoder retains the state between reading the 2 fields, it may accidentally read in "Kelly" as Japanese and presumably corrupt the output. Alternatively if "Kelly" was really intended to read in "japanese" mode, then any application starting to read at field 2 gets confused since it didn't see the shift at the end of field 1.

For that reason I like to make sure the fields are "complete", flushing the encoder at the end of each field (this is different than writing a pure-text document like XML). So then field 1 above would have a shift-back-to-ASCII sequence at the end.

For fixed fields this could introduce another problem because the shift-back-to-ASCII sequence may exceed the allowed field size. In that case the string would have to be made smaller before encoding to allow enough room for flushing.

For delimited fields there's an additional problem in that the delimiter could accidentally look like part of an encoded sequence. Delimiters should only be tested on the decoded data.

For counted fields you start having trouble if the count isn't in encoded bytes. If you counted the Unicode code points, then encode those code points, you don't know how many bytes to read back in when decoding. It isn't possible to "just guess" when to stop reading data because there may or may not be some state changing data that you are expected to either ignore or read. For example "Joe++" where ++ is a Japanese character could look like:

4<shift-to-ascii>Joe<shift-to-Japanese><+><+>, or
4<shift-to-ascii>Joe<shift-to-Japanese><+><+><shift-to-ascii>, or
4<shift-to-ascii>Joe<shift-to-Japanese><+><+><shift-to-mode-q><shift-to-mode-z><shift-to-mode-x>

where "4" represents the count, <+> represents the encoded character, and <shift...> indicates some sort of state change that doesn't cause output directly by itself.

Since the application doesn't know whether to expect the trailing <shift> sequence(s), it may not read enough data, and then may try to use <shift-to-ascii> as the count of the next field. Similarly if it does see a <shift-to-ascii> and tries to read it in, then maybe it'll be confused if that was actually the count of the next field that just happened to look like a mode change.

So the moral is: Use UTF-16 because that's what the strings look like so they're less likely to get shifty about their sizes.

Use Unicode. Either UTF-16, or maybe use UTF-8, though it still can change size and you have to be careful, but at least each code point represents a Unicode code point.
If you must count, try to count the actual encoded data size, not the unencoded form since that'll be confusing when decoding.
Be good and flush your encoder if you must encode, so that the state gets back into a known state (usually ASCII) and then the decoding application doesn't get confused if they don't reset their decoder.
Make sure you say which encoding you used.

Of course you may be talking to a GPS or something where you don't get to define the standard. In that case you can just watch out for these caveats. Should you be designing such a protocol however, make sure to use Unicode. If that cannot happen, at least make sure to pay attention to the impact of encoding and decoding the data when the protocol's used.

-Shawn

Don't use MB_COMPOSITE, MB_PRECOMPOSED or WC_COMPOSITECHECK

shawnste — Thu, 07 May 2009 05:39:00 GMT

This pretty much demonstrates another reason to Use Unicode, but if you do need to use some non-Unicode encoding until you can convert to Unicode, please don't use these flags.

MultiByteToWideChar() and WideCharToMultiByte() provide some interesting sounding flags that are actually useless, slow, badly broken, or far worse. All of these flags would be expected to behave like Unicode Normalization, so you should instead use NormalizeString() to handle the desired behavior, either Form C for composed strings or Form D for decomposed strings.

MB_PRECOMPOSED is the simplest to address: Basically this flag doesn't really do anything. Nominally it would put data into something like Normalization Form C, however most code pages are already in a composed form, so there's little real impact. Just to make sure, the flag's ignored internally :)

MB_COMPOSITE is my most hated of these flags. First of all, it nominally pretends to put the data into something like Normalization Form D, decomposed into a base character and combining characters. To me that's the opposite of "Composite". Indeed, I've seen numerous code examples that seem to be passing MB_COMPOSITE expecting Form C data, and pretty much zero examples expecting Form D data. Windows leans towards Form C internally (though you may use Form D or mixed data), so this flag probably isn't that helpful. If you really want to decompose your data, then use NormalizeString with Form D instead of this flag.

MB_COMPOSITE also is very slow because it does a lookup in some data tables. NormalizeString with Form D is probably faster.

MB_COMPOSITE also has some horrible behavior for many code points:

Several code points will not round trip if this flag is set, even if WC_COMPOSITECHECK is used when converting back to the code page.
Additionally its data tables are incomplete and inconsistent with the normalization
Worse, some characters are decomposed into nonsensical sequences.
Lastly some sequences decompose to strange choices, breaking some text. Japanese is particularly impacted.

WC_COMPOSITECHECK basically has all of the problems of MB_COMPOSITE (its used in the other direction). Its name isn't as annoying to me though. Nominally WC_COMPOSITECHECK puts the data into Normalization Form C before encoding. Since most code pages are in a composed form Normalization Form C isn't a bad idea, however please use NormalizeString with Form C instead of this flag.

WC_COMPOSITECHECK is also very slow because of the way it does lookup. NormalizeString with Form C is probably faster.

WC_COMPOSITECHECK also has horrible behavior for many code points:

It will convert sensible sequences into a form that, when round tripped by MB_COMPOSITE will end up in nonsensical forms.
Sequences of 3 code points created by MB_COMPOSITE aren't correctly decoded by WC_COMPOSITECHECK back into their single code point form, resulting in extra ? when round tripping data.
Several sequences map to a single code point, which MB_COMPOSITE will map back to a single form, so they won't round trip. If you really need similar behavior try Normalization Form C, or KC if you really need the multiple mappings. KC causes data to not round trip, so it might not be appropriate for all applications. (Of course converting to the code page will also likely cause data to be lost so that may not matter so much).
Again some sequences are composed in a strange form based on appearance rather than linguistics. This could cause some unexpected behavior.
Some scripts, like Japanese, are particularly impacted.

Hopefully I've terrified you and you'll stop using these flags, perhaps using NormalizeString() if you really need similar behavior. Most applications don't even really need that though. Of course you always have the option of Using Unicode!

'til next time,
Shawn

Front page uses windows-1252, shouldn't it be iso-8859-1?

shawnste — Thu, 13 Nov 2008 23:41:00 GMT

I received this question:

I use Frontpage for my webpage design and FP automatically inserts the meta tag "<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">".

Should I have reference to ISO-8859-1 ?

I'm not a front page expert, and I can't answer all questions like this, however this is an common confusion. Windows-1252 is very similar to ISO-8859-1, but they aren't identical. Web sites and browsers have historically often treated these as equivilent, but they aren't, which is a great reason to use unicode for your encoding. (No, I don't know how to make front page use UTF-8, but that'd be the best solution). Looking on search.live.com (of course) for iso-8859-1 and windows-1252 will find some discussion of the differences. Wikipedia has some articles (they change so I won't quote them directly, but their encoding related articles are usually informative and often accurate.)

Where to Look Up Information About Microsoft Code Pages?

shawnste — Wed, 04 Jun 2008 20:12:00 GMT

First of all, remember to Use Unicode when practical :) Sometimes older applications don't allow Unicode, although they usually then don't allow Microsoft code pages as well (usually being ASCII or Latin-1, which are different).

But when you do have a question about how Microsoft's "ANSI" (They're not really ANSI) code pages behave, there are a few places you can look. Unicode has a mapping on their server at http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/ of the Microsoft "ANSI" code pages. They're also mentioned on the Microsoft globaldev http://www.microsoft.com/globaldev/reference/cphome.mspx site, and in Appendix H of Developing International Software.

Unicode use on the web

shawnste — Mon, 05 May 2008 22:57:00 GMT

Google posted a blog about unicode use on the web http://googleblog.blogspot.com/2008/05/moving-to-unicode-51.html They also announced that they now support Unicode 5.1, which is probably a good thing, but I found the graph most interesting http://bp1.blogger.com/_Ap14FtNN91w/SBzrtHJfLnI/AAAAAAAAA5U/TV7_g2_sWq0/s1600-h/Unicode2.gif.

The graph shows UTF-8 as now being the most prevalent encoding on the web, with a steady decline of ASCII and declining trends for other common encodings. That's got to be a good thing for character portability between machines on the web. Lack of proper declaration and use of encodings is one of the biggest problems with interoperability on the web, its nice to see Unicode gaining ground. The rate of growth is also really good, a nice strong curve there at the end :)

Server 2008 U+FFFD behavior for unknown or illegal UTF-8 sequences.

shawnste — Mon, 10 Mar 2008 21:24:00 GMT

In my post Change to Unicode Encoding for Unicode 5.0 conformance I mentioned that the behavior of illegal characters has changed for Unicode 5 conformance in Windows Vista / .Net 2.0+. Those changes have also been inherited by Server 2008.

Also check out my collection of code page related articles at Code Pages, Unicode & Encodings

Code pages and security issues

shawnste — Fri, 18 Jan 2008 03:34:00 GMT

One of the reasons I always suggest "Use Unicode!" is that there are security problems converting between code pages.

One of the reasons I always suggest "Use Unicode!" is that there are security problems converting between code pages. In short if data is going to be converted between code pages after some sort of security validation is done, then that validation could be negated. This is true of lots of data transformations, but it seems to surprise people a lot when applied to code page transformations.

There are lots of reasons for this, but some are:

Transformations can have "best-fit" mappings. For example, if I test for "C:\windows" in some form, but a tranformation maps the fullwidth compatibility characters (~U+FF00, such as ff43 (ｃ)), then the security check gets invalidated.
Similar things happen with different versions of code pages. For example, ASCII is sometimes handled by "dropping" the high bit so 0xc3 ends up being 0x43. Alternatively sometimes applications just "pretend" that it was really ANSI and map it to whatever code page the user is using.
Code pages aren't always tagged correctly, so an ANSI validation using a default system code page on a server may yield different results than the same code with a different default system code page on a client.
Some code pages also have difference between systems. Windows itself has slightly different behavior between MLang (often used by IE) and MultiByteToWideChar().
Different systems handle unexpected, unassigned or illegal code points in different manners. Sometimes that means ? or the equivilent. Sometimes gibberish, sometimes dropped data (so then your C:\windows test on C?:?\?W?i?n?d?o?w?s doesn't work if all the ? disappear).
Sometimes behavior changes, such as Change to Unicode Encoding for Unicode 5.0 conformance . In this case the change is to the Unicode parsing itself, but still any security test should be done after reading the input data.
Escape sequences might modify data in some environments that provide escaping mechanisms for characters that a code page doesn't support.

A related problem is the IDN and code page parsing that browsers sometimes do. & named and numeric entities in HTML can end up with a different appearance. % escaping is common in URLs, and IDN xn-- encoding happens in domain names. An application may decode these, even at unexpected times, and cause problems if the data was assumed to be in a different state before the decoding.

So the moral is: Do any security tests after any conversions have been done. If you have to retransmit the data, try to use an encoding like Unicode that has fewer edge case behaviors that could trip you up. If possible, revalidate the data after the transmission if it has to be decoded.

Michael has a blog about converting apps from ANSI to Unicode

shawnste — Thu, 22 Nov 2007 00:18:00 GMT

Lots of apps are now Unicode, but some need to make the shift from ANSI (like Japanese shift-jis) to Unicode. Michael has a series of blog posts about a project conversion. http://blogs.msdn.com/michkap/archive/2007/01/05/1413001.aspx

I recently had a customer question about removing a shift_jis dependency and moving to Unicode, so I thought I'd blog about it, but I've been busy, so in the meantime I thought maybe Michael's blog would help :0)

Are we going to update or maintain the best fit &/or code page mappings?

shawnste — Mon, 24 Sep 2007 20:30:00 GMT

People wonder if we're going to update our best fit code page mappings, or even our code page mappings. The answer is no. Changing character mappings causes difficulties for applications and our experience has been that doing so breaks as much as it "fixes". We'd prefer applications move to Unicode, then you don't have to worry about best-fit, or if a character is supported.

Best fit behavior is the behavior of some code pages to map unknown unicode characters to a character that someone thought was similar that the code page supported. Examples would be mapping ｋ(U+FF4B, full width k) to k, or ĩ (U+0-129 latin letter small i with tilde) to i,or ∞ (U+221e, infinity symbol) to 8. Some of these seem reasonable, however we aren’t consistent in our mappings, most break the meaning, and some mappings (∞->8) changes the meaning completely.

The best fit mappings were created “a long time ago”, contained “omissions”, and haven’t been updated to include new Unicode characters. “Newer” code pages don’t necessarily include the same best fit mappings, and, by now, the mappings are fairly inconsistent and incomplete. So we don’t recommend that the mappings be used, and we don’t intend to change or “fix” the best fit behaviors.

We also don’t like to change other code page data either. “unassigned” code points can have arbitrary behavior or map to Unicode PUA code points. Some applications use those code points (perhaps unwisely) as formatting codes or to cause special behavior. Adding a mapping could break such an application. Other applications or systems may provide a glyph for an unassigned code point that round trips, however that might not be the designed intent, and changing the code point behavior could break those applications or fonts.

Code page standards are also sometimes extended, modified, or corrected. Changing the behavior however impacts all applications using that behavior and our experience is that such changes across the installed windows code base causes as much trouble as it solves.

So we like to keep the code page mappings stable. My recommendations for code page use are:

Use Unicode unless explicitly required for some standard or protocol (and try to upgrade the standards or protocols to allow Unicode).
If you can’t use Unicode, explicitly specify the mapping that is used. (Some applications or standards presume whatever the OS uses, ie: windows ANSI code page, which causes serious interoperability problems.)
Avoid best fit mappings. At best they cause spelling errors or offend customers. At worst they can cause security problems.
Avoid unassigned code points, their behavior is undefined and could cause difficulty if a different machines or software have a different interpretation.
Use care when using the Unicode private use area (PUA). Its use is private. If data is persisted in the PUA, then there is a risk that future versions or other machines may not read the data correctly. Eventually migration of data between different PUA mappings may become necessary, and migrating such data is rarely trivial. The Hong Kong HKSCS mappings are an example of such a difficulty.
Don’t rely on illegal or undefined code page behavior. Illegal sequences might change between versions or software. Shift modes that aren’t implemented could be implemented on other machines, etc.
Don't presume that illegal or undefined code page behavior will remain stable.
Don’t pretend binary data is text in some code page (or Unicode). Variations in code page mappings could then prevent the data from round tripping, particularly if the binary data ends up in undefined or illegal code point behavior.

Hope that helps, more posts about common code page concerns are at http://blogs.msdn.com/shawnste/pages/code-pages-unicode-encodings.aspx

Shawn

UTF-16, UTF-8 & UTF-32 update to conform with Unicode 5.0's security concerns.

shawnste — Mon, 23 Jul 2007 19:07:00 GMT

My post Change to Unicode Encoding for Unicode 5.0 conformance now applies to .Net 2.0 with MS07-040 applied. Updates include a list of known issues, please see the list of known issues for MS07-040 described in KB 931212 for more information. KB 940521 describes this behavior in pandrticular. This fix reduces the chance of spoofing similar strings. Unicode 5.0 specifies this change due to security concerns regarding spoofing.

As mentioned in the KB:

Before this change, invalid characters in the middle of text strings would only be silently removed. For example, the string "Ad\xD800min\xDC00istrator" would change to "Administrator" as the Unicode characters U+D800 and U+DC00 are invalid . This could cause a security problem for some programs. After you install the security update MS07-040, this string would now become "Ad\xFFFDmin\xFFFDistrator", and decode to "Ad�min�istrator" where the � is the Unicode replacement character.

The first time we introduced this behavior was in Vista, and since then I've received several reports of issues with the new behavior. In nearly all of those cases there were usually some flawed assumptions contributing to the problems. Some examples were:

Programs trying to convert byte[] arrays to Unicode (see Avoid treating binary data as a string) and then having problems when the data didn't round trip. Note that prior to this change the data didn't round trip either, data was lost, but after the change it is more obvious since the FFFD's are present (which is the point of the security aspect of the change by the Unicode consortium).
Doing something like that, then trying to make a hash of the resulting value. After the update the hash doesn't match. Note that even prior to the update a very large number of values have the same hash, so this was not nearly as secure as the application had hoped.
Some applications made oopses with the behavior of Unicode, accidentally decoding extra byte(s) instead of pairs causing illegal UTF-16 or UTF-8. Those were ignored and the app worked despite the bug, but the update prevents the error from working.

Note that before the update .Net 2.0 on Vista and .Net 2.0 RTM had different Unicode decoding behavior. With the update applied they have the same behavior.

Hope this is helpful,

Shawn

I see my favorite Ansi function has the behavior I want.

shawnste — Fri, 15 Jun 2007 20:04:00 GMT

Occasionally I am asked about the A version of a W function. Ie: GetLocaleInfoA does something that appears more convenient to some user than GetLocaleInfoW. The implied thought is that maybe they should just use the A version.

For the most part our A functions are just wrappers for the W functions, so any perceived benefit is probably not real. Additionally since the A function is just a wrapper function, under the hood we have to convert any input and output strings to and from Unicode. That'll probably make the call take several times longer if nothing else.

For inputs the A version doesn't really restrict the code to particular Unicode sets. Sometimes its perceived that some Unicode character(s) are undesirable in the input stream. In this case either the application already restricts the input to this Unicode subset, in which case there's no difference in the supported character set for Unicode; or the application passes unknown data, relying on the Ansi to Unicode conversion to filter unwanted data. Of course then the unwanted data is just mangled, converted to ? or whatnot. So the unwanted data isn't really stripped, and, even worse, the unwanted data can be corrupted in a manner that causes a security hole. For example, I've seen a password hashing algorithm that then converted the hash to code page 1252. Quite often this caused a bunch of ??? in the resulting hash. Since many combinations cause the ??? the password hash would match a very large number of inputs, pretty much defeating whatever security was provided originally.

For outputs the A version also doesn't prevent any Unicode code points from being read, they're just converted to junk like ? when the call returns. So then the results are restricted to a subset of Unicode, but the restriction is done in a fairly useless manner. I've seen configuration values being stored like this (imagine a user name), and then they pretty much just end up in ???? when read back.

So, in short, use Unicode! :)

Why can't we strip the diacritics?

shawnste — Sat, 09 Jun 2007 05:20:00 GMT

We have some "best-fit" behavior which we generally consider to be "bad". Any loss of data is generally a bad thing, so we recommend storing data in Unicode (so you don't lose anything). Assuming you can't use Unicode, why is it so bad to just make everything ASCII-like? Maybe you have a published house or direct marketing firm that can't handle Unicode, so you'll just get rid of those annoying decorations.

In American English the diacritics are effectively quaint decorations. Many people naïvely assume that when word auto-corrects naive to naïve that this is just a prettiness factor. When they resume spell checking their résumé the diacritics become more important. In English its fair to spell résumé as resume, but it seems cooler to add the accents. Since we stole (borrowed is more politically correct) the word from French, we have a french-like pronunciation of résumé, and aren't likely to confuse it with resume.

In most other languages diacritics aren't optional. You wouldn't exchange a z with an s in english just because they look similar. "A real singer" is a lot different than "a real zinger".

Recently I encountered the the following example, a user wanted to get around those pesky diacritics by mapping to ASCII.

The suggested input was:
último año de carrera

The desired output was:
ultimo ano de carrera

My Spanish is nearly non-existent, however word's spell checker tells me these are all legitimate Spanish words, even without the accents. The meaning goes from something like "the last year of the race" to "I completed the anus of the race."

Now imagine that you're trying to reach a new market and you do that to your customer's names or potential customer's names, how long will they remain your customer?

- Shawn

Encoder/Decoder Encoding fallbacks fail after 2GB of data has been converted

shawnste — Thu, 07 Jun 2007 23:19:00 GMT

We have an unfortunate bug in .Net v2.0+ that causes encoding or decoding of more than 2GB of data to fail. That's a lot of data, but it still shouldn't do that. This is a problem with our built in fallbacks.

Ironically, if you encounter bad bytes then the bug is reset and you're "good" for another 2GB. This bug happens to most of our code pages for valid data, but some optimizations make it unlikely to happen in Unicode, ASCII & Latin-1. There are some workarounds. Some of these don't work if you're insulated from the decoder/encoder (like using a StreamWriter):

Change the encoder and decoder fallbacks to custom fallbacks, or use the built-in EncoderExceptionFallback. If you have known-good data, the ExceptionFallback would be a good choice.
Use UTF-8 or UTF-16. I think this nearly completely solves the problem. At the minimum it extends the data by enough magnitudes that your computer would probably die of hardware failure before you hit the bug.
Unconvertible data resets the bug, so you have another 2GB before it'll die. You may be able to occasionally introduce an unconvertible code point (like U+FFFD).
This only happens when the encoder/decoder fallback buffers aren't reset. Using the Encoding.GetBytes/GetChars won't fail unless you try a string longer than 2GB. If you are using short text segments that don't need the Encoder or Decoder state this would be a good state. For example, if you're piping a bunch of messages to the console, you might consider just sending one line at a time using the Encoding class.
Getting a new Encoder or Decoder object when possible will give you a fresh start. For example if you process a bunch of smaller documents you might change encoders/decoders between documents, or between records or whatever.

Hope that helps,

Shawn