I'm not a Klingon ( ) : System.Text

What is Title Case?

shawnste — Tue, 18 Aug 2009 20:30:00 GMT

Disclaimer: I'm not an English teacher (that's my mom), so I'm sure my description of title casing in English probably has exceptions/variations.

Title casing has an interesting history in computer programming. Programmers like to use CamelCase to make variable names more readable, and, particularly amongst developers native to some languages, there's an idea that title casing is interesting, such as in String.ToTitleCase(), and in Windows 7, LCMapString(LCMAP_TITLECASE). Most title casing algorythms are linguistically bad, even in English. For other languages it's worse.

ToTitleCase() takes a very simple approach to title casing. Maybe in the future it'll be smarter, but for now it just uppercases the first letter in a group of letters, and tries to pay attention to non-letters and word breaks. It also tries to keep acronyms all upper-case.

Even in English this is a simplistic approach. The title of this post is "What is Title Case?" Is is supposed to be lower case, but ToTitleCase() would mess it up. Additionally unexpected word breaks or punctuation could trick the algorithm. Even the acronym test isn't complete since it just expects all-upper case and sometimes acronyms keep the lower case of the full title. Also it messess up names like DiSilva or McConnell. Contractions can also be messed up.

Outside of English, ToTitleCase() rapidly gets silly. In English we capitalize everything except articles, short prepositions and some other short words. In German it's just like a normal sentence, with only nouns getting capitalized, so the English slightly over-eager capitilization behavior becomes very over-eager. Other languages also can have letters before the main word, eg: l'État, so the ToTitleCase rules can mess up those words as well.

And then there're scripts/languages that don't even have an upper/lower case distinction, so ToTitleCase gets pointless.

Anyway, use care when using ToTitleCase(). It might work in some cases, but don't expect it to work linguistically, particularly globally, particularly in non-English cases. Also maybe we'll get smarter and figure out a more correct way to do it in the future.

-Shawn

Writing "fields" of data to an encoded file.

shawnste — Tue, 02 Jun 2009 03:38:00 GMT

The moral here is "Use Unicode," so you can skip the details below if you want :)

A common problem when storing string data in various fields is how to encode it. Obviously you can store the Unicode as Unicode, which is a good choice for an XML file or text file. However, sometimes data gets mixed with other non-string data or stored in a record, like a database record. There are several ways to do that, but some common formats are delimited fields, fixed width fields, counted fields. I'm going to ignore more robust protocols like XML for this problem.

A delimited field would be a character between fields that indicated that one field ended an another started. Common delimiters are null (0), comma, and tab. Using delimited fields, a list of names would look something like "Joe,Mary,Sally,Fred".

A fixed width field would be a field of a known size regardless of the input data size. Generally data that is too short is padded with a space or null, and data that is too long is clipped. If our "names" field was of fixed size four, then the previous list could look something like "Joe_MarySallFred". Note the _ to pad the 3 character name, that Sally is clipped, and that the other names are "run together".

A counted field would indicate the field size for each piece of data before outputting the data. The advantage is that it doesn't have the size restriction/clipping of fixed width fields, nor does it have to waste space with unnecessary padding. (It could still be clipped for large strings as the count is likely restricted so some # of bits). Similarly delimiters aren't a problem. Generally the count is binary, but I'll show an example using numbers "3Joe4Mary5Sally4Fred"

A somewhat obvious way to store and read Unicode char or Unicode string data in the above formats is to write it in Unicode. Counted fields can just count the Unicode code points to be read in. Fixed width fields can similarly check for the space available and use Unicode character counts. Delimited fields can also use Unicode.

When the desired output isn't Unicode (UTF-16) however, then you start running into some interesting problems. Encodings (code pages) don't have a 1:1 relationship with UTF-16 code points, so you have to be careful. Additionally some encodings shift modes and maintain state through shift or escape sequences.

For all of the fixed, counted, delimited techniques shift states cause an additional problem in that either the writer has to terminate the sequence, or persist the state until the next field. Consider 2 fields where field 1 has some ASCII data that looks like "Joe" followed by shift sequence, then a Japanese character, and field 2 has "Kelly" in what looks like ASCII. If the decoder retains the state between reading the 2 fields, it may accidentally read in "Kelly" as Japanese and presumably corrupt the output. Alternatively if "Kelly" was really intended to read in "japanese" mode, then any application starting to read at field 2 gets confused since it didn't see the shift at the end of field 1.

For that reason I like to make sure the fields are "complete", flushing the encoder at the end of each field (this is different than writing a pure-text document like XML). So then field 1 above would have a shift-back-to-ASCII sequence at the end.

For fixed fields this could introduce another problem because the shift-back-to-ASCII sequence may exceed the allowed field size. In that case the string would have to be made smaller before encoding to allow enough room for flushing.

For delimited fields there's an additional problem in that the delimiter could accidentally look like part of an encoded sequence. Delimiters should only be tested on the decoded data.

For counted fields you start having trouble if the count isn't in encoded bytes. If you counted the Unicode code points, then encode those code points, you don't know how many bytes to read back in when decoding. It isn't possible to "just guess" when to stop reading data because there may or may not be some state changing data that you are expected to either ignore or read. For example "Joe++" where ++ is a Japanese character could look like:

4<shift-to-ascii>Joe<shift-to-Japanese><+><+>, or
4<shift-to-ascii>Joe<shift-to-Japanese><+><+><shift-to-ascii>, or
4<shift-to-ascii>Joe<shift-to-Japanese><+><+><shift-to-mode-q><shift-to-mode-z><shift-to-mode-x>

where "4" represents the count, <+> represents the encoded character, and <shift...> indicates some sort of state change that doesn't cause output directly by itself.

Since the application doesn't know whether to expect the trailing <shift> sequence(s), it may not read enough data, and then may try to use <shift-to-ascii> as the count of the next field. Similarly if it does see a <shift-to-ascii> and tries to read it in, then maybe it'll be confused if that was actually the count of the next field that just happened to look like a mode change.

So the moral is: Use UTF-16 because that's what the strings look like so they're less likely to get shifty about their sizes.

Use Unicode. Either UTF-16, or maybe use UTF-8, though it still can change size and you have to be careful, but at least each code point represents a Unicode code point.
If you must count, try to count the actual encoded data size, not the unencoded form since that'll be confusing when decoding.
Be good and flush your encoder if you must encode, so that the state gets back into a known state (usually ASCII) and then the decoding application doesn't get confused if they don't reset their decoder.
Make sure you say which encoding you used.

Of course you may be talking to a GPS or something where you don't get to define the standard. In that case you can just watch out for these caveats. Should you be designing such a protocol however, make sure to use Unicode. If that cannot happen, at least make sure to pay attention to the impact of encoding and decoding the data when the protocol's used.

-Shawn

Why can't we strip the diacritics?

shawnste — Sat, 09 Jun 2007 05:20:00 GMT

We have some "best-fit" behavior which we generally consider to be "bad". Any loss of data is generally a bad thing, so we recommend storing data in Unicode (so you don't lose anything). Assuming you can't use Unicode, why is it so bad to just make everything ASCII-like? Maybe you have a published house or direct marketing firm that can't handle Unicode, so you'll just get rid of those annoying decorations.

In American English the diacritics are effectively quaint decorations. Many people naïvely assume that when word auto-corrects naive to naïve that this is just a prettiness factor. When they resume spell checking their résumé the diacritics become more important. In English its fair to spell résumé as resume, but it seems cooler to add the accents. Since we stole (borrowed is more politically correct) the word from French, we have a french-like pronunciation of résumé, and aren't likely to confuse it with resume.

In most other languages diacritics aren't optional. You wouldn't exchange a z with an s in english just because they look similar. "A real singer" is a lot different than "a real zinger".

Recently I encountered the the following example, a user wanted to get around those pesky diacritics by mapping to ASCII.

The suggested input was:
último año de carrera

The desired output was:
ultimo ano de carrera

My Spanish is nearly non-existent, however word's spell checker tells me these are all legitimate Spanish words, even without the accents. The meaning goes from something like "the last year of the race" to "I completed the anus of the race."

Now imagine that you're trying to reach a new market and you do that to your customer's names or potential customer's names, how long will they remain your customer?

- Shawn

Encoder/Decoder Encoding fallbacks fail after 2GB of data has been converted

shawnste — Thu, 07 Jun 2007 23:19:00 GMT

We have an unfortunate bug in .Net v2.0+ that causes encoding or decoding of more than 2GB of data to fail. That's a lot of data, but it still shouldn't do that. This is a problem with our built in fallbacks.

Ironically, if you encounter bad bytes then the bug is reset and you're "good" for another 2GB. This bug happens to most of our code pages for valid data, but some optimizations make it unlikely to happen in Unicode, ASCII & Latin-1. There are some workarounds. Some of these don't work if you're insulated from the decoder/encoder (like using a StreamWriter):

Change the encoder and decoder fallbacks to custom fallbacks, or use the built-in EncoderExceptionFallback. If you have known-good data, the ExceptionFallback would be a good choice.
Use UTF-8 or UTF-16. I think this nearly completely solves the problem. At the minimum it extends the data by enough magnitudes that your computer would probably die of hardware failure before you hit the bug.
Unconvertible data resets the bug, so you have another 2GB before it'll die. You may be able to occasionally introduce an unconvertible code point (like U+FFFD).
This only happens when the encoder/decoder fallback buffers aren't reset. Using the Encoding.GetBytes/GetChars won't fail unless you try a string longer than 2GB. If you are using short text segments that don't need the Encoder or Decoder state this would be a good state. For example, if you're piping a bunch of messages to the console, you might consider just sending one line at a time using the Encoding class.
Getting a new Encoder or Decoder object when possible will give you a fresh start. For example if you process a bunch of smaller documents you might change encoders/decoders between documents, or between records or whatever.

Hope that helps,

Shawn

How do I get HKSCS 2004 characters from Big-5 in .Net?

shawnste — Thu, 03 May 2007 23:29:00 GMT

Well, that's pretty tricky. We provide the Microsoft Character Code Conversion Routines For HKSCS-2004 functions, but those are intended for use with unmanaged code.

The fundemental problem is that these "HKSCS" characters were in use prior to the assigment of a code point for them in Unicode. In order to support them, we mapped Big 5 / Code Page 950 HKSCS characters to the Unicode Private Use area. So now there is data with these code points in the PUA and in Big 5, AND at the Unicode 5.0 code points. The expectation is to use Unicode long term, so these functions were provided to help map old data to the new Unicode 5 code points.

Another way for a managed application to solve this problem would be to create your own Encoding and map the Big 5 code points to their new Unicode code points instead of the old code page 950 mappings. It is nearly impossible for Microsoft to provide a patch to do this because some users have data in the old PUA code space and their applications would break if the data was suddenly migrated to the assigned HKSCS code points without them opting in. Eventually "all" the interesting data should be migrated from the PUA code points to the Unicode HKSCS code points, but until then the problem remains.

The code samples and links from the "Microsoft Character Code Conversion Routines For HKSCS-2004" document would be a good starting spot to generate the necessary mappings to make an Encoding that moved code page 950 data to the new HKSCS code points.

Please avoid UTF-7

shawnste — Tue, 01 May 2007 21:13:00 GMT

UTF-7 inherently some of the security issues that concern people about encodings. For example, by shifting in & out of the base64 mode one can create multiple representations of the same string, enabling spoofing and other problems.

UTF-7 is primarily interesting for legacy mail and NNTP applications that don't properly handle native or MIME encoded UTF-8. The need for new content to be encoded in UTF-7 is very low. In particular UTF-7 should be avoided with any modern systems that are natively 8-bit. For example XML files don't inherently have any limitations that would force the need for UTF-7, so there should be no need for UTF-7 in XML files.

Of course with any general rule there may be some exceptions, but I'd encourage you to support UTF-8 or UTF-16 and only use UTF-7 if you run into some system that can't support an 8-bit encoding. If you run into such 7 bit limitations it should probably be a warning that some redesign might be necessary. For mail this is being considered by the IETF's eai working group at http://www.ietf.org/html.charters/eai-charter.html

Some Reasons to Make Your Application Unicode

shawnste — Tue, 20 Mar 2007 22:58:00 GMT

[Updated Mar 30 2007: Mike pointed out errors which I've corrected]

Many applications are "still" ANSI and can't handle Unicode. We (Microsoft) have even released non-Unicode applications reasonably recently. even though we should know better. In particular there are a bunch of good reasons to move your app to Unicode. I'm rushed so I'm only listing a few here.

We have been adding many new locales and keyboards. Many of those don't have code pages. These "Unicode Only" locales have to either pick a system code page that only marginally supports their language, if it supports it at all. In these cases your ANSI application will be completely unusable in these locales.
Data passed between ANSI systems is easy to misinterprete if the systems have different code pages. This leads to random data corruption, some of which isn't always recoverable.
ANSI apps don't support the full range of characters, so users with unique requirements may not be able to enter data completely or correctly.
Mixed language environments fail with ANSI only applications.
ANSI only bugs like Some Keyboards fail with ANSI applications on Windows Vista RTM won't impact your application

We have encountered numerous customer issues which could've been solved fairly trivially by using Unicode applications.

Some popular messenging applications are not Unicode, so users cannot always send or receive messages properly in multi-lingual environments. For people working in other countries this is a common case.
Most people have seen "gibberish" on web sites due to mistagged data. UTF-8 or UTF-16 would solve most of this confusion.
Many media tagging systems didn't originally specify an encoding for metadata, causing corrupted metadata when viewing on other machines. (This is also an example of data that can be very difficult to recover)
Data being sync'd to phones, etc. hasn't always worked if part of the chain is ANSI.
Wireless SSIDs (wireless WLAN names) don't specify code pages, so if you're in a foreign airport or other multicultural environment you might get gibberish for the names when trying to find a network to connect to.
Customer names have accents dropped or turned into ? when unexpected code points are expected. (for example, you have a web form that the user enters their own name correctly, but when a printer merges this for a magazine subscription or whatever the non-ANSI/non-ASCII data gets lost). Some users are very irritated when their name gets misprinted in this manner.

Most of these issue are easily solved by considering the encoding requirements and character repertoires, but it is often overlooked. US developers seem to be particularly susceptible to this design problem since ANSI is easy to use and their applications have a large US market even if they only use ANSI characters.

There are some occasions, primarily for backward compatibility or existing protocols that didn't plan for Unicode, where applications can't avoid ANSI. In those cases I suggest A) trying to get a plan to remove the back compat issue or fix the protocol so that this problem doesn't continue for decades, B) use Unicode in the meantime and only convert it to the ANSI code page when necessary, and C) tag the data with the appropriate code page when possible so the receiver has a hope of decoding it properly. It is best to avoid these situations though because they invariably have edge cases that are difficult to handle.

Use Unicode :)

A History of Code Pages or What Made Code Page XXXX (or many other computer things) The Way It Is?

shawnste — Tue, 13 Mar 2007 21:29:00 GMT

Disclaimer: This is mostly my conjecture, so I could be completely wrong about some of this, but it seems plausible to me. I’m aiming for the general concepts here, not to start a discussion about the specific details of the history of code pages.

Taking a snapshot of the current windows code pages (or any other code pages), one can wonder how some of these code pages ended up in their current state. We also wonder about other things such as peculiarities of a function call and other related behavior.

It is important to remember that modern computer systems evolved from earlier systems and “we”, as in the entire computer science community on the planet, have learned a lot since the beginnings of computer science. Rarely do we get the chance to start with “a clean slate” and redesign APIs or systems. Even when we do, we only have our best intentions and previous lessons to learn from, and sometimes those new designs prove to have weaknesses that weren’t originally seen.

In DOS days of PC history, “code pages” were the bytes used to directly print to the console. Apple, Commodore, IBM, and probably many others used bytes to map to a character on the console. (Before that there were the values that showed up on Teletypes or punch cards, but I’m kind of focusing on Windows history). The US and “western” cultures seem to have had a great influence on the development of early PCs, and the ASCII standard was very common. Many future behaviors were based on ASCII or similar work.

ASCII only specified 7 bits of information, but since PCs had 8 bits most manufacturers extended the code pages to provide additional glyphs, such as diacritics or additional scripts (besides latin). This provided the ability to represent many languages, but at a hidden cost of data portability. Since most data was confined to single companies and global exchange of data wasn’t a primary concern this wasn’t a big problem at first.

Additionally, since these bytes were used to render glyphs on the screen it seemed wasteful to ignore the non-printable control sequences from 01-1f, so smiley faces, hearts, spades and the like were added.

Users Want Their Glyphs:

As computing evolved users wanted more glyphs and several techniques evolved to solve that problem.

Font changing was used by some systems (and continue to be used in some cases). Early DOS PCs effectively changed the font used for the display when they changed the “OEM code page”. Once multiple font use became common, this technique evolved to allow multiple glyph sets to be displayed in a single application merely by changing the font. In modern systems Unicode provides a Private Use Area (PUA) for users to stick their custom glyphs, but font hacks continue to be used. The PUA solution doesn’t work on the console or for ANSI applications, so some groups have created font hacks that render the desired glyphs, yet their system uses a code page with different characters than those the font displays. The adoption of this technique ranges from users “playing” with the invented Klingon script to national “standards” attempting to make computers work for them where OEMs have been slow to create fonts or other solutions.
Switching fonts is effectively built-in to the ISCII standard. The idea is that escape sequences are used to select which font is to be used for the 8th bit character ranges. Originally this included the idea of simple transliteration (by merely changing the rendered font), but this doesn’t seem to be used much in practice. This technique sort of standardizes the use of the font changing technique. This is obviously an evolution beyond the early PCs that could only display a single font.

For CJK (Chinese, Japanese & Korean) scripts, 8 bit fonts aren’t enough. CJK code pages are usually still ASCII compatible, but they’ve evolved other techniques for rendering additional characters.

Double byte code pages have the idea that a specific range of bytes are “lead bytes” and are to be followed by a “trail byte”. Combined the lead & trail bytes provide many additional characters. CP 932, etc. are examples of this technique.
This idea was extended by GB18030 to provide additional lead bytes that indicate 4 byte sequences, allowing even more characters to be encoded.
Shifting code pages are similar to ISCII in that they select additional modes. They typically “start” in an ASCII-like code page, but particular escape sequences cause the following bytes to be interpreted according to other rules. Generally these provide additional two byte or single byte sequences. Note that the shift sequences are typically single byte, even when currently in a double byte mode.

Evolution of the Character Repertoire:

In addition to the evolution of techniques, the repertoire of supported characters has been evolving. Unfortunately the drivers of this process are rarely coordinated across the industry. As a need for a new character becomes apparent, organizations add it to the standards that they influence or control, but this doesn’t guarantee adoption across the industry, particularly if they don’t coordinate with other standards or organizations.

This repertoire evolution can cause the behavior of code pages to evolve as well. For example, the Euro was invented well after the creation of ASCII and many of the many other code pages. Obviously it was needed, so it was added to most code pages, squeezing into unused spaces where possible. For single byte code pages that could mean replacing a previously rarely used code point. Of course if a vendor used that rarely used code point for something special in their application, then this caused behavioral changes.

For other standards the repertoire evolution has meant evolving iterations of the standard. Several organizations add characters to their standards, but it can take a while for those to make it to the font vendor or other level necessary for complete support. Shifting standards can also change existing user data or private use behavior, so supporting new standards isn’t always a trivial undertaking.

Some character sets have been complicated by standards dependencies. For example if a desirable standard assigns a bunch of characters and users want Windows support, then Windows has to find space in Unicode since Windows is Unicode based. In the best case the desired characters are already assigned to Unicode so windows can “just” add font support (not necessarily trivial) and is good to go. Historically however, characters are usually created by some other authority and may take a while to get official Unicode support. In those cases, the characters can remain unsupported, or someone can add PUA characters to support them until Unicode supports them.

If PUA characters are used to temporarily support additional characters, then there are additional problems when they are added to Unicode since existing data will need to be migrated from the PUA to the actual Unicode code point. Migration may also be complicated by the fact that all users may not be able to upgrade at the same time.

Implementation Details:

Another problem impacting the way code pages behave is how (and when) they’ve been implemented. Occasionally standards have had errors that were corrected in later versions. Other times a platform vendor may have interpreted the behavior in an unexpected way. Sometimes a font vendor for a common font could make an error with a code point. Additionally users may commonly confuse a glyph with a similar glyph and abuse the existing standard.

All of these contribute to variations in the way code page data is handled. Once data is coded in a particular way, correcting the data may be complicated. It can be easy to identify an implementation bug and find the “correct” solution, but making the fix can break existing behavior or data portability.

Oddities:

For historical reasons there are also some oddities in encoded data. Remember that code points were often merely glyphs on the computer screen? And those glyphs depended on the rendering of that machine? Well DOS used the \ character to delimit folders on the file system. CJK users however wanted to be able to type their currency symbol on their machines. Since people don’t use \ very often, it got replaced with the appropriate currency symbol on Asian machines. Internally it was always 0x5C however, and the machine always used that byte value to delimit folders. The end result is a mess where 0x5c doesn’t convert to Unicode very well, where users have different file system delimiter characters, and where fonts end up hacked to render ¥ instead of \ if you have a certain system code page. This is obviously really undesirable, yet it is pretty obvious how this happened and pretty difficult to “fix” at this time.

Conclusion:

I find it helpful to remember this stuff when confronted with another code page oddity. One of my goals is to reduce any further complexity in this evolutionary tree of code pages. It is often “clear” what the desired or proper behavior should be when you consider only current standards or when you know the lessons we’ve always learned. With a living system of data it isn’t always possible to get from the current state to the perfect state in a simple manner without causing pain to some users. In that case I try to limit the long term pain, and reduce the problems to as few users as possible.

Similar examples exist for nearly all API sets and programming languages, OS’s and techniques. The global “we” of computer science has learned a lot and continues to learn a lot, but sometimes it’s helpful to remember how an API may have evolved when it doesn’t seem to be doing the most appropriate thing.

For code pages this is a good reason to use Unicode. Windows is natively Unicode and most other systems understand it. It is also reasonably unambiguous, although it does have its own evolutionary quirks. By focusing on a single encoding (Unicode), we can reduce the complexities cause by natural variations introduced as encodings evolve.

Reminder: This is mostly my conjecture and seems reasonable to me, although it might be wrong or lack specifics.

Expected names of Microsoft Windows "ANSI" Code Pages (Encodings)

shawnste — Tue, 07 Nov 2006 02:52:00 GMT

I was asked about our use of the windows "ansi" code page names, as used in things like MIME types, http content-type tags, etc. Each "code page" has a name that most accuratly round trips back to the same code page, which I've listed as the "preferred name" below. Additionally, when you ask for a code page matching a name, some code pages have several aliases that map to the identical behavior. (listed as "aliases" in the table below).

Note that there are quite a few inconsistencies and other odd behaviors. Some have names of windows-xxx, and others don't even recognize that form as an alias. Additionally some reference material I've seen refers to any microsoft code page in the windows-xxxx or CPxxx form, whether or not windows itself recognizes those names.

We have no intention of trying to create a more consistent naming scheme, we prefer that applications use Unicode.

So in practice, you should use the "preferred name" to identify data tagged by a particular code page, but if you are accepting input data, recognize that it may also use one of the listed aliases. Encoding.GetEncoding() should "do the right thing". You might want to look at my previous posts "Encoding.GetEncodings() has a couple "duplicate" names" and "What's my Encoding Called?".

Code Page 874 (ANSI/OEM - Thai)

Preferred Name:

windows-874

Aliases:

DOS-874

iso-8859-11

TIS-620

Code Page 932 (ANSI/OEM - Japanese Shift-JIS)

Preferred Name:

shift_jis

Aliases:

csShiftJIS

csWindows31J

ms_Kanji

shift-jis

sjis

x-ms-cp932

x-sjis

Code Page 936 (ANSI/OEM - Simplified Chinese GBK)

Preferred Name:

gb2312

Aliases:

chinese

CN-GB

csGB2312

csGB231280

csISO58GB231280

GB2312-80

GB231280

GBK

GB_2312-80

iso-ir-58

Code Page 949 (ANSI/OEM - Korean)

Preferred Name:

ks_c_5601-1987

Aliases:

csKSC56011987

iso-ir-149

korean

ks-c-5601

ks-c5601

KSC5601

KSC_5601

ks_c_5601

ks_c_5601-1989

ks_c_5601_1987

Code Page 950 (ANSI/OEM - Traditional Chinese Big5)

Preferred Name:

big5

Aliases:

Big5-HKSCS

cn-big5

csbig5

x-x-big5

Code Page 1250 (ANSI - Central Europe)

Preferred Name:

windows-1250

Aliases:

x-cp1250

Code Page 1251 (ANSI - Cyrillic)

Preferred Name:

windows-1251

Aliases:

x-cp1251

Code Page 1252 (ANSI - Latin I)

Preferred Name:

Windows-1252

Aliases:

x-ansi

Code Page 1253 (ANSI - Greek)

Preferred Name:

windows-1253

Code Page 1254 (ANSI - Turkish)

Preferred Name:

windows-1254

Code Page 1255 (ANSI - Hebrew)

Preferred Name:

windows-1255

Code Page 1256 (ANSI - Arabic)

Preferred Name:

windows-1256

Aliases:

cp1256

Code Page 1257 (ANSI - Baltic)

Preferred Name:

windows-1257

Code Page 1258 (ANSI/OEM - Viet Nam)

Preferred Name:

windows-1258

Example of overriding your own Encoding.

shawnste — Fri, 13 Oct 2006 02:00:00 GMT

Previously I wrote about the Best Way to Make Your Own Encoding, but didn't include an example, so today I'm including an example of a replacement Encoding. I also included an EncoderFallback example, which replaces unknown characters with numerical entity style replacements (〹).

This example isn't complete. If you need Encoder or Decoder functionality you'd have to override those as well. Also I didn't include a DecoderFallback example. From this example those should be reasonably straight forward. The biggest issues are that Encoders/Decoders maintain state such as lead bytes or high surrogates, so they may have data buffered from a previous conversion.

I included a simple Main that just converts some text to bytes and back. Its not very pretty, but it demonstrates that something did actually happen :) Be forwarned that I spent almost no time testing this sample, so caveat programmer! As always this sample is provided as-is with no warrenties or guarantees.

Hope you find this helpful,

Shawn

Best Way to Make Your Own Encoding

shawnste — Sat, 30 Sep 2006 01:06:00 GMT

Martin recently asked what the best way to roll his own encoding in .Net 2.0, in particular can you override Encoding/Encoder/Decoder, or should he write his own StreamWriter.

#1 is, of course, to use Unicode :), but apparently Martin doesn't have that option.

The answer is that you can write your own Encoding derived from the Encoding class and use it as any of the built-in Encodings. They'll be a little slower in some cases due to some shortcuts we take internally, but otherwise you should be able to use them everywhere you use a normal Encoding/Encoder/Decoder object.

There's an example of using the fallbacks at http://windowssdk.msdn.microsoft.com/en-us/library/tt6z1500.aspx, which isn't quite the same thing, but might help a little.

[updated 12 Oct 2006]
I've stuck an example of overriding encodings in this post: Example of overriding your own Encoding.

- Shawn

Encoding.GetEncodings() has a couple "duplicate" names

shawnste — Tue, 18 Jul 2006 20:41:00 GMT

The Microsoft.Net v2.0 Encoding.GetEncodings() method returns a complete list of supported encodings, uniquely distinguished by code page. Note that in general I consider the code page number to be a poor way to exchange code page information since its not a standard, however for now it does provide a unique ID for .Net Encodings. By Name there are some duplicates, although they have different DisplayNames.

Encodings 20932 and 51932 both return the Name "euc-jp", and indeed are identical code pages. If you ask for "euc-jp", the framework will return 51932, so if you want to remove one, I'd remove 20932 from any list you make.

UPDATE - 11/29/1006 (snow day in Redmond). Actually it was pointed out that 51932 doesn't work in native windows APIs, so you'd have to pick 20932 for native applications and 51932 for .Net applications (so that it would round trip in .Net).

50220 and 50222 also return "iso-2022-jp" for their Name. If you ask for "iso-2022-jp", you'll end up with 50220, so I'd remove 50222 from any list of encodings. Which one you should prefer depends on the preferred treatment of the half width katakana.

Unlike the euc-jp encodings, the 50220 and 50222 encodings are slightly different. When encoding, 50220 will convert half width katakana to full width and 50222 will use a shift-in/shift-out sequence to encode half width katakana. The DisplayName for 50222 is “Japanese (JIS-Allow 1 byte Kana - SO/SI)” to distinguish it from 50222 “Japanese (JIS)” even though they have the same iso-2022-jp Name.

What's with Encoding.GetMaxByteCount() and Encoding.GetMaxCharCount()? Part 2

shawnste — Thu, 22 Jun 2006 03:38:00 GMT

A little over a year ago I wrote What's with Encoding.GetMaxByteCount() and Encoding.GetMaxCharCount()? to address the question "Why does GetMaxCharCount(1) for my favorite Encoding return 2 instead of 1." (Short answer is that the Decoder/Encoder could have stored data from a previous call).

To follow up, what about the special case of zero? It seems that GetMaxByte/CharCount(0) should always be 0. The answer again is because of the encoder/decoder and the fallback.

Consider that a call to Decoder.GetChars() ends with a lead byte for UTF-8. The decoder is going to remember that lead byte, expecting the next call to GetChars() to contain the remaining byte(s) necessary to decode a complete UTF-8 sequence.

However if the next call passes in an empty input buffer, yet requests that the buffer get flushed, then the decoder's going to have to process that lonely lead byte anyway. This happens for example at the end of a sequence. In this case, the decoder's going to call the fallback for the lone lead byte, which by default for UTF-8 will now return a U+FFFD. So even with an empty input buffer, UTF-8 can return a character.

Similar cases happen with most other encodings, although there are a few cases where encodings don't have left over bytes when decoding.

Change to Unicode Encoding for Unicode 5.0 conformance

shawnste — Sat, 17 Jun 2006 01:07:00 GMT

The behavior for UTF8Encoding, UnicodeEncoding and UTF32Encoding has changed in Windows Vista to conform better to the Unicode 5.0 requirements for Unicode Encodings. [23 July 2007: Now this behavior has also been made to .Net 2.0 with MS07-040 update applied. See the list of known issues for MS07-040 described in KB 931212. KB 940521 describes this behavior in particular.]

In .Net Framework V2.0 RTM we chose to respect the Unicode 4.1 standard which disallowed passing illegal UTF code points by dropping any bad data that was encountered, considering that this behavior would have the minimal impact to existing applications.

Since the .Net Framework 2.0 was released, the latest Unicode 5.0 specification has become stricter. There was a concern that just ignoring invalid bytes could allow insecure hostile data because invalid characters would be dropped so and invalid string could become valid. The new requirement for Unicode 5.0 is that bad bytes cannot be dropped, so we are now replacing them with U+FFFD, the Unicode Replacement Character, in Windows Vista, and future versions of the .Net Framework, including the .Net Framework 2.0 on Vista, and .Net 2.0 with the MS07-040 update applied.

The new default behavior is equivalent to setting the replacement fallbacks to "\xFFFD” instead of the empty string. If applications prefer the old behavior, they can create their UTF8Encoding with an EncoderReplacementFallback("") and DecoderReplacementFallback(""), causing the fallbacks to drop the bad data.

Because of the +- and other oddities with "UTF-7" its generally considered insecure anyway for similar reasons and UTF-8 is generally preferred.

FWIW: My recommendation is that applications shouldn’t make trust decisions on encoded data, this goes for the other code page encodings as well as Unicode. Encoding and decoding data can cause it to change its form. (See Best Fit in WideCharToMultiByte and System.Text.Encoding Should be Avoided for one example). If your application needs to make sure that an input string doesn’t include C:\windows, it should do the validation after decoding the data. I’ll probably blog more about this later.

'til then,

Shawn

Best Fit in WideCharToMultiByte and System.Text.Encoding Should be Avoided

shawnste — Thu, 19 Jan 2006 23:16:00 GMT

Windows and the .Net Framework have the concept of "best-fit" behavior for code pages and encodings. Best fit can be interesting, but often its not a good idea. In WideCharToMultiByte() this behavior is controlled by a WC_NO_BEST_FIT_CHARS flag. In .Net you can use the EncoderFallback to control whether or not to get Best Fit behavior. Unfortunately in both cases best fit is the default behavior. In Microsoft .Net 2.0 best fit is also slower.

The underlying problem that best fit behavior tries to solve is "Gee, Unicode has about a gajillion more characters than 1252, how do we get them all in?". Unfortunately that's the problem, they won't all fit in. 1252 has 256 characters, the nearly 100,000 Unicode characters just won't fit. So what Best Fit tries to do is to cram as many characters as possible into the limited set of the code page by mapping them to things they might look like. So c with a dot above, ċ, U+010b is mapped to a plain old c with no dot. Japanese full-width forms are mapped to their half width forms, etc. There are lots of problems with this solution, and I'll mention some of them here:

The mappings are somewhat random and sometimes bizarre. The infinity symbol, ∞, U+221e, is mapped to 8. Sure it looks like a sideways 8, but its sideways, and its meaning is very different.
The mappings are somewhat random and inconsistent between code pages. In some code pages Japanese fullwidth forms are "best fit" to the non full-width form, in others they are not.
The best fit behavior has not been updated in years, so new code points aren't present, so c, ć U+0107 c with acute, ĉ U+0109 c with circumflex, ċ 0x010b c with dot above, č 0x010d c with caron and ｃ U+ff43 fullwidth c, are all mapped to c in code page 1252. However ƈ U+0188 c with hook, ɕ U+0255 c with curl, с U+0441 Cyrillic es, ḉ U+1e09 c with cedilla and acute above and others are not mapped and turn into ?. Also, ç U+00e7 c with cedilla doesn't change since it has its own character in 1252.
Many mappings lead to security holes. A common test for ., \ and other characters to prevent ..\ style attacks on paths fail if fullwidth forms are used and not tested for. Since fullwidth forms are often mapped, any English string, like a user name or password can also have multiple variations, leading to security holes. Even if fullwidth forms are considered other mappings with diacritics as mentioned in the previous bullet exist for common English characters.
Most of the best fit mappings in our tables were thought of by English speaking Americans and could be culturally inappropriate for other locales.
ü and u aren't the same character. Düssledorf has the alternate spelling Duessledorf, replacing the ü with ue, not u. In languages that use diacritics the pronunciation of the character changes. If you made mailing labels for your customers would you really want to change their name? Best case the spelling looks stupid and the customer thinks "gee, these guys have an old computer too". Worst case you turned their name to crap... literally. In that case ? would probably be better, at least your customer would probably understand it was a computer limitation [:)]
For typical English US spellings UTF-8 is exactly as space efficient as ASCII or 1252. So if you use UTF-8 you won't need best fit and it won't cost you anything. In .Net and Windows Vista UTF-8 is also much faster than 1252 or ASCII. Even in other languages UTF-8 is faster and for most languages it doesn't even create significantly larger file sizes. A small price to pay to insure that you don't corrupt your data.
Best fit doesn't even help some of the alternate English spellings, such as the ae, those just turn into ? anyway.
As I alluded to above, frankly our best fit mapping is pretty poor, even if that behavior is desirable. We're inconsistent with the behavior for different code pages, we haven't added new characters, and we've made some strange decisions.

Its also worth noting that there are a few rare cases where best fit can happen when decoding data with MultiByteToWideChar or Encoder.GetString or the Decoder class.

For both Windows and Microsoft .Net, the best plan is to use Unicode when possible, either UTF-8 or UTF-16 is usually a good choice. Sometimes its not possible, usually because of a protocol limitation. Often best fit behavior is a poor choice when hitting a protocol problem, since such protocols are usually explicit and such mappings could cause security holes or protocol violations. In those cases finding extensions or newer protocols that handle Unicode are good, but some, like e-mail headers [;)], we're stuck with.

In Windows you can disable the best fit behavior by using the WC_NO_BEST_FIT_CHARS flag. In the framework you can do so by changing the EncoderFallback and DecoderFallback. Encoder.GetEncoding(xxx, EncoderFallback.ReplacementFallback, DecoderFallback.ReplacementFallback) or ExceptionFallback are good choices. Note that in the .Net 2.0 there is no "best fit" fallback, except for an internal best fit fallback that is used by default, so once you change a class's EncoderFallback or DecoderFallback you cannot easily retrieve the best fit fallback.

If you are aware of the limitations of the fallbacks and want consistent behavior anyway, one option to consider is making your own fallback. I made a prototype fallback that uses Normalization to decompose a string to its component parts. This is particularly nifty because characters can be decomposed to their component parts. By doing this, things like the kPa symbol can change to k + P + a. It still doesn't work across all languages though since ü would still become a u instead of a ue in German. So even though this can be a fun experiment, it's still better to Use Unicode!