Putting your ducks in a row

Collation fascinates me.

It has fascinated me for most of the last eight years, since the first time I saw Appendix D of the first edition of Developing International Software for Windows 95 and Window NT. Like most people, I knew that some languages had different letters but it had just never occurred to me that letters would ever be ordered differently.

What I find most fascinating is that by and large people understand the order well enough to make use of it to retrieve information or to recognize when it is out of order. Yet they usually cannot clearly articulate the rules even when they have a clear subconscious knowledge of it.

Like all good internationalization features, people really only notice them if they do not meet expectations. This is something else that I find fascinating because I have never felt like it was all about me, even if I am doing cool stuff. It is about the cool stuff.

There reason that languages have ordering is obvious -- people need to be able to find information. How else can they find words in a dictionary or names in a phone book, if there is not a deterministic ordering that they are expecting?

The principles used to create the orderings vary, depending on the language and to some extent the script. I will give some examples here....

• Some ideographic languages (such as Chinese and Japanese) use a stroke based ordering. This can either be a simple count of the total number of strokes used in the ideoraph or it can be a radical/stroke based ordering where the basic radicals all have a specific order and then all of the strokes outside of the radical are then counted. For any one ideograph there can be only one count although the number of strokes counted may vary depending on whether the unified ideograph is Hanja (Korean), Kanji (Japanese), or Han (Chinese). Since there are generally under 50 strokes on even the most complex ideographs yet there are at minimum tens of thouands of them, there are many duplicates for each count and thus there must be a seconary method describing how to break those ties. Examples of tie breaking methods include order in a National standard, order in Unicode, or the order in which the strokes are ordered according to ideal construction.
• Some ideographic languages (such as Chinese and Japanese and Korean) use a pronunciation based ordering. Like with the stroke based orderings there sill be many duplicates, but unlike the stroke based ordering a single ideograph can have multiple pronunciations. This problem is eased in some cases by the fact that a dictionary has a clear interest in placing a word in multiple positions if there are multiple pronunciations with different meanings and the order of the dictionary is based on pronunciation. Beyond that, computers generally solve the problem either by allowing the user to strore the pronunciation so they can easily find an entry later or by using the most poopular/common/generic pronunciation.
• Some languages base the ordering on an older language (upon which the current language was based). The greek Alpha (Α) Beta(Β) Gamma (Γ) Delta (Δ) compared to the Cyrllic A (А) Be (Б) V (В) Ghe (Г) De (Д) compared to the Hebrew Alef (א) Bet (בּ) Vet (ב) Gimel (ג) Daled (ד) compared to the Armenian Ayb (Ա) Ben (Բ) Gim (Գ) Da (Դ) compared to the Georgian An (ა) Ban (ბ) Gan (გ) Don (დ) obviously have some sort of shared heritage of ordering. At this point we just do it this way because they did it that way before, and any attempt to ask about the order will usully elicit a blank stare.
• Some languages which use the same letters as another change the ordering based on phonemic principles (thus in Lithuanian the letter Y sorts after the letter I, rather than nearer to the end of the alphabet.
• Some languages which use similar letters as another put the letters after the letters to which they are similar. Thus A Macron (Ā) comes after A in most languages that use it, as does A Ring (Å).
• Some languages consider those similar letters to nevertheless be entirely separate letters which are put at the end of the Alphabet. Thus in Swedish A Ring (Å) is a unique letter that comes after Z, rather than a variation of the letter A.
• Some languages use a regular, repeated order combining consonant and vowel sounds. Thus in Japanese the pattern A (あ) I (い) U (う) E (え) O (お) is repeated with successive "rows" of the alphabet like Ka (か) Ki (き) Ku (く) Ke (け) Ko (こ) and Ga (が) Gi (ぎ) Gu (ぐ) Ge (げ) Go (ご). When one asks a native speaker about the AIUEO order the same blank stare can often be seen.

Other languages simply stick additional letters at the end or next ones that look similar. They may use points of articulation, or phonemic values.

More often than not, people know neither the whys nor the wherefores. But they know the right order when they see it....

In future posts I'll look into some of the more interesting trends I have noticed in some of the languages of East Asia, Southeast Asia, and South Asia.