Sorting it all Out

Early last year, Raymond Chen talked about how Char.IsDigit matches more than just 0 through 9 and later last year I talked about Crossing the DIGITal divide. But in both cases the conversation is limited to digits, and not the wide world of numbers which includes a lot more than just different ways of saying 0123456789. 

The distinction between digits and numbers in Unicode is an important one, since the formatting and parsing of numeric values is highly dependent on whether a number acts like the ASCII digits 0 - 9 or not.

Now the bulk of the modern number systems use the same Arabic-Indic system conventions to which software developers are accustomed, but others do exist, some of which are still see use today.

As an example people can relate to, most of us are aware of the Roman numeral system where there is no Zero and you sometimes have to use a lot of addition in subtraction in a deterministic manner (such that any time a smaller number comes before a larger one, the smaller one is subtracted; otherwise if they are the same value or the larger one comes first, it is added). Thus Ⅰ is one, Ⅲ is three, Ⅳ is 4, Ⅴ is 5, and so on. Although it is not used too much, it is still commonly seen in the credits of movies and television shows for the copyright date (e.g. MCMLXXXIX for 1989). Many people who are not used to Roman numerals breathed a sigh of relief at the year 2000 since MM is so much easier to read....

It is of note that the Roman Numerals are encoded in Unicode even though they can all be represented as existing letters. The primary reason for this is that there are character properties associated with each encoded character, and these properties are used by many implementations of Unicode to get actual work done. Therefore, the letter V (U+0056, LATIN CAPITAL LETTER V) has a General Category of Lu (Letter, Uppercase) while Ⅴ(U+2164, ROMAN NUMBERAL FIVE) has a general category of Nl (Letter, Number).

And yes, even that claim falls apart a little since the hexidecimal digits ABCDEF are not separately encoded for reasons of backwards compatibility with decades of existing practice on computers which is not the case with Roman numerals. Even the argument for having encoded the Roman numerals is a little specious since for the most part they have not been encoded and when they are the style never seems to be consistent typographically. Though YMMV since you may have better fonts than I do! Try "ⅯⅭⅯⅬⅩⅩⅩⅨ" for the test....

All of this goes to show that Unicode is a very complex standard. In the end, Unicode can always do what it needs to do without fear of the occasional contradiction, since there will always be some precedent with which to be consistent. :-)

Ethiopic numbers are based on a different alternative system, one that can really wreak havoc with a formatting/parsing architecture like that in Windows or the .NET Framework if you try to bring Ethiopic data in without writing code do the work (just like with Roman numerals). I'll talk about Ethiopic numbers another time....

Yet another system, the one I will talk about here, is that of Tamil numerals. It is an additive and positional system (unlike Roman numerals, there is no subtraction involved) that has no zero but includes characters for 10, 100, and 1000.

In the traditional system the number 3,782 would be represented as ௩௲௭௱௮௰௨ (literally Three-Thousand(s)-Seven-Hundread(s)-Eight-Ten(s)-Two, or மூன்று-ஆயிரத்து-எழு-நூற்று-எண்-பத்து-இரண்டு in Tamil).

At least since the early 1800s, however, usage of the Tamil numerals as digits has been more and more common. Thus the number 3,782 would often be represented as ௩௭௮௨ (literally 3782).

The following table gives a bunch of different numbers and how they are represented in both the older, more traditional style and in the "modern" style where they act as digits. Note that the table is treating U+0eb6 as TAMIL DIGIT ZERO even though it is not being added to Unicode until version 4.1. Up until now the ASCII DIGIT ZERO was used as needed, as I do in the table below for display purposes, and if you want to represent these numbers before Unicode 4.1 is released you should likely use U+0030 (DIGIT ZERO). The modern Tamil column using the LOCALE_SGROUPING setting of Tamil....

1 - a.k.a. Lakh
2 - a.k.a. 10 Lakhs
3 - a.k.a. crore
4 - a.k.a. 10 crore
5 - a.k.a. 100 crore
6 - a.k.a. thousand crore
7 - a.k.a. 10 thousand crore
8 - a.k.a. lakh crore
9 - a.k.a. crore crore

Some examples of both types of usage:

• Modern practice, using Tamil digits for chapter numbers: mozi varalARu, by munucAmi varatarAcan, published by The South India Saiva Siddhanta Works Publishing Society, Tinnevelly, Limited, November 1954, p. 357-358 (page numbers from 14th Edition, December 1996).
• Traditional practice, using the older format (and the source for large parts of the table above!): iniya tamiz ilakkaNam by yokisri cuttAnan~ta pAratiyAr, published by Kavitha Publications, p. 201-204. (you can see the scanned source of some of it here).

Note that the traditional form is not currently handled by any code in either Windows or the .NET Framework, though it is sometimes seen in even modern contexts such as calendars. The system is not too complicated and figuring out the algorithm to parse or format with it seems like the sort of thing that would make an interesting Microsoft interview question. Though perhaps I will post some potential solutions another day....

Special thanks to Sivaraj Doddannan, Dr. N. Ganesan, and Working Group 02 of INFITT (of which they are both members) for helping to dig up the excellent resources for Tamil numbers. INFITT (International Forum for Information Technology in Tamil) is a liaison member of Unicode and has been instrumental in providing character addition and usage reports to help finish up the Tamil block in Unicode.

This post brought to you by "௧௨௩௪௫௬௭௮௯" (U+0be7 - U+0bef, a.k.a. TAMIL DIGIT ONE - TAMIL DIGIT NINE)
and they all welcome their new compadre U+0be6, which is coming soon to a Unicode near you!