Sorting it all Out : Every Character Has a Story

Every character has a story #31: U+272f0 from CJK Extension B, an ideograph that proves that every rose has its thorn! (aka It wasn't my fault, but [from the Windows standpoint] it was because of me....)

Michael S. Kaplan — Mon, 03 Dec 2007 17:31:00 GMT

Yes, the end of the title is an allusion to a late 80s Poison power ballad based on a Bret Michaels love affair that did not work out (due to a philandering lady, in that case).

The other day, I posted How bad does it need to be in order to be not good enough, anyway? and I was focusing on the differences between Traditional and Simplified stroke data that was being used for stroke-based collation, and wondering about the net effect in Macao (a Traditional Chinese-using region for which Microsoft uses the Simplified Chinese collation, as I discussed in Is it Macau or is it Macao?).

But Andrew West was looking at some of these more extreme cases of stroke count differences (as you can tell from the comments), in particular 𧋰 (U+272f0). and managed to prove in his own blog (his post CJK-B Case Study #1 : U+272F0) that every ideograph also has a story!

He was worried that he posted too much detail to be very interesting:

I'm sorry if I spoiled your follow-up, but I'm sure you have a different, probably more interesting take on the subject than me -- my post is probably too detailed for anyone but the most dedicated CJK/Unicode geeks. I wasn't going to blog on the subject originally, but there was just too much information to put into the comments to someone else's blog.

though speaking personally I disagree. The only thing that would keep me from doing such a post myself here is that I lack the knowledge/wherewithall to do so....

Luckily I can simply link to him, instead! :-)

From Andrew's "case study" post:

I guess that once the Taiwan source glyph is corrected and the Taiwan stroke count data is amended it should be the end of the story, but the one thing that nags at me (as is the case with so many characters which only have a single Taiwan source reference) is what is the ultimate source of this character and which texts is it used in ?

For Microsoft, it raises an interesting problem if/when the reference glyph is fixed....

Okay, let's say they do fix the reference glyph, and subsequently, the stroke count.

What does Microsoft do?

Note that our Traditional Chinese font that includes U+272f0 does not have this problem (we did not pick up the incorrect glyph, possibly the font foundry realizing the same thing Andrew did and not wanting to perpetuate the mistake, but then also not telling us, either -- not to imply that there is or isn't a definite mechanism for such? Or perhaps there is a separate quality issue in the font itself?).

So either way, at this point it is just an anomaly in the sorting table, a known bug with no official communication on the change yet, but we expect at some point there might be such communication.

Since we are litedrally based on a standard in this case, no change could even be considered until it is known through official sources.

In a total stroke based collation such as this one, the difference between 13 and 19 is pretty huge, so one assumes that eventually the change would have to be picked up.

But even a change to one code point could cause index corruption in a database, which means a new major version would be required for the character.

And before rushing in to fix U+272f0 (which as Andrew mentioned it is not clear where it is needed), we have to consider the bigger problems in the other 9,767 differences and with Extension B in general.

How reliable is the ret of the data? And how many additional problems are already fixed in the font that ships even in Vista but are not fixed in the collation tables since those tables are based on a standard working from what amounts to a completely different set of reference glyphs?

I always tended to think of pronunciation-based sorts as being more worrisome technically, since an ideograph can have multiple pronunciations and by putting a stake in the ground for a version and saying that one pronunciation is the most common, we have to allow fro the fact that over time things change, and in the future the most common pronunciation might be different. We had several such changes for Hanja in the Korean collation in Vista, for example.

But now it seems like we have to look at stroke count data with the same careful eye, never knowing when future corrections would come in based on bugs....

Maybe an automated program should be run over all of the characters in PMing-ExtB, counting strokes and comparing against the stroke data in the standard, and then figuring out where other bugs might be.

But I imagine getting resources for such a review would be a challenge, and the notion of assuming the font is always right here is also flawed -- the font and data could both be wrong, after all.

The engineer in me has a hard time dealing with the fact that there are an unknown number of mistakes here some of which could perhaps be ferreted out, and the linguist-wannabe does not feel much better about that (though he is less convinced of the overall usefulness of a total-stroke-based collation and is thus less troubled by anomalies).

Plus, there is nothing to say that there are not also mistakes on the Simplified side too. More worries and more resources needed (and this one troubles that linguist-wannabe a bit more since the stroke count/stroke order based sort has in theory a bit more utility, though the notion that there are millions of people who would know the correct order to draw these ideographs they have never seen from millenia ago is also suspect!).

It is a mess, to be sure. Inevitably I am back to Andrew again, and his intro text from the post:

The CJK Unified Ideographs Extension B [13MB] block that was added to Unicode/10646 in 2001 comprises 42,711 characters, and it is no secret that there are many problems with this huge collection of mostly quite rare characters, including hundreds of cases of unifiable characters that have been erroneously encoded separately and even a handful of completely duplicate characters. There is enough material to keep a dedicated CJK-B blogger busy for years to come, but I certainly don't want to go down that particular path.

I worry more about the ones that cause implementation issues like U+272f0 will, but even so I would be just as worried about having to go down that path as he, perhaps more. Technically I worry more for my sucessors who own the area, though I do feel partially responsible since the errors of the Taiwanese standard based on errors in Unicode/10646 were perpetuated into Windows on my watch.

Should I feel worse that it was literally my request to the subsidiaries to provide the additional data I would need to extend the tables?

(They had requested us to extend them and had been refused for a long time based on technological issues that I figued out workarounds for.)

Well, either way I do feel worse. It wasn't my fault, but from the Windows standpoint it was because of me....

This post brought to you by 𧋰 (U+272f0, an Extension B CJK ideograph causing me to lose a bit of sleep!)

Microsoft is a Form 'C' shop, Part 1

Michael S. Kaplan — Mon, 29 Oct 2007 17:01:00 GMT

Microsoft has had Unicode as a part of its operating system offerings since the easrliest days of its 32-bit platforms.

And a lot that support predates asnything that Unicode later chose to provide, thus we don't use the Unicode Collation Algorithm for our sorting, for years we did not use Unicode normalization for our equivalences, and all kinds of random snafus like that somewhat random Tibetan/Myanmar thing with us not picking up Unicode changes when they happened still manage to pop up after all of these years.

Now for the most part, data coming out of Microsoft's keyboards, data entry methods, functions, methods, and algorithms has always been in what we for years called the precomposed form, which Unicode calls Unicode Normalization Form "C" in their UAX #15. Other than hiccups like code page 1258 (discussed here and here), data always tended to be in Form "C".

In fact, if you convert data to Form "D" then there are a bunch of places like in collation that you won't get the most accurate results, even in Vista where most of the equivalent forms were added to the tables to try to make the impact of using Form "D" text less noticeable....

Yet even today if you convert to Form "D" then all kinds of languages from Korean to Tibetan won't always sort as expected or as deisgned. And Vista features like LINGUISTIC_IGNORE* flags won't always return exactly equivalent results if you compare Form "C" text to Form "D" text. You are always better off converting text if you are getting it from other sources before using the NLS API for the text....

Chalk it up to gremlins in the computers and such.... not converting what they do not seem to handle on their own....

Now note that products like Access and SQL Server, being based on similar technologies only up[dated less often, still had problems even doday..

Anyway, future posts in this series will be explaining other uences our "Form 'C'- ness". This is just the intro.

This post brought to you by ೀ (U+0cc0, a.k.a. KANNADA VOWEL SIGN II)

Tell them to GO FISH?, aka Carpe Carp [Sarcalogos] (Seize the [Christian] Fish), aka Unicode List (where characters discuss character encoding)

Michael S. Kaplan — Mon, 01 Oct 2007 10:01:00 GMT

If you are a member of the Unicode List, and you do not have any real sense of what you should or should not contribute, then it is worth realizing that you have no disadvantage compared to the people who do contribute regularly -- they don't know much about it either. :-)

But if there is one rule worth learning, it is simple.

Avoid Hyperbole.

Making an outrageous or in some other way extreme claim about a character encoding proposal is a huge mistake, even if you suggest it in the context of an "April 1st" prank proposal.

And it happened this weekend^1,2 when perennial mischief maker Phillipe Verdy suggested to Jon Hanna's worry about a particular thread that after reading the suggestions given he'd have to find a different post to use for April 1st:

Why not proposing the historical Christian fish symbol at this date (April 1st)?

The fish was used and displayed on many monuments and graves, is still seen on old artistic features (like ceramics) and probably on old scriptures too, by the first Christians, before the Cross during the Roman Empire before it converted officially to the Christian religion, and accepted to use the Cross as an easily recognizable symbol to commemorate those that died on it.

An innocent start, with now retired Asmus Freytag suggesting to the Frenchman:

You haven't been in the US in a while, apparently, or you'd know that the fish symbol is widely used there. I'm not (just) referring to its use as a symbol on cars, but in print and other advertising to label commercial enterprises owned by or catering to the "true believers".

While I'm surprised that it hasn't been proposed before, it's clear that it's so far out of the realm of character encoding that it can be safely considered an April Fool's joke.

And then William J. Poser's amusing contribution:

If the Christian fish is encoded, I demand equal space for the Darwin fish.

And now we have a party, with at least another 60 posts in increasing distances from a sensible topic with each successive post.

All I know is that I am sure the UTC will see a proposal at some point for the little bugger....

Anyway, Phillipe added some historical info on the Christian fish (ref: here), which includes info about the "keyboard fish", which looks like this:

<><

And then the conversation moved to bumper stickers and then the various religious/Darwinism arguments, a threat from Her Divine Effulgence Sarasvati to cease and desist with the off-topicality of the thread³ before morphing into a discussion of Egyptian hieroglyphs and the many small fishes that are set to be encoded later.

Like I said, I'm willing to bet this is not the last we have heard about the Christian Fish proposal, and it will not be just an April 1st joke post. I have decided it is an expected risk of putting a bunch of characters in a list to discuss encoding characters....

1- For the record the only bright spot in this whole mess is that most of these people seem to have jobs because the threads that gets tons of participation seem to get it on the weekend.
2 - Of course the flip side of this is that some of these people need to try and get some lives. :-)
3 - This request was of course, mostly ignored in the subsequent 20+ posts.

This post brought to you by ◉, ⥼, ⥽, ⥾, and ⥿ (U+25c9, U+297c, U+297d, U+297e, and U+297f, a.k.a. FISHEYE, LEFT FISH TAIL, RIGHT FISH TAIL, UP FISH TAIL, and DOWN FISH TAIL - all packed in like a bunch of sardines on this one)

Every character[ sequence] has a story #30: The SMILEY (a 25-year old story, in fact)

Michael S. Kaplan — Wed, 19 Sep 2007 21:44:00 GMT

Given all of the blather about emoji and emoticons and symbols, the mail I got from Sergey earlier today puts in all in perspective.

It had the following in it:

Note the date and time, and when this post goes live.

For more on Scott and Smiley lore, see here and here.

Considering how often I use this particular item, I feel like I owe him a serious thank you!

This post brought to you by ⌣ (U+2323, a.k.a. SMILE)

Every character has a story #29: U+1000^H^H^H^H0f40, (TIBETAN or MYANMAR LETTER KA, depending on when you ask)

Michael S. Kaplan — Tue, 28 Aug 2007 10:59:00 GMT

So I was chatting with Goldie the other day and I think just after or maybe it was just before I made some ridiculous stretch of a joke joke about Anatevka (forgetting momentarily that she did not go by Golde; her nom de plume was Goldie) she asked me if there was a test case I knew off the top of my head where collation results changed between XP and Server 2003.

Interestingly, this is a question I have been waiting years for someone to ask, ever since I first pieced together the change that happened! :-)

You see, prior to Server 2003, there was no version support. You know, those functions I mentioned in posts like this one, (IsNLSDefinedString and GetNLSVersion.

As a part of the Server 2003 update, a bunch of code points got removed from the table. I'll list a bunch of them and you tell me if you see a pattern:

0x1000 32    2   2 2 ;Tibetan Ka
0x1001 32    3   2 2 ;Tibetan Kha
0x1002 32    4   2 2 ;Tibetan Ga
0x1003 32    5   2 2 ;Tibetan Nga
0x1004 32    6   2 2 ;Tibetan Ca
0x1005 32    7   2 2 ;Tibetan Cha
0x1006 32    8   2 2 ;Tibetan Ja
0x1007 32    9   2 2 ;Tibetan Nya
0x1008 32   10   2 2 ;Tibetan Reversed Ta
0x1009 32   11   2 2 ;Tibetan Reversed Tha
0x100a 32   12   2 2 ;Tibetan Reversed Da
0x100b 32   13   2 2 ;Tibetan Reversed Na
0x100c 32   14   2 2 ;Tibetan Ta
0x100d 32   15   2 2 ;Tibetan Tha
0x100e 32   16   2 2 ;Tibetan Da
0x100f 32   17   2 2 ;Tibetan Na
0x1010 32   18   2 2 ;Tibetan Pa
0x1011 32   19   2 2 ;Tibetan Pha
0x1012 32   20   2 2 ;Tibetan Ba
0x1013 32   21   2 2 ;Tibetan Ma
0x1014 32   22   2 2 ;Tibetan Tsa
0x1015 32   23   2 2 ;Tibetan Tsha
0x1016 32   24   2 2 ;Tibetan Dza
0x1017 32   25   2 2 ;Tibetan Wa
0x1018 32   26   2 2 ;Tibetan Zha
0x1019 32   27   2 2 ;Tibetan Za
0x101a 32   28   2 2 ;Tibetan Aa
0x101b 32   29   2 2 ;Tibetan Ya
0x101c 32   30   2 2 ;Tibetan Ra
0x101d 32   31   2 2 ;Tibetan La
0x101e 32   32   2 2 ;Tibetan Sha
0x101f 32   33   2 2 ;Tibetan Reversed Sha
0x1020 32   34   2 2 ;Tibetan Sa
0x1021 32   35   2 2 ;Tibetan Ha
0x1022 32   36   2 2 ;Tibetan A
0x1026   1    0   3 0 ;Tibetan Vowel Sign I
0x1027   1    0   4 0 ;Tibetan Vowel Sign Short I
0x1028   1    0   5 0 ;Tibetan Vowel Sign U
0x1029   1    0   6 0 ;Tibetan Vowel Sign E
0x102a   1    0   7 0 ;Tibetan Vowel Sign O
0x102b 32   37   2 2 ;Tibetan Chuchenyige
0x102c 32   38   2 2 ;Tibetan Visarga
0x102e   1    0   8 0 ;Tibetan Anusvara
0x102f 32   39   2 2 ;Tibetan Right Brace
0x1030   1    0   9 0 ;Tibetan Under Ring
0x1031 32   40   2 2 ;Tibetan Ditto
0x1033 32   41   2 2 ;Tibetan Single Ornament
0x1034 32   42   2 2 ;Tibetan Shad
0x1035 32   43   2 2 ;Tibetan Tseg
0x1036   1    0 10 0 ;Tibetan Candrabindu
0x1037   1    0 11 0 ;Tibetan Candrabindu With Ornament
0x1038 32   44   2 2 ;Tibetan Comma
0x1039 32   45   2 2 ;Tibetan Rinchanphungshad
0x103a 32   46   2 2 ;Tibetan Rgyanshad
0x103b   1    0 12 0 ;Tibetan Honorific Under Ring
0x103c 32   47   2 2 ;Tibetan Left Brace
0x103d   1    0 13 2 ;Tibetan Vowel Sign Ai
0x103e   1    0 14 2 ;Tibetan Vowel Sign Au
0x1040 12   16 70 2 ;Tibetan Digit Zero
0x1041 12   47 70 2 ;Tibetan Digit One
0x1042 12   66 70 2 ;Tibetan Digit Two
0x1043 12   84 70 2 ;Tibetan Digit Three
0x1044 12 102 70 2 ;Tibetan Digit Four
0x1045 12 121 70 2 ;Tibetan Digit Five
0x1046 12 140 70 2 ;Tibetan Digit Six
0x1047 12 158 70 2 ;Tibetan Digit Seven
0x1048 12 176 70 2 ;Tibetan Digit Eight
0x1049 12 194 70 2 ;Tibetan Digit Nine
0x104a 32   48   2 2 ;Tibetan Double Shad
0x104b   1    0 15 0 ;Tibetan Virama
0x104c   1    0 16 0 ;Tibetan Lenition Mark

The problem here? The data is all wrong!

This version of Tibetan, first described in Unicode Technical Report #2, was removed in Unicode 1.1 when the ISO 10646 merger happened, and then Tibetan was added back in Unicode 2.0 in an entirely different place.

If you look at DerivedAge.txt, you will see that the new Tibetan was added in July 1996.

But Windows had been carrying data around from Unicode 1.0 since the very beginning of its 32-bit life, possibly as far back as NT 3.5 or even NT 3.1 (I am almost curious enough to go try and find out which, actually!).

In Server 2003, it was decided that this incredibly invalid data had to be removed.

For one thing, it is just really bad to start a formal versioning functionality with crap like that in there.

And for another, this space that was left empty after the 1.1 merge was actually filled as of Unicode 3.0 in 1999 -- with the Myanmar script. And even though Windows did not add weights for it yet (we did not do so until Vista), keeping known bad data seemed like a pretty bad idea...

So, all of the above code points had weight in Windows from the early 32-bit days until XP, and then again in Vista (and were essentially weightless in the years between).

And of course the snapshots in Jet 4.0, ACE (the version of Jet that ships with Access >= 2007), SQL Server 7.0, 2000, and 2005 all have these somewhat bogus code points as well....

Oops for them (plus we can be snotty and superior about it now that is fixed in Windows!)

When one talks to old timers about the 1.1 merge between Unicode and ISO 10646, you have trouble getting a straight answer -- it is like that bit from The Number of the Beast:

I've given up trying to find out what happened in 1965: "The Year They Hanged the Lawyers." When I asked a librarian for a book on that year and decade, he wanted to know why I needed access to records in locked vaults. I left without giving my name. There is free speech -- but some subjects are not discussed....

So that is all I can say about the old U+1000 TIBETAN LETTER KA which died in Unicode in the early 1990s only to rise from its ashes in 1996 at U+0f40 with U+1000 being assigned to MYANMAR LETTER KA in 1999. The same character lived on at Microsoft until 2003, only to be reborn along with its Myanmar cousin in Vista....

This post brought to you by ཀ and က (U+0f40 and U+1000, a.k.a. TIBETAN LETTER KA and MYANMAR LETTER KA)

Every character has a story #28: U+1e9e (CAPITAL SHARP S)

Michael S. Kaplan — Fri, 24 Aug 2007 10:01:00 GMT

That night I saw in the pipeline fair
A character that wasn't there
Non-existence won't stop the encoding; it's true
So it's coming soon to a Unicode near you!

It all started with Every character has a story #15: CAPITAL SHARP S (not encoded), and then continued in Every character has a story #26: CAPITAL SHARP S (might be encoded?).

And you can read the title of this latest blog post and know what is happening now without any hints from me....

Though I must admit the trip has been both long and strange.

It was decided within both ISO 10646 and Unicode that this interesting character was indeed going to be encoded (as per the pipeline, it was officially accepted on May 18th of this year and as of April 27th is in Stage 5 of the ISO process.

And I have probably learned more about the nature of letters within typography than any experience before or since!

Immediately after this process started, there was a whole bunch of discussion on the Unicode List about a very important topic:

WHAT DOES A CAPITAL SHARP S LOOK LIKE?!?

There were a whole bunch of proposals here, and much of the conversation then took a southward turn.

Like people suggesting that DIN should be dissolved by law for supporting the proposal.

And others pointing out that the proposal specified an enlarged version of ß. nothing more and nothing less.

But I have told you about the Unicode List, the next 100 messages oscillating between discussing typographic innovations that would make sense if the letter did indeed exist based on different theories of its etymology and people who remained unconvinced by the proposal even after it had been accepted since in their view it isn't a freaking letter in the first place.

Plus lots of SZ vs. SS arguments.

An informal survey of the Germans I knew all seemed to fall squarely in the camp of the insanity of DIN, though many of them considered the opinion to be redundant....

And then with a few people talking about the consequences for Unicode properties, just to add the vague scent of relevance to the discussion. :-)

John Hudson had in my opinion the most amusing observation:

The irony of the recent exchanges is not lost on me:

On the one hand, we have Marnen Laibow-Koser, who thinks that this character should not exist, but that it does, and therefore needs to be encoded.

On the other hand, we have me, who thinks that this character should exist, but that it does not, and therefore does not need to be encoded.

Just so.

For Microsoft, it raises some interesting questions for both collation and case for the next version of Windows.

I mean, think about the issues I have already talked about in posts like What the %#$* is wrong with German sorting? where we make ss equal to ß so that the uppercase version "SS" will sort near the ß in a sort ignoring case -- where we do things that make less linguistic sense in order to give regular results that are intuitive.

So who would expect that if U+00df is equal to ss that U+1e9e wouldn't be made equal to SS? Meaning that in the collation tables, U+00df and U+1e9e would simply be case variants, with no real choice in the matter.

And as to casing....

Now just because we make the relationship in casing does not mean we make it in collation. After all, as I have pointed out several times before, collation != case.

But on the other hand, the case table is used in order to enforce the case insensitivity in the NT object namespace and the file system. And one clear issue is that there is no good reason to allow one to put filenames differing only by the presence of U+00df and U+1e9e in the same directory. Users would either never try it or they would never expect it to work. So it is quite possible that in the next version of Windows (which only does simple casing) it may make the most sense to make the two characters case variants of each other -- to enforce reasonable use of both letters!

There is still lots of time to decide, though at present I am leaning this way since it will give the most intuitive behavior for end users (even at the expensive of giving slightly unintuitive results for developers).

This post brought to you by ß and ß (U+00df and U+1e9e, LATIN SMALL LETTER SHARP S and CAPITAL SMALL LETTER SHARP S)

Fractions may be your friends, but they won't pick you up at the airport!

Michael S. Kaplan — Sat, 14 Jul 2007 22:01:00 GMT

To me, fractions will always have a special place. The teacher pointed out we all knew what ½ was, and we all knew what 0.5 was, and we all knew about division. Then he blew my mind when he pointed out they were not connected because they were multiple things to memorize, but because they really were the same thing.

Fractions are the first time I saw math operations as all being connected. And it mostly because that one teacher was the first one to teach it that way (and the book didn't, it was him). I hope the books are better now, since I think my old teacher is retired....

He also was the first person I remember telling me that fractions were our friends. I think I almost believed it at the time, as it got me realizing that math had the potential to be cool.

Thanks, Mr. Snodgrass!

Anyway, back to fractions.

In Unicode, there are a whole bunch of fractions encoded. Here they are in numeric order (something not even Shell number sorting will do for you!):

Code Point	Character	Character Name	1.0 Character Name
U+215b	⅛	VULGAR FRACTION ONE EIGHTH	FRACTION ONE EIGHTH
U+2159	⅙	VULGAR FRACTION ONE SIXTH	FRACTION ONE SIXTH
U+2155	⅕	VULGAR FRACTION ONE FIFTH	FRACTION ONE FIFTH
U+00bc	¼	VULGAR FRACTION ONE QUARTER	FRACTION ONE QUARTER
U+2153	⅓	VULGAR FRACTION ONE THIRD	FRACTION ONE THIRD
U+215c	⅜	VULGAR FRACTION THREE EIGHTHS	FRACTION THREE EIGHTHS
U+2156	⅖	VULGAR FRACTION TWO FIFTHS	FRACTION TWO FIFTHS
U+00bd	½	VULGAR FRACTION ONE HALF	FRACTION ONE HALF
U+2157	⅗	VULGAR FRACTION THREE FIFTHS	FRACTION THREE FIFTHS
U+215d	⅝	VULGAR FRACTION FIVE EIGHTHS	FRACTION FIVE EIGHTHS
U+2154	⅔	VULGAR FRACTION TWO THIRDS	FRACTION TWO THIRDS
U+00be	¾	VULGAR FRACTION THREE QUARTERS	FRACTION THREE QUARTERS
U+2158	⅘	VULGAR FRACTION FOUR FIFTHS	FRACTION FOUR FIFTHS
U+215a	⅚	VULGAR FRACTION FIVE SIXTHS	FRACTION FIVE SIXTHS
U+215e	⅞	VULGAR FRACTION SEVEN EIGHTHS	FRACTION SEVEN EIGHTHS

That last column is really interesting -- it is the name of the character in Unicode 1.0, prior to the merger with ISO 10646. Name changes were done to go along with particular preferences in 10646, which in most cases related to compatibility with other ISO standards that involved names nad inother cases involved particular conventions.

Though I have to wonder how we are going to get the next generation of kids interested in considering fractions to be their friends if we spend all of our time telling them that fractions are vulgar!

Now all of these characters are really thought to be compatibility characters -- the "right" way to do fractions is to use regular numbers and U+2044 (FRACTION SLASH) between them...

Which would perhaps explain why they are vulgar? :-)

Actually, the reason they are called vulgar is apparently that the reference glyphs use diagonal slashes rather than horizontal bars. Though the two different ways to write fractions are considered glyph variants of each other (as they should be, since they are). Which means that a font developer can use either way to show them.

There are other standards that actually do try to distinguish between them and encode both -- for example the ones in the DPRK (which caused for some interesting discussions in WG2 and UTC when the DPRK additions to Unicode were being discussed back in 2001, with interesting conversations about variations selectors, if memory serves. I do remember that no one from UTC wanted to encode that extra set of fractions.

In any case, I guess the moral of the story using the principles of logic is that your friends are vulgar (which actually was an alternate title I considered for this post)....

This post brought to you by ⁄ (U+2044, a.k.a. FRACTION SLASH)

Every character has a story #26: CAPITAL SHARP S (might be encoded?)

Michael S. Kaplan — Thu, 03 May 2007 22:15:00 GMT

(ref: Every character has a story #15: CAPITAL SHARP S (not encoded))

When Michel Suignard came back from the WG2 meeting, I scooted over to get the quick word on what interesting things happened....

I was surprised to hear that the Capital Sharp S was accepted, which means it will probably be discussed at the upcoming UTC meeting.

Kind of ironic given John Hudson's conversation with Andreas Stötzner I mentioned before.

(Here is the revised proposal for those who are interested)

I have talked about the Capital Sharp S proposal previously, I won't comment much now though probably I will after the UTC, though the linguistic and technical issues are both likely to be the subject of future posts!

In the meantime, Adam Twardoch pointed folks to a blog entry from Ivo Gabrowitsch on the subject, for those of you who know German (you can read it here).

This post brought to you by ß (U+00df, a.k.a. LATIN SMALL LETTER SHARP S)

Every character has a story #25: U+00a4 (CURRENCY SYMBOL)

Michael S. Kaplan — Thu, 15 Mar 2007 11:39:00 GMT

(Don't these 'Every character has a story' posts remind you of a Colbert-esque Better Know a District series? Maybe I should rename the series to Better Know a Character. What does everyone think?)

Just recently, John asked:

Just a quick question I can't seem to find an answer for: what is the purpose and actual utility of the International Monetary Symbol? (ALT-0164)

I have never seen it in use (except as a non printable character used as an end-of-cell marker) but I have been asked if it could/should be used in some of our applications.

Although it is true that in Unicode Every Character has a Story, there are some characters encoded in Unicode whose story eludes us.

It is easy to look at ¤ (U+00a4, a.k.a. CURRENCY SIGN) and assume it must have a hell of a story.

People in the wide wide world on the whole don't recognize it, and it isn't actually a currency sign in any country in the world, nor has it ever been.

Unicode does give it a general category of Sc (Symbol, Currency), and it does have a place not only in Unicode but 0xA4 in Windows Code Page 1252 and 0xCF in OEM Code page 850 and 0xA4 in ISO-8859-1.

Everywhere you look, there's this currency sign, one that is used for no currency.

I even tried asking Ken Whistler, who had no conclusive thoughts to add:

Michael,

Nothing more than my guesses. It predates my involvement, since it is an ISO 8859-1 thing, whence it got into Unicode.

And 8859-1 got it from IBM, perhaps. It is:

SC010000 International Currency Symbol

in the old IBM Graphic Character Identification System.

Maybe some old hand at IBM would know what it was originally intended for.

I'm gonna guess that it was intended as a placeholder character for EBCDIC code pages, to enable formatting of currency that had symbols not otherwise representable on the code page.
In other words, as a *replacement* currency symbol for a missing currency symbol.

But that's just a guess, absent information from the horse's mouth.

A few more discrete inquiries long the lines Ken suggested also yielded nothing, though I did verify that IBM Graphic Character Identification System connection.

It is funny that when space was at such a premium in ISO 8859-1 that they would use up a slot for a character that isn't really used for anything. Though it does seem to be in every font and it is not used for anything else (maybe that is why it is used in Word as an end of cell marker in tables:

Weird, huh?

To answer John's question, in my opinion, why not use it? It has nothing better to contribute, so have fun! :-)

Maybe somebody out there knows for sure what IBM's original plans were for it. Plans so important that ISO 8859-1, which managed to miss out on an uppercase version of a letter (a character story for another day, one that I actually do know!), had to include it.

Because every character has a story, even if we don't know what it really is....

This post brought to you by ¤ (U+00a4, a.k.a. CURRENCY SIGN)

Every character has a story #24: U+0308 (COMBINING DIAERESIS)

Michael S. Kaplan — Mon, 04 Sep 2006 10:01:00 GMT

I am reminded of a scene from the 1991 film The Doctor starring William Hurt, modified here to be a bit more linguistic than medical:

Linguist: Nancy, are my repeated vowels pronounced differently?
Nancy: No, doctor.
Linguist: That's funny, I always trema when you're near.

There are essentially two¹ different traditions for the meaning of two dots on top of a vowel:

Umlaut - Described in Wikipedia as a "...modification of a vowel which causes it to be pronounced more similarly to a vowel or semivowel in a following syllable."

Trema or Diaeresis - Described in Wikipedia as the "...division of two adjacent vowels as two syllables rather than as a diphthong."

Now in Unicode and ISO 10646, these two very different diacritical purposes are unified under a single character -- the diaeresis. Which is kind of ironic given that the meaning of 'diaeresis' tends to suggest a division rather than any sort of unification....

Ignoring that bit of irony in the naming decision, a unification does make sense since they really do look pretty much the same, and a disunification would be a huge target for spoofing (something we really do not need any more of, frankly!). Though to tell the truth, in quality typography the umlaut dots are usually a bit closer to the letter than the trema dots.

Back in 1993, Deutches Institut für Normung (DIN) sent a proposal to WG2 that stated

Currently, a substantial amount of existing German data distinguishes between Umlaut and Trema. Both diacritics have a similar, but not necessarily identical representation, both have quite different properties e. g. with regards to sorting (cf. DIN 5007).

In particular, German library data is currently stored according to ISO 5426 "Extension of the Latin alphabet coded character set for bibliographic information interchange" which distinguishes between the two diacritics Umlaut (4/9) and Trema (4/8). However, in ISO/IEC JTC1/SC2 N3125 "Finalized Mapping between Characters of ISO 5426 and ISO/IEC 10646-1 (UCS)" both are mapped to the same UCS character, U0308. There is thus no standardized way to ensure roundtrip compatibility between the two standards.

For Germany and in particular for its national library (Deutsche Bibliothek) it is imperative for the integrity of German data that it be possible to maintain the distinction between Umlaut and Trema also in the UCS in a standardized way. Lack of ability to do so affects millions of bibliographic data records in the Deutsche Bibliothek alone (to be exact, 14 956 289 records as of October 2002) and about 110 million bibliographic data records in German and Austrian regional library networks.

In other words, they had a need to distinguish these two diacritics, which are actually not unified in a different ISO standard. Their initial proposal from document N2593:

We therefore request

a) the encoding of two new characters, LATIN VARIATION SELECTOR UMLAUT in position U0241 and LATIN VARIATION SELECTOR TREMA in position U0240 (the positions are suggestions only).

b) the insertion of the following text into informative Annex F "Alternate format characters" as F.2.6 "Latin selectors"

"LATIN VARIATION SELECTOR UMLAUT (U0241): Uniquely identifies the preceding character as using /being the Umlaut diacritic (cf. ISO 5426, code position 4/9)

LATIN VARIATION SELECTOR TREMA (U0240): Uniquely identifies the preceding character as using / being the Trema diacritic (cf. ISO 5426, code position 4/8)

In the absence of any variation selector, neither the character COMBINING DIAERESIS U0308 nor any of the Latin letters with diaeresis can be interpreted as representing uniquely the Umlaut or uniquely the Trema.

The LATIN VARIATION SELECTOR UMLAUT or the LATIN VARIATION SELECTOR TREMA should only be used directly following the Latin characters shown below:

00C4 LATIN CAPITAL LETTER A WITH DIAERESIS
00D6 LATIN CAPITAL LETTER O WITH DIAERESIS
00DC LATIN CAPITAL LETTER U WITH DIAERESIS
00E4 LATIN SMALL LETTER A WITH DIAERESIS
00F6 LATIN SMALL LETTER O WITH DIAERESIS
00FC LATIN SMALL LETTER U WITH DIAERESIS

U0308 COMBINING DIAERESIS

Neither the LATIN VARIATION SELECTOR UMLAUT nor the LATIN VARIATION SELECTOR TREMA carry a defined meaning when they follow any other character ."

c) Change in ISO/IEC JTC1 SC2 N3125 (= ISO/TC46/SC4 WG1), section 3 "Mapping of Characters" the table to:

4/8 Trema, Diaeresis 0308 0240
4/9 Umlaut 0308 0241Z

Unfortunately, Variation Selectors can only be used on base characters, not on combining characters. so while the scenario is valid, the DIN suggested soluion is not. The UTC discussed possible solutions at length before producing the following recommendation, instead:

While recognizing the drawbacks to all of the alternatives to encoding a new COMBINING UMLAUT character outlined in WG2 N2766, we believe that there is a workable alternative solution which has, to date, been overlooked. The solution consists, essentially, of using U+034F COMBINING GRAPHEME JOINER (CGJ), in its intended semantics in 10646/Unicode, to make the relevant sorting, searching, and data mapping distinctions required for umlaut versus tréma. In particular, the distinction we propose is:

U+0308 → umlaut
<CGJ U+0308> → tréma
<a U+0308> → a umlaut
<a CGJ U+0308> → a tréma

The sequences <a U+0308> and <a CGJ U+0308> are not canonically equivalent. this means that the distinction will not be normalized away on conversion in and out of bibliographic systems. This eases the interoperability problem. Both sequences will display as ä, as they should. Furthermore, the semantics of CGJ are such that it should impact only searching and sorting, for systems which have been tailored to distinguish it, while being ignored in other respects in interpretation.

The reason for treating the existing sequence <a U+0308> as representing the umlaut in German bibliographic systems, despite the name of U+0308 COMBINING DIAERESIS, is that this is the unmarked case, representing the vast majority of extant data. The marked form <a CGJ U+0308> should be utilized for the marked case in the data, namely the tréma, which is far, far less frequent in German bibliographic data. This minimizes the conversion and data rectification issues, and also guarantees that representations including CGJ will be uncommon in data converted out of the German bibliographic records.

The existence of separate representations for umlaut and for tréma, which are not canonically equivalent (and thus not neutralized by normalization processes in the data) enables German implementations which need to distinguish the two for searching and sorting, to systematically maintain weighting distinctions to do the right thing. <a U+0308> = <ä> can be treated as equivalent to <a, e> for sorting purposes, while the tréma <a CGJ U+0308> can be weighted as a secondary variant of <a> thus resulting in the desired behavior for such systems. Existing collations which do not distinguish tréma and umlaut in German data will continue to work exactly as they currently do, since in default collation tables CGJ is ignored in weighting.

We believe that this proposed solution has the correct mix of technical attributes to enable the German library networks to make the required distinction, to correctly convert existing ISO 5426 bibliographic records, and to implement the desired sorting and searching behavior for German data represented directly in 10646/Unicode.

At the same time, this solution does not introduce incompatibilities or non-interoperability issues for other existing implementations of 10646/Unicode which handle German data.

It is again ironic that in the (rare) situation where an attempt to distinguish them is required that the default case is suggested as being the unlaut while exceptional case is the diaeresis. :-)

The use of a combining diacritic is still (to this day) controversial in Unicode when people unfamiliar with the standard who are native speakers of languages like Swedish or Finnish and who are asked to think of these standalone letters as equivalent to a different letter plkus a diacritic. The many people who would prefer all of the Indic languages to separately encode all the instances of base letter plus virama have an analagous complaint.

1 - At this point I will take judicial notice of the phenomenon known as the Heavy Metal Umlaut, described ad nauseum here. It is in fact the Heavy Metal Umlaut that inspired Cathy's desire for a bumper sticker that would say Stop Indiscriminate Umlauting!, although I find that approach to be a tad reactionary. The importance to our culture of Spin̈al Tap and Blue Öyster Cult is undeniable, as is the need to avoid fear of the reaper and to turn the volume up to 11.....

This post brought to you by ̈ (U+0308, a.k.a. COMBINING DIAERESIS)

Every character has a story #23: U+00ad (SOFT HYPHEN)

Michael S. Kaplan — Sat, 02 Sep 2006 20:34:00 GMT

Last night I was upon the stair
A little hyphen that wasn't there.
It wasn't there again today;
Oh how I wish he'd go away!

The SOFT HYPHEN has a long if not entirely distinguished history.

It starts back in ISO 8859-1, which puts it at 0xAD, and in a rare exception to the usual practice of not explaining semantics of the encoded characters, it spends a bit of time talking about the soft hyphen, saying:

A graphic character that is imaged by a graphic symbol identical with, or similar to, that representing hyphen, for use when a line break has been established within a word.

As you can see, we are already in trouble here. It is a graphical character with a visible definhed glyph that is usually invisible and which impacts line break, a formatting operation.

And of course beyond the sloppiness in the definition there is the fact that it is usually unreasonable to assume that a person would type in this character explicitly. Clearly it is a better answer to have per language dictionaries that contain hyphenation rules in them, as the SOFT HYPHEN "do it yourself" principles are simply not going to work in practice.

The HTML 4.0 spec has its own content on the soft hyphen. In section 9.3.3. (Hyphenation) the following text is provided:

In HTML, there are two types of hyphens: the plain hyphen and the soft hyphen. The plain hyphen should be interpreted by a user agent as just another character. The soft hyphen tells the user agent where a line break can occur.

Those browsers that interpret soft hyphens must observe the following semantics: If a line is broken at a soft hyphen, a hyphen character must be displayed at the end of the first line. If a line is not broken at a soft hyphen, the user agent must not display a hyphen character. For operations such as searching and sorting, the soft hyphen should always be ignored.

In HTML, the plain hyphen is represented by the "-" character (- or -). The soft hyphen is represented by the character entity reference  ( or )

Ok, so once again we have a graphic character that is actually a formatting character tied up with line breaking. And the text seems pretty ambivalent about how a browser might be expected to interpret the soft hyphen -- clearly it is not some terrible sin if it does not do special line breaking behavior.

Doesn't  sound like perfect character entity reference for a character that may or may not be visible and which, even if visible, should be ignore for searching and sorting? The little bugger even sounds shy!

Now if you look at the ECMA 94 standard, available online for free here, it does have a wording that is almost the same as 8859-1's, and the difference may be striking to some but it did not impact me as much. Perhaps it iois edging more towards the formatting role of the character....

At this point, before I jump into Unicode, I'll mention that the other day Tihiy asked in the Suggestion Box:

Can you explain why Charmap refuses to display characters with 0xAD code?

Well, given the rules surrounding SOFT HYPHEN and the fact that it is impossible for any character to break a line when it is displayed in the single line text control in the Windows Character Map, it is obvious why the simple program that builds thae grid can display it even if it will not appear in the textbox below:

Of course this does not mean that it isn't there -- the SOFT HYPHEN, if included in the text stream, will be there even if it is usually invisible and ignored.

Now I say usually because in operations like collation on Windows, the SOFT HYPHEN will be ignored (it is given no weight) but it will also break compressions. In practice this shoul not matter since one should never break a word in the middle of a compression, and in fact this trick could be used to force collation to work right in cases like this one in Hungarian, though I'd recommend against it since you would probably not want to break a line in the middle of a word even in that case....

So, what does Unicode say?

In 2.0, it said:

U+00AD soft hyphen indicates a hyphenation point, where a line-break is preferred when a word is to be hyphenated. Depending on the script, the visible rendering of this character when a line break occurs may differ (for example, in some scripts it is rendered as a hyphen -, while in others it may be invisible).

In other places, the soft hyphen is described as a "discretionary hyphen", which clearly suggests the formatting role as well. It is becoming less and less of a graphic character all the time!

Unicode 4.0, after extensive discussion and review, made the switch for good, and the following two points are called out in the "changes for 4.0" text, with the following two bullet points:

Default Ignorables. Added Hangul Filler characters, U+00AD soft hyphen, CGJ, and ZWS
Soft Hyphen. U+00AD soft hyphen was also changed to General Category Cf. Its semantics were clarified: it marks a position for hyphenation, rather than being itself a hyphen character. (The Hyphen property itself was stabilized, and thus not changed to reflect this.)

And the text in UAX#14: Line Breaking Properties points out yet another issue that people may not have considered, buried in section 5.3 Use of Soft Hyphen:

The action of a hyphenation algorithm is equivalent to the insertion of a SHY. However, when a word contains an explicit SHY it is customarily treated as overriding the action of the hyphenator for that word.

Every time I think about the issue, I am reminded of this case, whether an attempt to optimize actually inhibited other optimization attempts. Yet another reason to avoid the soft hyphen? :-)

There is other trivia, like it is removed by nameprep for IDN, Apple's ATSUI does not support it in version 1.1 and later, and Microsoft Typography talks about it a bit as well.

Now after the 4.0 change, Markus Kuhn wrote a strongly worded dissenting opinion on the change, which Ken Whistler responded to:

I believe the recent "clarification" of the semantics of the SOFT HYPHEN (U+00AD) character in Unicode 4.0 had an unfortunate outcome. In particular, changing its class from Pd to Cf in UnicodeData.txt breaks backwards compatibility with how this character was widely used in ISO 8859-1 terminals for the past 15 years and causes now headaches with the designers of VT100-style terminal emulators with ISO 8859-1 and UTF-8 support.

This may well be the case. I don't have any particular iron in this fire, since I was neither in the camp advocating for this change nor was I particularly set up to argue against making the change.

But the fact that this issue came up, was argued at length, was put up as a public issue for an extended period of time, and then argued some more before it was decided, indicates to me that the status of U+00AD SOFT HYPHEN as a gc=Pd character was causing other people headaches as it stood.

As Unicode claims for U+0000 to U+00FF to be compatible with ISO 8859-1, it should also respect the intended and de-facto use of ISO 8859-1 characters and should not change their semantics over a decade later.

The establishment of Unicode character properties for Unicode characters does not, ex post facto, change the semantics *of* ISO 8859-1. If that were the case, then any number of character property assignments (including compatibility and canonical decomposition mappings), and character property assignment *changes*, such as those for U+00B7 MIDDLE DOT, could be equally attacked as ex post facto changes to ISO 8859-1.

But the additional character behavior specified by the Unicode Standard does not impose constraints back onto standards that those characters map to -- including ASCII and ISO 8859-1. Nothing that the Unicode Standard says about *Unicode* characters can suddenly make a conformant ISO 8859-1 implementation nonconformant in the way it handles characters.

The issue, instead, is interoperability for implementations of Unicode that map back and forth to implementations of 8-bit character encodings (or others), including ISO 8859-1. And I suspect, in the case of SOFT HYPHEN, that the problem we are facing is really that SOFT HYPHEN has had a long history of legacy implementations in two (or more) incompatible ways.

Certainly the terminal display protocols that insert line-ending SOFT HYPHENS as graphic characters which can be stripped back out when presentation text is restored to content text has a long history. But the other model also way predates the examples you cited, going at least as far back as WordStar's internal use of nondisplaying soft hyphen characters as line break opportunities that only displayed visibly (with a hyphen) at actual line breaks. For WordStar it was 0x1E for 'inactive soft hyphen', which was an inserted line break opportunity for word-wrap, and 0x1F for 'active soft hyphen', which was an actually broken word for word-wrap, displayed (and printed) visibly. (WordStar *predates* ISO 8859-1, by the way, since it was first released in 1979.)

As discussed in detail for example on

http://www.cs.tut.fi/~jkorpela/shy.html

the ISO 8859-1 standard defines, in section 6.3.3 the SOFT HYPHEN as "[a] graphic character that is imaged by a graphic symbol identical > with, or similar to, that representing hyphen".

The ISO 8859-1 standard uses unfortunately only the rather unclear words "for use when a line break has been established within a word" as the complete definition of the intended usage of this character. This clearly falls short completely of setting up a document processing model and defining unambiguously what role SOFT HYPHEN plays it its various phases and functions.

Yep. And that has contributed to the confusion for years. It didn't help that 8859-1 didn't image 0xAD SOFT HYPHEN with a hyphen glyph in the chart, but instead with a "SHY" acronym, implying that it was, in fact, a "funny" character that might not always display visibly. That, plus the less than clear wording in the note on SOFT HYPHEN (now in Clause 5.3.3 in 8859-1) was symptomatic of the aversion of SC2 standards to define "character processing" behavior, but also reflected, I suspect, a deliberate willingness to allow for inconsistent processing models. It isn't much of a stretch to interpret the wording in Clause 5.3.3 as:

   "A graphic character that [when imaged] is imaged by a graphic symbol identical with, or similar to, that representing HYPHEN, ..."

which opens the door to the Word/WordPerfect etc. style interpretation.

The definition "graphic character that is imaged by a graphic symbol identical with, or similar to, that representing hyphen" made it clear to users familiar with the above mentioned problem that the SOFT HYPHEN is just an alternative of the normal graphical character HYPHEN, for use when a hyphen is inserted by a line formatting routine.

I don't think it was quite so clear as that.

[ snip HTML discussion ]

This HTML 4 reinterpretation is essentially the semantics that Unicode then adopted as well.

Nevertheless, there is a vast number of VT100 terminal emulators, printers, and similar 8-bit output devices out there that treat the SOFT HYPHEN as a full graphical character, as had been suggested by ISO 8859-1

The problem with this is that is assumes that "graphical character" is well-defined and never involves ambiguities of display for SC2 standards.

If you look at ISO 8859-8 (Hebrew), when it was revised to make the use of LEFT-TO-RIGHT MARK and RIGHT-TO-LEFT MARK part of the standard, so that implicit order bidi with 8859-8 was well-defined, those characters were *also* described as "graphic characters", cloning the wording right out of the longstanding and traditional, if somewhat bizarre wording used to describe the SPACE character. For LEFT-TO-RIGHT MARK:

"A graphic character the visual representation of which consists of the absence of a graphic symbol, which acts like a left-to-right character in a bidirectional context..."

If, for an 8859 standard, a character which *never* has a visible display glyph (except for charts or "Show Hidden" contexts) can be considered to be a "graphic character", you can see why the situation for SOFT HYPHEN can be considered less than
clear.

and by the old application need to distinguish between content and hyphenation hyphens in formatted presentation data streams.

It is used today by a number of UTF-8 terminal applications to decide, by how many character cell positions the cursor will advance if the Unicode character provided as an argument is sent to the terminal. The rules for generating its semantics from Unicode tables are very simple and include the rule

- Other format characters (general category code Cf in the Unicode database) and ZERO WIDTH SPACE (U+200B) have a column width of 0.

With the change of SOFT HYPHEN from general category code Pd to Cf in the Unicode 4.0 database, this causes now terminal behaviour to change from wcwidth(0x00ad) = 1 to wcwidth(0x00ad) = 0. In other words, what used to be a spacing graphical character in accordance with ISO 8859-1 that always advances the cursor by one cell after printing the glyph of a hyphen is not an ignoreable and usually invisible format character.

It seems like the obvious fix here is to exempt U+00AD from the generic class treatment of Cf characters, both in the stated documentation and in the implementation. In other words, for the purposes of those UTF-8 terminal applications, SOFT HYPHEN is not an "other format character", but is an exception that should go on behaving exactly as you have it currently defined.

By the way, implementers cannot, now, assume that gc=Cf characters (format controls) should *always* be invisibly displayed. The addition of the various Arabic prepositive numeric accumulators (U+0600 ARABIC NUMBER SIGN and the like) have added a subclass of format controls which *do* have visible display glyphs. And there is now a separate Unicode character property, Default_Ignorable_Code_Point, which should also be taken into account when deciding whether a particular character, by default, should be displayed with a zero glyph or a black box glyph, for example, if uninterpreted.

In this sense, Unicode 4.0 breaks with the well-established tradition of interpreting the SOFT HYPHEN as a graphical character in output devices.

It would have been nice, if Unicode hadn't done that. Unicode could instead have chosen to add a new ignorable formatting character for marking possible hyphenation points in documents, which could be called for instance HYPHENATION POINT. A formatting function can then either discard a HYPHENATION POINT (if it ended up inside a formatted line), or convert it into the graphical SOFT HYPHEN character, where the hyphenation point ended up at the end of a line in the presentation data stream.

This possible approach was also debated, but was rejected. The opponents of that approach can speak for themselves, but if I recall, this approach would, itself, have had at least as many legacy compatibility issues.

This would have preserved backwards compatibility with the zillions of ISO 8859-1 output devices out there that treat SOFT HYPHEN as a graphical character.

What shall I now do as the implementor of an ISO 8859-1 terminal emulator when I receive a SOFT HYPHEN?

Exactly what you are currently doing.

Will the next edition of ISO 8859 be changed, to remove the definition of the SOFT HYPHEN as a graphical character?

Of course not. It will stay exactly as it is.

The ambiguity in "graphic character" will say unchanged in the SC2 standards. Note that 10646 itself keeps the traditional SC2 definition:

A character, other than a control function, that has a visual representation normally handwritten, printed, or displayed.

but then proceeds to encode a whole host of space characters and format control characters which normally *don't* have a visual representation. These are then swept under the rug with the same logical nicety used for SPACE:

"A graphic character the visual representation of which consists of the absence of a graphic symbol."

Uh, huh. O.k., well, then... ;-)

Or, my preferred outcome, do you agree that all this SOFT HYPHEN = Cf revision was probably a mistake and we should undo everything quickly in the next revision?

I think that is most unlikely at this point. The issue for SOFT HYPHEN was up for public review for rather a long time. The decision was not hurried for it, but extended through a number of UTC meetings, precisely because people were worried about compatibility and legacy issues. But I don't think the issue should be reopened and redecided differently. The only thing worse than a poor decision by a standards committee is waffling about decisions by a standards committee.

And in this case, I don't really see why you cannot keep on doing what you are currently doing for the UTF-8 terminal emulations. If you document how U+00AD behaves in those emulations, and you should be fine.

And after that. Michel Suignard pointed out an issue that had been overlooked by some, which was 10646 stepping up!

Note that unusally, the latest text from ISO 10646 both in the
10646-1:2000 2nd amendment and the consolidated version capture
verbosely the latest view on this as follows:

SOFT HYPHEN (00AD): SOFT HYPHEN (SHY) is a format character that indicates a preferred intra-word linebreak opportunity. If the line is broken at that point, then whatever mechanism is appropriate for intra-word line-breaks should be invoked, just as if the line break had been triggered by another mechanism, such as a dictionary lookup. Depending on the language and the word, that may produce different visible results, such as:

inserting a graphic symbol indicating the hyphenation and breaking the line after it,
inserting a graphic symbol indicating the hyphenation, breaking the line after the symbol and changing spelling in the divided word parts,
not showing any visible change and simply breaking the line at that point.

The inserted graphic symbol, if any, can take a wide variety of shapes, such as HYPHEN (2010), ARMENIAN HYPHEN (058A), MONGOLIAN TODO SOFT HYPHEN (1806), as appropriate for the situation. When encoding text that includes explicit line breaking opportunities, including actual hyphenations, characters such as HYPHEN, ARMENIAN
HYPHEN, and MONGOLIAN TODO SOFT HYPHEN may be used, depending on the language.

When a SOFT HYPHEN is used to represent a possible hyphenation point, the character representation is that of the text sequence without hyphenation (for example: "tug<00AD>gumi"). When encoding text that includes hard line breaks, including actual hyphenations, the character representation of the text sequence must reflect the changes due to hyphenation (for example: "tugg<2010>" / "gumi").

This was discussed at length during the UTC and WG2.

And Kent Karlsson also pointed out some facts that had been ignored by Markus and others who were stating their opinions:

That text is unfortunately too easy to misread (and overinterpret!). Having talked to one of the authors, he was very surprised at the Kuhn/Korpela interpretation. I think it is a case of being too close to a text to have seen how it could be misread by others. Kuhn's interpretation was definitely not intended (and very few interpret it that way).

The intent of that text, that you partially quoted, is that SOFT HYPHEN is graphic (and imaged) WHEN an (automatic) line break has been made (while it is otherwise invisible, which was not clearly stated). Unicode, SC2/WG2, SC2/WG3, IBM, MS, Adobe, and many others agree on that. Whether it is visible just before an explicit line break (e.g. an LF), is still not clearly stated (though in practice it is, by the already deployed software that does suppport SHY).

What IS new with Unicode 4.0 is that **when imaged** the SOFT HYPHEN may take any suitable hyphen shape (to be nitpicking, it best not to see the SOFT HYPHEN as ever being imaged (except in a "show invisibles" mode, it is just a hyphenation point indication, and the hyphen being imaged when there is a line break is not the actual SHY character). E.g., in Mongolian texts, it should be imaged as a MONGOLIAN TODO SOFT HYPHEN (the "soft" in that name has been decided to be a mistake). In an Armenian text, SOFT HYPHEN *when imaged* takes on the shape of an ARMENIAN HYPHEN.

Some may also remember my Not all GetUnicodeCategory methods are created equal post, which clearly notes the fact that there was some managed code that was depending on the old categorization of SOFT HYPHEN....

All in all, it makes for a fascinating story. :-)

This post brought to you by U+00ad, a.k.a. SOFT HYPHEN

Every character has a story #22: U+0c27 (CARON)

Michael S. Kaplan — Mon, 14 Aug 2006 10:21:00 GMT

The CARON has a long and unhappy history, one that is tied up with that whole Sk/Lm general category thing I talked about in this post.

Ken Whistler laid it out for the CARON just recently, starting with the meandering path through UnicodeData.txt:

UnicodeData-1.1.5.txt:

02C7;CARON;Lm;0;L;0020 030C;;;;N;MODIFIER LETTER HACEK;Mandarin Chinese third
tone;;;

UnicodeData-2.0.14.txt:

02C7;CARON;Sk;0;L;;;;;N;MODIFIER LETTER HACEK;Mandarin Chinese third tone;;;

UnicodeData-4.0.0d1.txt:

02C7;CARON;Sk;0;ON;;;;;N;MODIFIER LETTER HACEK;Mandarin Chinese third tone;;;

UnicodeData-4.0.0d2.txt:

02C7;CARON;Lm;0;ON;;;;;N;MODIFIER LETTER HACEK;Mandarin Chinese third tone;;;

So: gc=Lm --> Sk --> Lm

The difference for U+02C7 being that Unicode 1.1 mistakenly indicated that it was a spacing clone of a diacritic (by that 0020 030C decomposition), which was corrected in Unicode 2.0.

The name anomaly comes from the fact that U+02C7 was mapped to the 8859-2 0xB7 CARON, and the 10646 merger and SC2 rules made us change the name from MODIFIER LETTER HACEK as a result.

But the intent of U+02C6 and U+02C7 as encoded characters in Unicode, since the days of Unicode 1.0, has always been completely parallel -- which is why their General Category history is equally sorry.

It becomes a great example of making things look less stable than they [usually] are. But as I said in this post, the difference between the two general categories is that one is meant to be usable in identifiers, and the other is not. As Ken pointed out:

This whole problem with modifier letters and gc=Sk versus gc=Lm is like that proverbial pebble in the shoe, I'm afraid. Every few years it becomes a "problem" to sort out again, and ends up with a few more characters jiggered one way or another across that boundary.

He then talked about the history of Sk:

For those who care about the history here, gc=Sk didn't exist at all in the original set of General Category values invented by Mark. All the characters *named* MODIFIER LETTER WHATEVER
in UnicodeData-1.1.5.txt got the gc=Lm value.

Mark introduced gc=Sk in Unicode 2.0 to solve a different problem, which was the UTC then groping towards an identifier syntax that would do the right thing for Unicode strings based on Unicode character properties. See Section 5.14 Identifiers, pp. 5-25 to 5-27 in Unicode 2.0 if you can find a copy. In Unicode 2.0, identifiers were constructed on the [alphabetic] property, plus a number of additions and exceptions. But [alphabetic] itself, whose values were printed in the book, by the way, at pp. 4-14 to 4-15, claimed to include "modifier letters". That was problematical, and some of the modifier letters that clearly didn't look like they belonged in identifiers, were
drained from [alphabetic] by inventing the new General Category Sk (symbol modifier), so they got classed with the other symbols, rather than getting lumped with the letters and such under [alphabetic].

Incidentally, the only place in the Unicode 2.0 standard, other than

http://www.unicode.org/Public/2.0-Update/ReadMe-2.0.14.txt

where General Category values are explicitly enumerated was in the discussion of locating text element boundaries (Section 5.13), where the addition of gc=Sk got overlooked and was not yet properly accounted for. The boundary specification assumed that "MODIFIER LETTERS" were, well, modifier letters, and the description even explicitly says:

Lm = Modifier Letter (includes spacing versions of non-spacing marks)

So that part of the 2.0 standard was inconsistent with the changes that had been made to deal with identifier syntax.

It was Unicode 3.0 that revised the identifier syntax to make the classes specifically be based on General Category values, rather than [alphabetic] with exceptions. And there were a significant number of General Category changes which were driven by this. See Appendix D of Unicode 3.0, which notes the then-significant issue of trying to establish convergence between the Unicode definition of identifiers and the ISO TR 10176 definition of identifiers, which was being bandied about at that point as essential for formal programming languages. Page 979 of TUS 4.0:

General Category. A series of General Category changes were made to assist the convergence of the Unicode definition of identifier with ISO TR 10176.

Post Unicode 3.0 was when Mark staked out more territory in character properties, took over PropList.txt and started producing derived properties, using sets of tools for checking consistency, and introducing more properties of the Other_XYZ type to enable more robust derivation rules. The period between Unicode 3.0 and Unicode 4.0 saw all kinds of jiggering of General Category values that resulted from this, including the long list of proposed changes in L2/02-267.

Most of the revisions that resulted were undoubtedly improvements, but in the area of "MODIFIER LETTERS" things have just gotten more confused, in my opinion. In part this has resulted from an essential disconnect between the people proposing new characters for new scripts and additions of miscellaneous abstruse and oddball stuff for Latin, and the people maintaining and extending the Unicode Character Database.

And then at the end of this description came the most amusing summary:

To make this perhaps too pointedly ad hominem, but nevertheless fairly accurate, Michael Everson does not fully understand character properties or their interactions as demonstrated by Mark Davis' manifold property tools, and Mark Davis does not fully understand the functioning of modifier letters in newly encoded scripts and the numerous technical extensions for the Latin script. This tends to leave both of them, and the UTC as well, scratching their heads over the "Is it Lm? Or is it Sk?" decision that inevitably has to be made for all of these additions.

I guess you could say that not only does every character have a story; the truth is that some of them inspire monologues! :-)

This post brought to you by ˇ (U+02c7, a.k.a. CARON, f.k.a. MODIFIER LETTER HACEK)

Every character has a story #21: U+0f77 U+0f79 (TIBETAN VOWEL SIGN VOCALIC [RR|LL])

Michael S. Kaplan — Wed, 28 Jun 2006 10:01:00 GMT

Peter Constable asked some Unicode folks:

I’m just curious to know why 0f77 and 0f79 were given compatibility decompositions rather than canonical decompositions? (I don’t see any obvious reason why canonical decompositions would not have been feasible.)

(Yes, I know this can’t be changed – that’s not my objective.)

Peter Constable

And Ken Whistler stepped up with a good historical look at these two characters (which in my humble opinion deserves a more permanent location for others to see!):

0F71;TIBETAN VOWEL SIGN AA;Mn;129;NSM;;;;;N;;;;;

0F76;TIBETAN VOWEL SIGN VOCALIC R;Mn;0;NSM;0FB2 0F80;;;;N;;;;;
0F77;TIBETAN VOWEL SIGN VOCALIC RR;Mn;0;NSM;<compat> 0FB2 0F81;;;;N;;;;;
0F78;TIBETAN VOWEL SIGN VOCALIC L;Mn;0;NSM;0FB3 0F80;;;;N;;;;;
0F79;TIBETAN VOWEL SIGN VOCALIC LL;Mn;0;NSM;<compat> 0FB3 0F81;;;;N;;;;;
0F80;TIBETAN VOWEL SIGN REVERSED I;Mn;130;NSM;;;;;N;;;;;
0F81;TIBETAN VOWEL SIGN REVERSED II;Mn;0;NSM;0F71 0F80;;;;N;;;;;

0FB2;TIBETAN SUBJOINED LETTER RA;Mn;0;NSM;;;;;N;;*;;;
0FB3;TIBETAN SUBJOINED LETTER LA;Mn;0;NSM;;;;;N;;;;;

                NFD             NFC
0F76            0FB2 0F80       0FB2 0F80
0F77            0F77            0F77            <-- discouraged (strongly)
0FB2 0F71 0F80 0FB2 0F71 0F80 0FB2 0F71 0F80 <-- preferred
0F78            0FB3 0F80       0FB3 0F80
0F79            0F79            0F79            <-- discouraged (strongly)
0FB3 0F71 0F80 0FB3 0F71 0F80 0FB3 0F71 0F80 <-- preferred
0F80            0F80            0F80
0F81            0F71 0F80       0F71 0F80       <-- discouraged
0F71 0F80       0F71 0F80       0F71 0F80       <-- preferred

Note that the preferred forms appear in both NFD and NFC, with the decomposed form for 0F81 resulting from the non-starter exclusion and the decomposed forms for 0F76 and 0F78 resulting from explicit addition to the script-specific composition exclusions.

If you gave 0F77 and 0F79 *canonical* decompositions, then:

0F77 --> <0FB2, 0F81> --> <0FB2, 0F71, 0F80>
0         0     0          0    129   130

0F79 --> <0FB3, 0F81> --> <0FB3, 0F71, 0F80>
0         0     0          0    129   130

                NFD             NFC
0F76            0FB2 0F80       0FB2 0F80
0F77            0FB2 0F71 0F80 ????            <-- discouraged (strongly)
0FB2 0F71 0F80 0FB2 0F71 0F80 0FB2 0F71 0F80 <-- preferred
0F78            0FB3 0F80       0FB3 0F80
0F79            0FB3 0F71 0F80 ????            <-- discouraged (strongly)
0FB3 0F71 0F80 0FB3 0F71 0F80 0FB3 0F71 0F80 <-- preferred
0F80            0F80            0F80
0F81            0F71 0F80       0F71 0F80       <-- discouraged
0F71 0F80       0F71 0F80       0F71 0F80       <-- preferred

Now you've made your life more difficult and normalization implementations maybe more complex. The decompositions <0FB2, 0F71, 0F80> have to be prevented from recomposing. They won't decompose partwise, because <0F71, 0F80> is blocked from recomposing, and <0FB2, 0F80> is also blocked from recomposing, but the sequence of 3 has, at least in principle, a target it should recompose to, unless blocked. Depending on how you set up your tables, you might or might not get this right, and in any case, you end up introducing the strongly discouraged characters as a source of valid sequences that you have to contend with in NFC and NFD, whereas under the current scheme you don't.

Also, this was all part of a very head-breaking set of problems for Tibetan when decompositions and canonical combining classes were being reviewed for the introduction of normalization in the first place.

In Unicode 2.0, 0F77 and 0F79 *were* given canonical decompositions, but they were *different* decompositions, to wit:

0F77 = 0F76 + 0F71 = 0FB2 + 0F80 + 0F71
0F79 = 0F78 + 0F71 = 0FB3 + 0F80 + 0F71

*and* they had funky fixed position class assignments, as well:

0F77 = 0F76 + 0F71 = 0FB2 + 0F80 + 0F71 (not in canonical order)
135    134    129     6     143    129

0F79 = 0F78 + 0F71 = 0FB3 + 0F80 + 0F71 (not in canonical order)
137    136    129     6     143    129

That was clearly hosed, as it broke all kinds of rules that we were trying to establish for normalization, including ensuring that all decomposition mappings produced sequences in canonical order and ensuring, as much as was possible, given the constraints in place, that the resulting sequences would follow the logic of the script *and* that NFC forms would decompose if that was what the users of the script preferred (hence the introduction of script-specific composition exclusions for several scripts, including Tibetan).

During that conversion from Unicode 2.0 to Unicode 3.0 with normalization, the UTC did the best it could with the mess for Tibetan. It was clear after the analysis that 0F77 and 0F79 should never have been encoded at all -- which was why they got those "strongly discouraged" labels -- but there was nothing to do about that mistake at that point. The compatibility decompositions
were the best compromise to keep them from contaminating the normalization processing of the rest of the Tibetan vowels.

--Ken

Anyway, like I said, this seemed to me like good historical information to put out there, and certainly to help show that Every character has a story!

This post brought to you by ཷ and ཹ (U+0f77 and U+0f79, a.k.a. TIBETAN VOWEL SIGN VOCALIC RR and TIBETAN VOWEL SIGN VOCALIC LL)

Every character has a story #20: U+210e (PLANCK CONSTANT)

Michael S. Kaplan — Fri, 21 Apr 2006 10:00:00 GMT

Unicode is a standard that fits all kinds of different needs.

Obviously the need to represent text that may contain important symbols used in science has always been important.

At some fundamental level, the identity of that symbol goes beyond an italicized small letter 'h', and therefore the need to represent Planck's constant even in plain text was a real one.

Thus, it was encoded at U+210e. And has been since at least Unicode 1.1 (and I believe 1.0).

There is a nice description of the constant itself up on Wikipedia. It looks something like this:

ℎ

For fun you can even italicize it? :-)

ℎ

Recently, Andreas Prilop asked on the Unicode List:

The symbol for Planck's constant (6.626E-34 J·s) is an italic "h". Why is there a special Unicode character U+210E for it?

The symbol for the elementary charge is an italic "e". But of course there is no special Unicode character for it.

The symbol for the speed of light is an italic "c". But of course there is no special Unicode character for it.

The symbol for the fine structure constant is an italic "alpha". But of course there is no special Unicode character for it.

etc. ad inf.

So what's this U+210E for? IMHO, this character should be listed as deprecated.

Deborah Goldsmith of Apple was first to point out the answers to these questions:

To differentiate it for purposes of representing mathematics in plain text....

....Note that there is not a MATHEMATICAL ITALIC SMALL H precisely because U+210E exists.

And she is right -- if you look at the Unicode charts at the Mathematical Alphanumeric Symbols block, there is a reserved space at U+1d455. The reason for these spaces is described in UTR#25: Unicode in Mathematics right after Table 2.1:

The plain letters have been unified with the existing characters in the Basic Latin and Greek blocks. There are 24 double-struck, italic, Fraktur and script characters that already exist in the Letterlike Symbols block (U+2100—U+214F). These are explicitly unified with the characters in this block and corresponding holes have been left in the mathematical alphabets.

Kind of says it all, doesn't it? :-)

This post brought to you by "ℎ" (U+210e, a.k.a. PLANCK CONSTANT)

Every character has a story; some of them have cartoons!

Michael S. Kaplan — Fri, 14 Apr 2006 20:20:00 GMT

Adam Hill contacted me via that contact link to point out Hacklog (Blogamundo). With a subtitle like "Poking holes in the language barrier since approximately one month from now" I guess it is saying something interesting!

A quick perusal shows lots of interesting topics -- it even linked to me after the Channel 9 interview....

Anyway, Adam specifically gave me a heads up about a potential regular feature:

Unicode comic strips, the End Days are nigh.

Pat promises/threatens to do more :)

Well, since they all have stories, and some may now have cartoons, I have to wonder how long before we get the first graphic novel?

Blogamundo seems to me like a quite worthy addition to the blogs I read, in any case. :-)

This post brought to you by "⚇" (U+2687, a.k.a. WHITE CIRCLE WITH TWO DOTS)