UCS-2 to UTF-16, Part 10: Variation[ Selector] on a theme...

Whither WM_UNICHAR in Windows 7 (and 8!)

Sorting it all Out — Mon, 21 May 2012 14:04:34 GMT

The WM_UNICHAR message has had an interesting history.

Over time it has come up in this Blog occasionally

Should considering UTF-16 be harmful be considered harmful?

Sorting it all Out — Fri, 27 Apr 2012 14:03:42 GMT

Like many of the people I know, I find myself looking over at Stack Overflow and related sites periodically

re: UCS-2 to UTF-16, Part 10: Variation[ Selector] on a theme...

Michael S. Kaplan — Wed, 24 Jun 2009 02:45:45 GMT

Not all clipboard cases grab surrounding spaces and most do not if you do not select the space.

The Bidi marks are another case, one I talk about tomorrow....

re: UCS-2 to UTF-16, Part 10: Variation[ Selector] on a theme...

John Cowan — Wed, 24 Jun 2009 00:57:57 GMT

Cut/paste isn't quite the same thing as selection: in some contexts, when we cut selected text, following whitespace also goes with it. In that case, I'd probably extend the selection to include any following invisible character (what is currently done for bidi marks in that situation?)

In addition to Andrew's reasons, another reason why VSes shouldn't be used for simplified characters is that N traditional characters often map to the same simplified character, so it would be impossible to tell by looking at a text what underlies it. There are other situations where this is true, like:

english = CIBARA

(which word do you read first?), but that's no reason to add to it.

re: UCS-2 to UTF-16, Part 10: Variation[ Selector] on a theme...

Andrew West — Fri, 12 Jun 2009 12:25:20 GMT

"new simplified characters should be traditional characters with variation selectors"

I know that their are certain people in the UTC who favour this approach, but I don't think it has any chance of being adopted, because there would be massive opposition from IRG and WG2. At this stage in the game it is simply too late to change the model for Han encoding; and all we can do is bite the bullet and encode all required simplified character forms as separate characters (about half of the CJK-D characters currently under ballot are simplified forms of existing traditional characters).

re: UCS-2 to UTF-16, Part 10: Variation[ Selector] on a theme...

Andrew West — Fri, 12 Jun 2009 12:17:39 GMT

I think I'm too close to variation selectors to be able to answer your questions objectively. I like to have full control over my text, including variation selectors, but I can see that many users would be annoyed if the appearance of their text kept getting screwed up because of some invisible entity that was not staying in its place.

re: UCS-2 to UTF-16, Part 10: Variation[ Selector] on a theme...

Michael S. Kaplan — Thu, 11 Jun 2009 19:20:39 GMT

As a by the way, this is one of the many reasons I was against the VS notion the entire time it has been alive.

Does the answer to any above change with the argument (of some, and which I also disagree with) that new simplified characters should be traditional characters with variation selectors?

re: UCS-2 to UTF-16, Part 10: Variation[ Selector] on a theme...

Michael S. Kaplan — Thu, 11 Jun 2009 19:18:24 GMT

Okay, differences in opinion are good, always. :-)

But I am curious, especially of John and Andrew but really of everyone: in the myriad of cases, what is the best result? Imagine (in addition to the specific scenarios John and Andrew mentioned) the following:

-- Delete a section, included an alternate glyph at the end -- Should the VS be left in the old text, and does the answer change when the VS is alone at the beginning vs. attached to a new (possibly illegal or potentially worse legal but unintended choice of) base character?

-- Copy/Paste -- All of the above, plus: -- should the alternate glyph be lost?

You see what I am getting at -- what do you think should happen with the orphaned VS, which is now just in the text, unintended? I am worried about two principles, 1) what to do with the original intent of the text, and 2) what to do with the potentially illegal, potentially unintended result of the remaining text....

re: UCS-2 to UTF-16, Part 10: Variation[ Selector] on a theme...

Michael S. Kaplan — Thu, 11 Jun 2009 17:10:32 GMT

Okay, an opinion not brought up with Unicode (or at least not written in for its own recommendations...).

But let's take the cut/paste case as an example -- you'd really want the VS to be left dangling alone with no base character, even though the user selected the "changed" character?

Or cursor movement, when display works right. You really want the cursor weirdness that would manifest itself, with no other visual indication of what's going on?

re: UCS-2 to UTF-16, Part 10: Variation[ Selector] on a theme...

Andrew West — Thu, 11 Jun 2009 16:01:32 GMT

Actually, the result is more likely to look weird with the VS; and if stripped of the VS the result would generally look normal. But, anyway I agree with John that I would not consider the VS and its base character as an inseperable entity like a surrogate pair, and personally like to be able to treat VS's as characters when editing text -- deleting them and trying new different VS's on different characters to get the desired result.