CharNext(ch) != ch+1, a lot of the time

Earlier today, Raymond Chen sent me a piece of email that mentioned an important point for developers who iterate a string one character at a time. Its a lot more interesting than what I was going to post so I'll do the boring one later or tomorrow (or if I am smart I'll just give it a miss entirely), and I'll do this one now, instead.

It has to do with the CharNext and CharPrev APIs

The email had a title similar to the one of this posting. Basically he was pointing out that CharNextW(p) != p+1. And he is right, they are not equal, a lot of the time.

The same issue that applies to CharNext(ch) not being the same as ch+1 applies to CharPrev(ch) and ch-1. Put more verbosely, this is because incrementing and decrementing a character pointer within a string is not nearly as functional as these APIs can be.

The reasons for this are many and as I said they apply to both CharPrev and CharNext (and incidentally also CharNextExA and CharPrevExA). They include:

• When not dealing with Unicode, strings on CJK (Chinese, Japanese, and Korean) systems have many characters that take up two bytes (including all Han/Kanji/Hanja, all Hangul and all full-width non-Kanji). Using simple byte increments will mean one is moving through half a character with each iteration.
  
• When dealing with Unicode, strings that use combining characters like å (U+0061 U+030a, a plus combing ring) or ü (U+0075 U+0308, u plus combining umlaut) take up two code points, so once again one will be dealing with moving through half a character with each iteration.
  
• Stretching to languages like Vietnamese where double diacritics are commonly seen in characters such as ộ (U+006f U+0323 U+0302, o plus combining dot below plus combining circumflex) take up three code points, one is dealing with only a third of a character with each iteration!
  
• When one uses Unicode supplementary characters such as U+21532 (an Extension B ideograph) where a Unicode string on Windows will actually be represented by a surrogate pair (U+d845 U+dd32) one will actually once again be dealing with half a character.

I crossed that last item off of the list because CharNext and CharPrev do not currently handle surrogate pairs properly, so one will be just as bad off using the APIs as one would be just doing simple pointer arithmetic.

(Never fear, I will be looking into seeing if I can do something about this for the future now that someone reminded me!)¹

How does it do this work? Well, CharNextA/CharPrevA use the IsDBCSLeadByte API on appropriate platforms to determine if the byte is a lead byte in a lead byte/trail byte pair, and CharNextW/CharPrevW use the GetStringTypeW API to figure out if a character is a non-spacing character like a ring or a diaresis.

Other APIs that do an even better job can be found in Uniscribe. The ScriptBreak API will return an array of SCRIPT_LOGATTR structs, and that structure's fCharStop member, when set, indicates that this is a valid place for a cursor to jump to. When such a cursor jump is valid, it indicates that you have moved forward by one "character" in the sense that a user might think of a character. And therefore if you use Uniscribe you will be handle this job properly, even for supplementary characters.

Unicribe is a useful enough library that I will talk more about it in a future post, maybe even some sample code for easier operations (like this one). Uniscribe is behind most of the support of international text on Windows.

It was ported to Windows CE 5.0 as well, though it is described in documentation about the CE Platform Builder, which implies (to me) that it might only be included on SKUs that are built with it for shipment to places that require it. Those who know more about Windows CE and Uniscribe should feel free to contact me with more info so I can sound more intelligent the next time I talk about it!

This is a much easier task in managed code, whether you can use the StringInfo class and its GetTextElementEnumerator and GetTextElement methods, which allow for easy iteration of a string. You can also use its ParseCombiningCharacters method to get an array of integers representing the same character boundaries represented by Uniscribe's SCRIPT_LOGATTR.fCharStop member.

For those of you who are still awakereading, I will point out one annoying issue I discovered while typing in the å, ü, and ộ characters earlier in this post. Problems? Well...

1. The .Text blog system's editor to type did not handle this properly and moved past 1/2 or 1/3 of the character with each arrow keypress.
   
2. When the character was being selected it either selected the whole character and then appeared to do nothing or appeared to do nothing and then selected the whole character, depending on whether I moved from left-to-right or right-to-left with the cursor.
   
3. The deletion behavior was just as dismal. I don't even want to talk about it. 

I also saw the same behavior in plain old EDIT text boxes in Internet Explorer, so it looks like .Text is off the hook. And that even though IE does use Uniscribe, they forgot to implement some of the possible features that library provides². :-( 

If someone wanted to try this in other browsers (cough! FireFox or Opera), I'd be curious about the results; the IE results were pretty disappointing to me. Try selecting the string "åüộåüộåüộåüộ" and pasting into their browser wherever you see an EDIT box to test, or just try putting thr cursor into and running the cursor through the INPUT control text right here:

and let me know what you see (unless you see 20 other people have  already responded!). Be sure to mention the browser and version (I am using IE 6.0.3790.0 on Windows Server 2003 with all of the latest updates).

(All that talk about Uniscribe reminds me I want to talk about MLang at some point too -- I've added MLang and Uniscribe to my list of things to talk about!)

1 - It first started when I was working on MSLU, but even to this day I have not gotten fully used to being able to say stuff like this. I'll try to write more about it another time, cuz it is kind of cool.
2 - It works fine in Notepad and Word and VS.Net, all of which use Uniscribe.

This post brought to you by "å" (U+00e5, a.k.a. LATIN SMALL LETTER A WITH RING ABOVE)
A character that is downright snooty about the fact that it has none of the problems mentioned in this article.
Despite the fact that a circle almost bigger than its body is super-glued to its head, something that would make me feel at least a little self-conscious.