Every character has a story #19: U+200c and U+200d (ZERO WIDTH [NON] JOINER)

The subtle difference between ශ්රී ලංකාව and ශ්‍රීලංකාව

Sorting It All Out — Sun, 14 Oct 2007 18:59:37 GMT

(The two words might look alike if you don't have the latest and greatest set to render on your machine!)

Why don't all the half forms sort right?

Sorting It All Out — Mon, 25 Sep 2006 11:47:15 GMT

George asked via the Contacting Me... link:

I tried to use the Unicode method of creating half...

re: Every character has a story #19: U+200c and U+200d (ZERO WIDTH [NON] JOINER)

UdaraG — Tue, 09 May 2006 11:11:12 GMT

Thanks very much, Michael!

re: Every character has a story #19: U+200c and U+200d (ZERO WIDTH [NON] JOINER)

Michael S. Kaplan — Mon, 08 May 2006 17:20:41 GMT

Well, the shaping engine support will not yet be there in the CE Uniscribe, so there really is no way to make it all happen just yet.

I do not know the timeline for updates on the mobile platform, but if I find out, I'll post about it....

re: Every character has a story #19: U+200c and U+200d (ZERO WIDTH [NON] JOINER)

UdaraG — Mon, 08 May 2006 14:42:45 GMT

Hi Michael;
Thanks for the prompt reply, and extremely sorry for the much delayed follow up!

Yes, I do understand the licensing concerns very well - myself being a software developer... - the only reason I modified the Tahoma font was since I had a small problem getting the font linking to work. :-)

Now everything is perfect, expect for the ligatures - of which Sinhala language has a number of. I have tried so many devices and WinCE OS versions, including the very latest, but to no avail!

My conclusion is what you guessed right at the start - "updates for Sinhala that were added so recently are NOT in the CE version of Uniscribe".

1. Is there a way to ascertain this from anybody at MS, so that I need not worry about this any further?

2. From the ref-link you provided above, looks like i have no option short of moulding out an OS image (with WinCE Platform Builder) with Sinhala language support.
What may be the next steps in building support for a custom language (Sinhala) with WinCE PB?
Assuming I work fulltime on it ,may be even burning some midnight oil :-), what's the effort/time estimate look like? I do have extensive C/C++ experience with me, and have already downloaded the evaluation copy of WinCE PB (my employer is hoping to buy it soon).
However, this is my first stint with WinCE PB, and am expecting a considerable learning curve.

Thanks and regards,
Udara

re: Every character has a story #19: U+200c and U+200d (ZERO WIDTH [NON] JOINER)

Michael S. Kaplan — Fri, 21 Apr 2006 16:05:23 GMT

Hi Udara,

I see a few problems with what you have mentioned here:

"I imported the Sinhala character range from a Unicode-compatible font (called "Malithi Web") into the standard MS Tahoma font"

Of course the licensing concern is a very real one -- I do not know whether it is legal to modify Tahoma in such a way?

Beyond that, there is the problem of making sure all of the opentype tables are moved over properly. It is not enough to be 'Unicode-compatible' as Arial in Win95 is that; what is important is the right OT table info.

And beyond that, see http://blogs.msdn.com/michkap/archive/2005/05/19/420145.aspx which points out that Uniscribe is not present in every WinCE install, even if it is of the latest version.

Of course beyond all that would be the question of whether the updates for Sinhala that were added so recently are in the CE version of Uniscribe. They may not have the XPSP2/ELK/Vista updates just yet....

re: Every character has a story #19: U+200c and U+200d (ZERO WIDTH [NON] JOINER)

UdaraG — Fri, 21 Apr 2006 11:24:03 GMT

Hi Michael;

Firstly, I am a Sri Lankan who uses Sinhala as my mother tongue, and was quite fascinated to see your use of the same in the example above!

Secondly, I am at the early stages of developing a Sinhala IME for the PocketPC, and am having problems in sending the ZWJ along with leading/trailing characters so that it could be correctly interpreted and rendered on, say Pocket Word.

1. I imported the Sinhala character range from a Unicode-compatible font (called "Malithi Web") into the standard MS Tahoma font, which I have copied over to the PPC.
2. With a rudimentary IME developed in eVC 4.0, I did a IMCallback.SendString() for the Unicode string {0x0DC1, 0x0DCA, 0x200D, 0x0DBB, 0x0DD3, NULL}.

Though this should give me (as I understood) the first form in your example above, it actually gives me the second non-ligated form with a verical bar (which I presume to be the ZWJ) drawn in between the two characters!

What am I doing wrong?
Should I use IMCallback.SendCharEvents() instead?
Can you please point me to some code samples?

Thanks and regards,
Udara

What do you get when you combine a base character with a buttload of diacritics?

Sorting It All Out — Fri, 17 Feb 2006 16:51:53 GMT

The other day I was looking at a particular bug repro (it was actually that BACKSPACE vs. DELETE bug...

re: Every character has a story #19: U+200c and U+200d (ZERO WIDTH [NON] JOINER)

Michael S. Kaplan — Thu, 16 Feb 2006 17:17:27 GMT

Well, take a look at http://blogs.msdn.com/michkap/archive/2006/02/16/533226.aspx which talks about this issue a bit -- definitely more complicated than disallowing a few characters! :-)

re: Every character has a story #19: U+200c and U+200d (ZERO WIDTH [NON] JOINER)

Yosuke HASEGAWA — Thu, 16 Feb 2006 08:56:17 GMT

Using ZERO WIDTH (NON) JOINER or ZWNBSP(BOM) to filename or registory key and values, you can create several files that appearance is the same name.

This may cause visual problems in security domain. So I hope to disable using Unicode control characters for filename in Windows.
Of course, Bidi control characters such as "RLO" too.