SIAO to Search engines -- would you please normalize, already?

Suzanne has been riffing on me in relation to Vietnamese and then she shifted over to talk about Google and other languages, so I thought I would riff off of her a bit. :-)

By the way Suzanne -- I did not find your terminology to be inaccurate; it was just different. I was explaining why I was confused!

I am going to take her string bãi biển from that first article and run it through various search engines in various forms, trying to look for some patterns. I am not using Suzanne's test (using Google Image) looking for pictures of beaches) since not all of the search engines have a comparable service. it is pretty easy to tell from the text excerpts if the search has found Vietnamese sights or not without too much trouble....

First, the engines:

1. google.com
2. search.msn.com
3. start.com
4. live.com
5. altavista.com
6. ask.com
7. excite.com
8. yahoo.com

Second, the strings to test:

1. Normalization Form C: bãi biển (0062 00e3      0069 0020 0062 0069 1ec3           006e)
2. Intermediate form   : bãi biển (0062 0061 0303 0069 0020 0062 0069 00ea 0309      006e)
3.  " "  " w/o accents : bai biên (0062 0061      0069 0020 0062 0069 00ea           006e)
4. Normalization Form D: bãi biển (0062 0061 0303 0069 0020 0062 0069 0065 0302 0309 006e)
5.  " "  " w/o accents : bai bien (0062 0061      0069 0020 0062 0069 0065           006e)

Now if you look at these five strings being tested #1, #2, and #4 are all canonically equivalent and thus should give identical results in search engines that conform to Unicode and its principles of canonical equivalence.

I will put the strings in double quotes for all search engines.

And here are the comparison results:

Conclusions:

• Excite (using the default settings) does not understand Unicode from UNICEF, and the only reason it had 42 hits from Form C was that it munged that second word into "bi" and linked to inappropriate sights that I would never really be interested in looking at, even if they were in Vietnamese. This is (by the way) why the link is not live. :-)
• Excite (using advanced search, which defaults to all languages) fared a little better, with Suzanne's post high up on the list for #2 and #5.
• Lycos (which I did not put in the table) returned reults that were practically identical to ask.com.
• That one link on ask.com and lycos.com for #3 is to Suzanne's post, which I think is pretty funny. :-)
• Both altavista and yahoo appear to be using Unicode normalization, returning identical results for all canonically equivalent forms.
• Google appears to be stripping combining characters out of either its seasrch strings, its indexes, or both.
• Any claim that Google is normalizing appears to be crap -- at least insofar as one considers Unicode normalization. They are doing their own thing rather than the standard. But then they are only Associate members of Unicode, so I guess they aren't in all the way just yet....
• Microsoft (msn, start, and live) have a whole bunch of work to do and I am having trouble fathoming what precisely they are doing.
• Not enough of the Search community is taking the important of canonical equivalence seriously, to the detriment of many language communites, including Vietnamese.
• Until that time, a better keyboard solution for Vietnamesse in particular suddenly seems more and more compelling.

This post brought to you by "ổ" (U+1ed5, a.k.a. LATIN SMALL LETTER O WITH CIRCUMFLEX AND HOOK ABOVE)