SIAO to Search engines -- would you please normalize, already?

re: SIAO to Search engines -- would you please normalize, already?

Andrew West — Tue, 15 Nov 2005 13:01:06 GMT

"Andrew West's claim that Google is normalizing appears to be crap"

NOT ME! "Andrew C." is someone else ... I don't mnake crap claims ;)

... and anyway he claimed that Google was *not* normalizing.

re: SIAO to Search engines -- would you please normalize, already?

Michael S. Kaplan — Tue, 15 Nov 2005 14:17:32 GMT

Sincerest aplogies, Andrew -- I have removed the bogus reference to you. It was Simon who was trying to make the claim (I am not sure who Simon is here).

Correction made, and again I am very sorry.

re: SIAO to Search engines -- would you please normalize, already?

Suzanne McCarthy — Wed, 16 Nov 2005 03:02:52 GMT

Isn't Simon's claim abaout Greek correct? Normalization is happening for some sequnces but not others. I get the same results in French and German with and without the precomposed 'vowels plus diacritics'.

re: SIAO to Search engines -- would you please normalize, already?

Suzanne McCarthy — Wed, 16 Nov 2005 03:09:01 GMT

BTW I forgot to add that I really appreciate this little experiment that you created here. Thank you, Mike.

re: SIAO to Search engines -- would you please normalize, already?

Michael S. Kaplan — Wed, 16 Nov 2005 03:30:10 GMT

If I had to guess, I would say they are not using Unicode normalization at all -- they are building their own homegrown system that happens to work in some cases but not others....

re: SIAO to Search engines -- would you please normalize, already?

Michael S. Kaplan — Wed, 16 Nov 2005 04:19:26 GMT

You are very welcome, Suzanne. It was fun. :-)

re: SIAO to Search engines -- would you please normalize, already?

Jim — Thu, 17 Nov 2005 13:44:40 GMT

Ahem, a microsoft employee rubbishing google through speculation...

re: SIAO to Search engines -- would you please normalize, already?

Michael S. Kaplan — Thu, 17 Nov 2005 16:31:15 GMT

Hi Jim,

This is not rubbish -- tet yourself. Canonically equivalent Unicode forms do not find the same pages.

If I were trying to diss just Google I would not have pointed out that Microsoft is also guilty though, obviously. Both of them need to do this.

re: SIAO to Search engines -- would you please normalize, already?

Mike — Fri, 18 Nov 2005 03:46:03 GMT

Very good info - hopefully this will be a big wakeup call for the big 2. I hope those that have the power to do something about this have been "educated" now.

What is equal to some may not be equal to others

Sorting It All Out — Mon, 06 Feb 2006 11:29:41 GMT

Someone going by the handle AC asked me via email:

You have mentioned that Google has trouble with...

Harder intermediate forms of characters

Sorting It All Out — Sun, 14 May 2006 12:32:33 GMT

In the post Getting intermediate forms, I gave an example three character sequences that look the same...

The search for someone who does Search correctly

Sorting It All Out — Mon, 26 Feb 2007 12:05:38 GMT

Thinking about the issues involved with à ≠ a (unless à = a) made me think back to other posts where

SIAO is still underwhelmed by search engines (all of them)

Sorting It All Out — Mon, 16 Jul 2007 10:16:01 GMT

You may have read in Arial Unicode MS effectively [bites|sucks|blows] about how Microsoft MVP Omi Azad