Search Form
Syndication
Administrivia

You can't ignore diacritics when a language does not give them diacritic weight

Sorting it all Out
Michael Kaplan's random stuff of dubious value
Be sure to read the disclaimer here first!
MSDN Blogs > Sorting it all Out > You can't ignore diacritics when a language does not give them diacritic weight

You can't ignore diacritics when a language does not give them diacritic weight

5 Feb 2005 5:55 AM
  • Comments 14

This post is about a not entirely intuitive fact that will be seen in the implementation of collation in Microsoft products. It affects the results of both CompareString and LCMapString in Windows, the results of using the CompareInfo and Sortkey classes in the .NET Framework, and in the results in products like Jet and SQL Server.

To help show what is happening under the covers, I will use the sort keys.

We'll use the letters A (U+0041, LATIN CAPITAL LETTER A) and Ą (U+0104, a.k.a. LATIN CAPITAL LETTER A WITH OGONEK), as well as their lowercase counterparts.

When getting sort keys using the default table (LOCALE_INVARIANT), the weights look like the following:

a    U+0061    0E 02 01 01 02 01 01 00
A    U+0041    0E 02 01 01 12 01 01 00
ą    U+0105    0E 02 01 1B 01 02 01 01 00
Ą    U+0104    0E 02 01 1B 01 12 01 01 00

Note the Unicode weights (in blue), the diacritic weights (in green) and the case weights (in red). Now when we ignore case:

a    U+0061    0E 02 01 01 02 01 01 00
A    U+0041    0E 02 01 01 02 01 01 00
ą    U+0105    0E 02 01 1B 01 02 01 01 00
Ą    U+0104    0E 02 01 1B 01 02 01 01 00

And when we ignore diacritics:

a    U+0061    0E 02 01 01 02 01 01 00
A    U+0041    0E 02 01 01 12 01 01 00
ą    U+0105    0E 02 01 01 02 01 01 00
Ą    U+0104    0E 02 01 01 12 01 01 00

And then we ignore both:

a    U+0061    0E 02 01 01 02 01 01 00
A    U+0041    0E 02 01 01 02 01 01 00
ą    U+0105    0E 02 01 01 02 01 01 00
Ą    U+0104    0E 02 01 01 02 01 01 00

Clearly, in the default table LATIN CAPITAL LETTER A WITH OGONEK is little more than a LATIN CAPITAL LETTER A with a hook in it's foot. A small diacritic weight is added to show that it is still primarily a LATIN CAPITAL LETTER A. And the act of ignoring the diacritic gives identical results to when the diacritic was never there in the first place -- you can see it right in the weights.

Now, how about when we move to Polish, LCID 0x00000415? In Polish, LATIN CAPITAL LETTER A WITH OGONEK is a letter with a unique Unicode weight, and this causes a difference in the results:

a    U+0061    0E 02 01 01 02 01 01 00
A    U+0041    0E 02 01 01 12 01 01 00
ą    U+0105    0E 04 01 01 02 01 01 00
Ą    U+0104    0E 04 01 01 12 01 01 00

Do you see what happened here? Since in Polish LATIN CAPITAL LETTER A WITH OGONEK has a unique Unicode weight, ignoring the case weight has a predictable effect:

a    U+0061    0E 02 01 01 02 01 01 00
A    U+0041    0E 02 01 01 02 01 01 00
ą    U+0105    0E 04 01 01 02 01 01 00
Ą    U+0104    0E 04 01 01 02 01 01 00

And Ignoring the diacritic weight will have no effect whatsoever (since there is no diacritic weight to ignore):

a    U+0061    0E 02 01 01 02 01 01 00
A    U+0041    0E 02 01 01 12 01 01 00
ą    U+0105    0E 04 01 01 02 01 01 00
Ą    U+0104    0E 04 01 01 12 01 01 00

So the net effect is that for Polish, passing a NORM_IGNORENONSPACE flag in Windows, a CompareOptions.IgnoreNonspace in the .NET Framework, or a collation in SQL Server such as Polish_CI_AI (Polish, case insensitive, accent insensitive) will never see LATIN CAPITAL LETTER A WITH OGONEK as a LATIN CAPITAL LETTER A. Because Polish does not give the letter diacritic weight.

This is a common issue, whether you look at å (U+00e5, a.k.a. LATIN SMALL LETTER A WITH RING ABOVE) in Swedish, Č (U+010c, a.k.a. LATIN CAPITAL LETTER C WITH CARON) in Slovenian, or any of the other hundreds of examples that exist in supported collations. The key is that in each case you must consider not only whether the character appears to have a diacritic on them but how the language is looking at the string....

This post brought to you by "Ą" (U+0104, a.k.a. LATIN CAPITAL LETTER A WITH OGONEK)

,
Comment on the blather
Leave a Comment
  • Please add 7 and 2 and type the answer here:
  • Post
Blog - Comment List
Comments
  • CN
    5 Feb 2005 7:40 AM
    Are there any "diacritic" characters that are treated like they were separate letters in the default table?
  • RolfBjarne
    5 Feb 2005 8:10 AM
    Even though this might seem a little bit strange, it is indeed correct, for even though the letters "a" and "å" look very similar, in Norwegian (and Swedish) they are two different letters, just like "a" and "b" in English.
  • Michael Kaplan
    5 Feb 2005 10:43 AM
    CN -- no, the default table stays pretty clean of that sort of thing. It is something for specific languages with different conventions.

    RolfBjarne -- exactly. But it is amazing how often it is considered a bug when a developer runs across the issue, especially in other products like SQL Server or Outlook.
  • CN
    5 Feb 2005 1:04 PM
    Do they ever encounter that "Vindovs" is equal to "Windows" while using the same flag? To get back to the original question, that is one case where the default table will consider characters as different, while a few locales think that they are identical/variations of each other, and within the set of ordinary latin letters.

    BTW, I know this is not the place to mention it -- but the signature listed for CompareString in MSDN Library is wrong! When I first looked it upp I started wondering if it only took atom strings or something else that could be mapped to a DWORD easier than a pointer. Thankfully (in this case), the parameter names were Hungarian...
  • Michael Kaplan
    5 Feb 2005 1:09 PM
    Well, not true ever in the default table, but in some languages the V and the W are to be treated the same. See Finnish and Swedish, for example: http://www.microsoft.com/globaldev/dis_v1/disv1.asp?DID=dis33d&File=S24C3.asp

    The CompareString doc bug was reported by someone here, before. See http://blogs.msdn.com/michkap/archive/2005/02/02/365251.aspx#366109 . :-)
  • CN
    5 Feb 2005 2:37 PM
    I know it isn't true in the default table, but I meant that there is a case where the default table makes a distinction, while the locale tables do not. Your examples with the accented characters were the other way around, and I found it interesting that they are in both ways.

    Oooops, and so recent... I guess I read that posting while it was fresh and then never got back to it. Hmhm...
  • Michael Kaplan
    5 Feb 2005 4:41 PM
    Ah, I see what you are saying -- yes, you are right. Things really do go in both directions.
  • Sorting It All Out
    19 Feb 2005 2:55 PM
  • Sorting It All Out
    12 Oct 2006 3:39 AM

    I do manage to get a lot of random email (between my two main accounts, several hundred a day, not including

  • Sorting It All Out
    25 Feb 2007 4:26 AM

    The mail I got read: Hi I'm Laurent Gébeau , FRench MVP on Windows. I met you at the last MVP summit

  • Sorting It All Out
    26 Feb 2007 4:05 AM

    Thinking about the issues involved with à ≠ a (unless à = a) made me think back to other posts where

  • Sorting It All Out
    11 Apr 2007 11:49 AM

    Sometimes, in order get the best results in collation, one has to use constructs that from a linguistic

  • Sorting It All Out
    14 Sep 2007 3:31 AM

    Previous posts in this series: Part 0: The empty string sorts the same in every language Part 1: The

  • Hitesh Patel
    23 Nov 2012 4:02 AM

    Hello Sir,

    I have a problem my question when i fetch the data in may class and set in text view in android not proper character display it.and ask question in stack overflow but anyone proper answer suggest me .please help me sir.and My Question link below:

    stackoverflow.com/.../how-can-i-display-latin-words-in-android

    Advance Thanks!!  

Page 1 of 1 (14 items)
MSDN Blogs > Sorting it all Out > You can't ignore diacritics when a language does not give them diacritic weight