Comparison confusion: INVARIANT vs. ORDINAL

re: Comparison confusion: INVARIANT vs. ORDINAL

Panos Theofanopoulos — Thu, 30 Dec 2004 16:29:00 GMT

for 1000000 comparisons

Ordinal : 47 mseconds
InvariantCulture : 299 mseconds

OrdinalIgnoreCase : 188 mseconds
InvariantCultureIgnoreCase : 245 mseconds

This post brought to you by "Ώ" (U+038F, GREEK CAPITAL LETTER OMEGA WITH TONOS) which beats Ω, ω and ώ in a binary sort :-)

re: Comparison confusion: INVARIANT vs. ORDINAL

Michael Kaplan — Thu, 30 Dec 2004 16:36:00 GMT

Heh heh heh -- see what I mean?

You do have to be careful with the perf testing in an actual app situation to use representative strings. The reason is that in both cases (ordinal *and* invariant) the code will exit as soon as it finds the winner.

In order (therefore) to get the most effective test you have to craft the strings to match the kind of comparisons you will see in your app.

I'll get more into how it works internally in a future post. :-)

re: Comparison confusion: INVARIANT vs. ORDINAL

Michael Kaplan — Thu, 30 Dec 2004 16:48:00 GMT

BTW -- Panos, in answer to your suggestion to the MSDN Feedback Center:

http://lab.msdn.microsoft.com/ProductFeedback/viewfeedback.aspx?feedbackid=2682c87e-5c46-4697-bbed-6f0de0047a7d

True benchmarks for these methods under load, with indications of where the true slowdowns are, is the way to proceed here. IMHO, all perf. work has to be done that way....

re: Comparison confusion: INVARIANT vs. ORDINAL

Norman Diamond — Wed, 05 Jan 2005 01:10:00 GMT

> Note that these string comparisons still had
> much linguistic value -- half of the locales
> in Windows use that default table, so an
> invariant sort would not only avoid varying,
> it would also look right to a lot of the
> world.

Isn't linguistic value counterproductive for the intended value of the invariant locale? Didn't you say in an earlier posting that people shouldn't use the invariant locale for human-oriented operations? Surely the way to discourage people from using the invariant locale for sorting is to make the results appear wrong for every known culture in the world?

re: Comparison confusion: INVARIANT vs. ORDINAL

Michael Kaplan — Wed, 05 Jan 2005 01:20:00 GMT

> Isn't linguistic value counterproductive for the intended value of the invariant locale?

No, not at all. I have explained the exact purpose of the invariant locale, and that is NOT it.

> you say in an earlier posting that people shouldn't use the invariant locale for human-oriented operations?

No, I said that about ORDINAL comparisons.

> Surely the way to discourage people from using the invariant locale for sorting is to make the results appear wrong for every known culture in the world?

Well, no one is trying to discourage invariant sorting. So I have to reject the premise here.

re: Comparison confusion: INVARIANT vs. ORDINAL

Norman Diamond — Wed, 05 Jan 2005 06:51:00 GMT

In page
http://weblogs.asp.net/michkap/archive/2004/12/08/278170.aspx
you said:
> The invariant locale is pretty weird. Lets
> take a look at its interesting
> chracteristics.
[...]
> Its data was chosen to dissuade people from
> trying to use it as a locale. And that is
> putting it charitably.

That did not say ordinal comparisons, that said the invariant locale. Please look at your own posting an un-reject the premise here.

Now, to help dissaude people from trying to use the invariant locale as a locale, would it not have been beneficial to avoid giving it linguistic values? Would it not have been better if sort results were wrong for every human language?

re: Comparison confusion: INVARIANT vs. ORDINAL

Michael Kaplan — Wed, 05 Jan 2005 08:01:00 GMT

It was intended to be used for sorting, we want it used for sorting. This was the REASON it was added. It will not change, and we would not want it to change even if we had the option.

When I spoke of trying to make it less appealing as a locale, I am referring to EVERYTHING ELSE beyond collation.

I hope these words are plain enough.

re: Comparison confusion: INVARIANT vs. ORDINAL

Norman Diamond — Thu, 06 Jan 2005 00:38:00 GMT

Yes it is clearer this time.

Though I still think it would be better if the invariant locale's sort ordering differed from every human language so that no one would be tempted to use it for human-oriented sorting. For internal ordering of databases and finding hash keys and running scripts in a uniform manner it would still be fine (which I thought its intended purpose was).

re: Comparison confusion: INVARIANT vs. ORDINAL

Michael Kaplan — Thu, 06 Jan 2005 01:03:00 GMT

Norman, I understand you think this.

Do you understand that we do not and that we intend it to be used and we have no ingtention of telling people not to use it?

It is basically the same as saying "use English" or whatever -- the default table. Thats what its job is, and it does that well.

Its not fair to change the requirement based on what you think it ought to do, and thus make it nor perform well. :-(

re: Comparison confusion: INVARIANT vs. ORDINAL

Norman Diamond — Thu, 06 Jan 2005 07:05:00 GMT

Then you should call it the English locale and not the invariant locale. (Unless you really meant the en-US locale.) Attaching a non-provincial label for the purpose of appearing non-provincial while still making the factual behavior every bit as provincial as your other thread said you were trying to avoid, just yields another bit of pretense with continued provincialism. Also since facts of the invariant locale look like en-US, it will no longer be surprsing when US developers overuse it and use it for undesirable purposes.

re: Comparison confusion: INVARIANT vs. ORDINAL

Michael Kaplan — Thu, 06 Jan 2005 09:08:00 GMT

Norman, you are wrong once again. Over half othe languages there are also in the defualt table. I can say use Hebrew. Or German. Or any number of locales instead. Thinking it is provincial to take the word "English" out of it is plain silly.

No if you look at the REST of the data, it looks nothing like English. Or Arabic. Or German. Or Hebrew. Or any of the other 50 locales that use the table. So go ahead and try to claim it is a provincial attempt to push a particular culture if you must -- but those who are reading will have long seen who gets it.

What we do here with invariant has a valid and explainable purpose.

re: Comparison confusion: INVARIANT vs. ORDINAL

Michael Kaplan — Fri, 07 Jan 2005 01:13:00 GMT

Further comments that find to be inappopriate the intended purpose and use of the invariant locale's collation behavior (to use the DEFAULT collation table used by half of the locales in Windows) will not be accepted.

Those with dissenting opinions on the topic are invited to create their own blogs or web pages where they can explain the amount of evil-ness (evility?) in Microsoft's implementaion choices.

re: Comparison confusion: INVARIANT vs. ORDINAL

Norman Diamond — Sat, 08 Jan 2005 03:41:00 GMT

1/6/2005 1:08 AM Michael Kaplan

> Thinking it is provincial to take the
> word "English" out of it is plain silly.

Of course it is. That's why I didn't use the word provincial that way. I used the word provincial in the way you used it in page
http://weblogs.asp.net/michkap/archive/2004/12/08/278170.aspx
and I agree with the way you used it there.

re: Comparison confusion: INVARIANT vs. ORDINAL

Michael Kaplan — Sat, 08 Jan 2005 03:56:00 GMT

Yes, but did you read the whole article?

"...since the only real goal anyway was to "use the default table" for sorting..."

Our point is therefore entirely clear here for all to see (and read).

Microsoft Windows and the .NET Framework work as designed in regard to the INVARIANT locale/culture. The only bug we have now is the inappropriate expectations of one Norman Diamond. :-)

Invariant and Ordinal Redux

Sorting It All Out — Wed, 13 Apr 2005 14:58:04 GMT

I have talked about LOCALE_INVARIANT / CultureInfo.InvariantCulture before, in Comparison confusion:...

Intelligent unmanaged string comparison

Sorting It All Out — Tue, 26 Apr 2005 13:32:41 GMT

If you look at the documentation for CompareString&nbsp;(but not LCMapString, though it probably ought...

Why do NTFS and Explorer disagree on filename sorting?

The Old New Thing — Fri, 17 Jun 2005 18:17:14 GMT

Because they have different goals.

Invariant vs. Ordinal, the third

Sorting It All Out — Wed, 24 May 2006 14:41:32 GMT

Mike Sheldon is one of the smartest guys I know.
You may have seen him on Channel 9 talking about Windows...

It has not always been so invariant

Sorting It All Out — Mon, 28 Aug 2006 03:51:32 GMT

The other day, Bill asked the following question:

Anyway, I have the following...

When methods use collation to 'disturb the peace' we charge them with being 'out of sorts'

Sorting It All Out — Tue, 10 Apr 2007 15:05:56 GMT

You know how I talk about best practices here sometimes? And the worst ways to misuse various globalization/internationalization

The nature of OrdinalIgnoreCase vs. intuitive expectations

Sorting It All Out — Wed, 25 Apr 2007 18:45:07 GMT

A while back Patrick asked me: Hi Michael, Please forward if you’re not the right person to ask the question…

The exception that proves the rule that was the exception that proves another rule (aka On the variability of the Invariant)

Sorting It All Out — Sat, 12 May 2007 14:57:24 GMT

It was over two years ago that Shawn talked about how Culture data shouldn't be considered stable (except

re: Comparison confusion: INVARIANT vs. ORDINAL

Phil Hackett — Tue, 02 Sep 2008 19:00:08 GMT

Thanks Michael - that article was -really- useful. It's a shame the official Microsoft help/documentation doesn't explain this, really, as it's such an obvious question that people are bound to ask.