November, 2008

Once again, a blog for your reading pleasure that is technical though not on the usual subjects. If iBOT crap bores you, then please skip gracefully!

Prior blogs in the series here and here and here and here and here.

It was actually in a response to that last one that regular reader and long-time friend and colleague Tony Toews asked:

So I gotta ask a basic question.   Do the batteries last all day when you're spending large parts of the day in balance mode on two wheels?

Now of course we know from past experience that it is the basic questions that can have some of the most complicated answers!

Now at one level it is simple --they did manage to last that long while I was at PASS.

And also simple at a more technical level, as the provided documentation for the iBOT describes it all.

Of course the usefulness of that text:

When tested in conformance with ISO 7176-4 (1997), the theoretical distance range while driving steadily on flat, level ground exceeds 15.5mi (22km) range in Standard, 4-Wheel, and Balance Functions.

is somewhat obviated by the disclaimers of the following sentences:

This data should only be used for comparison purposes with other power wheelchairs. The actual distance you can travel will depend on total weight carried, weather, surface conditions, driving style, other power usage (e.g., transitions between functions), and climbing stairs.

and really it kind of tends to ignore the actual question Tony is asking -- which is a very reasonable question to have.

Now ISO 7176-4 (Wheelchairs -- Part 4: Energy consumption of electric wheelchairs and scooters for determination of theoretical distance range) is useful as far as those things go, though it ignores some crucial information such as the vastly different characteristics of the Ni-Cad batteries of the iBOT from the Lead Acid Gel batteries seen in most power wheelchairs -- differences that will really impact actual usage. even 7176-4 says as much on its introduction:

Distance range is also strongly dependent on the way in which a wheelchair is driven, and a single value for theoretical range can be insufficient to provide an understanding of the performance of a wheelchair. Two methods for determining theoretical range are provided in this part of ISO 7176, for driving and for manoeuvring. These values are intended to facilitate wheelchair comparison in a manner analogous to the extra-urban and urban fuel consumption figures published for motor vehicles.

Funny, that's what I was about to say this was as useful as. :-)

So why is it that a document that claims that various modes (Standard, 4-Wheel, and Balance) all will allow for analogous ranges?

Well, technically it doesn't -- it just says they can all go THAT far, wihout claiming any one can go farther. Even if it can.

Is there truly no power cost to the more heavy-duty 4-wheel mode even when it is on flat ground, analogous to the way that a car in 4-wheel drive mode needs more gas than one on 2-wheel drive mode?

Maybe -- because as observers have noted, the wheels do not work independently; all four always turn even if two are off the ground.

But is the power cost of the iBALANCE functionality (used in Balance mode and to a lesser but still present extent in 4-wheel mode) truly so negligible that the three modes are considered to be the same in tems of power cost?

This just seems wrong....

Every taximeter I have ever seen while sitting in a taxi charges for both the driving time and the waiting time by some unknown (to some) formula that one could clearly use to one's advantage if one wanted to charge too much. It would be easy to think the iBOT above such petty nickel-and-diming as the taximeter and such criminal designs as the one who would actively subvert one to make more money.

But I have seen my battery meter decrease while I was just standing in balance mode in a presentation or at the bus stop waiting for the 545 to Seattle.

So I know that such a thought is baseless.

Even more than I "know" that the whole Ni-Cad memory effect is probably a crock since it hasn't really been proven and I am the sort of person who thinks that anything people "know but can't bother to prove" is more likely to be a crock than not. More on my opinions here in a blog post some other day....

Because this one, I know through direct observation, through admittedly suspect measurements.

I trust the battery meter on my iBOT like I have never trusted a battery meter before -- because this is the first time it ever really mattered that the gauge looked accurate enough that it could give me useful information.

And that meter showed some cost to iBALANCE in the iBOT's Balance mode.

Not a lot. But some....

Could the decreased cost of some other factor cancel out the increased cost of the iBALANCE stuff, making all three modes really have the same range? Hmmm...

This is, by the way, the kind of thing that irks me some. I mean, all of the testing and work they do and they can't just provide some data on average power usage of iBALANCE that could be compared as amount of time one could have instead been driving if the human dignity thing wasn't as important as the distance?

Perhaps the next time I am reading a book I'll get in the iBOT in Balance mode and read until the battery drains two ticks on the battery meter, then move to 4-wheel mode and read until the battery drains two ticks on the battery meter.

If either takes an unconscionably long time, I'll apologize for my the part of my irk not related to the fact that they have almost certainly has done this kind of testing but never bothered to publish the results, and be happy with the original non-information they provided. And if either time is significantly short enough, I'll be at full irk mode and try to look into a fuller, more formal amount of data to be provided....

Until then, the question is still pending, Tony. Stay tuned. :-)

This post brought to you by ♿ (U+267f, a.k.a. WHEELCHAIR SYMBOL)

I was asked the other day what I thought about string comparisons. What with all the different recommendations floating around, all of the different possibilities, and all the unclear guidelines, the question was whether there was a succinct way to describe the best way to know what to do?

Obviously this is an area that does not inspire intuitive thought. That train has already left the station, but not before the conductor sneered at us for presuming we would have seats on it.

But is there a way to build in something to pinch hit for intuitive behavior?

I think there is. :-)

Let me try it on for size here....

There are four points to keep in mind, which I will describe now.

The second point is another test -- one has to decide how one would feel if characters that look kind of the same even though they aren't were to be sorted together -- either as being identical or being right next to each other. If the answer is Yes, Michael, I think it would be great if stuff that looks th same got treated the same then this is a linguistic comparison, rather than a binary/ordinal/lexicographic type one.

The third point is yet another test -- one has to decide how one feels about characters that have no weight in sorting, either because they are unassigned anywhere, not yet assigned in the tables, or intentionally are given no weight. If the answer is Yes, Michael, if no one thought the character should be given weight then I don't see why I would want to then this is once again a linguistic comparison, rather than a binary/ordinal/lexicographic type one.

The fourth point is something to keep in mind -- and that is the fact that one should know exactly what the comparison is for, and especially that the same data might be sorted different ways in different situations. Only by knowing the scenario can one make a correct decision.

Perhaps examples would help here. :-)

Think of the first test as the Turkic I test -- whether one is okay with the way one looks at

I İ ı i

U+0049 U+0130 U+0131 U+0069

and the fact that sometimes the first and fourth items there are the casing pair, and other times it is the first/third and second/fourth that are. You can imagine plugging those strings right into your code and deciding how you feel about the results.

Think of the second test as the Small Capital Y test -- whether one is okay with the following three characters

Y y ʏ

U+0059 U+0079 U+028f

being right next to each other and sometimes even considered equal to each other. You can once again imagine pluging those strings right into your code and deciding how you feel about the results.

And you can think of the third test as the IE Grave test (also known as the "Cyrillic E" test by the stubborn) -- whether one is okay with letters like the following:

Ѐ ѐ

U+0400 U+0450

being treated as if they weren't there at all, because some versions of Microsoft products will do just that. You can once again imagine plugging those strings right into your code and deciding how you feel about the results.

That fourth point is the easiest of all. Just imagine components like NTFS (the file system) or the registry. At the lowest levels they happen to use an ordinal/binary kind of collation, while at the most user visible levels (e.g. Windows Explorer, RegEdit) they use a linguistic one. The rule is that circumstances alter cases, so that even the same list can sort differently in different situations and that is okay.

So, three simple mental tests one can do and immediately come up with an answer on what comparison method to use....

Easy!

This blog brought to you, as you likely could have guessed, by the above nine characters

So it all started in a conversation with some of the folks from the SQL Server team when I was at PASS.

What did they expect? Send 600 Microsoft employees to an event and you are bound to run into some of them just in the act of walking around!

They were telling me about an interesting bug in a couple collations in SQL Server 2008 -- bugs that only impact non-Unicode columns for those couple of collations.

Apparently they got a couple code page values wrong.

This was "Azeri Serious Bug".

Once I was back in front of my computer, I ran the following query:

SELECT
  name,
  COLLATIONPROPERTY(name, 'CodePage') As CodePage,
  CONVERT(binary(4), COLLATIONPROPERTY(name, 'LCID')) As LCID,
  CONVERT(binary(4), COLLATIONPROPERTY(name, 'ComparisonStyle')) As ComparisonStyle,
  description
FROM
  ::fn_helpcollations()
WHERE
  name LIKE '%Azer%' OR name LIKE '%Turki%'

in SQL Server 2005:

Azeri_Cyrillic_90_BIN    1251    0x0000082C    0x00000000    Azeri-Cyrillic-90, binary sort
Azeri_Cyrillic_90_BIN2    1251    0x0000082C    0x00000000    Azeri-Cyrillic-90, binary code point comparison sort
Azeri_Cyrillic_90_CI_AI    1251    0x0000082C    0x00030003    Azeri-Cyrillic-90, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Azeri_Cyrillic_90_CI_AI_WS    1251    0x0000082C    0x00010003    Azeri-Cyrillic-90, case-insensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Azeri_Cyrillic_90_CI_AI_KS    1251    0x0000082C    0x00020003    Azeri-Cyrillic-90, case-insensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Azeri_Cyrillic_90_CI_AI_KS_WS    1251    0x0000082C    0x00000003    Azeri-Cyrillic-90, case-insensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Azeri_Cyrillic_90_CI_AS    1251    0x0000082C    0x00030001    Azeri-Cyrillic-90, case-insensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Azeri_Cyrillic_90_CI_AS_WS    1251    0x0000082C    0x00010001    Azeri-Cyrillic-90, case-insensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Azeri_Cyrillic_90_CI_AS_KS    1251    0x0000082C    0x00020001    Azeri-Cyrillic-90, case-insensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Azeri_Cyrillic_90_CI_AS_KS_WS    1251    0x0000082C    0x00000001    Azeri-Cyrillic-90, case-insensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Azeri_Cyrillic_90_CS_AI    1251    0x0000082C    0x00030002    Azeri-Cyrillic-90, case-sensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Azeri_Cyrillic_90_CS_AI_WS    1251    0x0000082C    0x00010002    Azeri-Cyrillic-90, case-sensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Azeri_Cyrillic_90_CS_AI_KS    1251    0x0000082C    0x00020002    Azeri-Cyrillic-90, case-sensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Azeri_Cyrillic_90_CS_AI_KS_WS    1251    0x0000082C    0x00000002    Azeri-Cyrillic-90, case-sensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Azeri_Cyrillic_90_CS_AS    1251    0x0000082C    0x00030000    Azeri-Cyrillic-90, case-sensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Azeri_Cyrillic_90_CS_AS_WS    1251    0x0000082C    0x00010000    Azeri-Cyrillic-90, case-sensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Azeri_Cyrillic_90_CS_AS_KS    1251    0x0000082C    0x00020000    Azeri-Cyrillic-90, case-sensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Azeri_Cyrillic_90_CS_AS_KS_WS    1251    0x0000082C    0x00000000    Azeri-Cyrillic-90, case-sensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Azeri_Latin_90_BIN    1254    0x0000042C    0x00000000    Azeri-Latin-90, binary sort
Azeri_Latin_90_BIN2    1254    0x0000042C    0x00000000    Azeri-Latin-90, binary code point comparison sort
Azeri_Latin_90_CI_AI    1254    0x0000042C    0x00030003    Azeri-Latin-90, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Azeri_Latin_90_CI_AI_WS    1254    0x0000042C    0x00010003    Azeri-Latin-90, case-insensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Azeri_Latin_90_CI_AI_KS    1254    0x0000042C    0x00020003    Azeri-Latin-90, case-insensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Azeri_Latin_90_CI_AI_KS_WS    1254    0x0000042C    0x00000003    Azeri-Latin-90, case-insensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Azeri_Latin_90_CI_AS    1254    0x0000042C    0x00030001    Azeri-Latin-90, case-insensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Azeri_Latin_90_CI_AS_WS    1254    0x0000042C    0x00010001    Azeri-Latin-90, case-insensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Azeri_Latin_90_CI_AS_KS    1254    0x0000042C    0x00020001    Azeri-Latin-90, case-insensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Azeri_Latin_90_CI_AS_KS_WS    1254    0x0000042C    0x00000001    Azeri-Latin-90, case-insensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Azeri_Latin_90_CS_AI    1254    0x0000042C    0x00030002    Azeri-Latin-90, case-sensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Azeri_Latin_90_CS_AI_WS    1254    0x0000042C    0x00010002    Azeri-Latin-90, case-sensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Azeri_Latin_90_CS_AI_KS    1254    0x0000042C    0x00020002    Azeri-Latin-90, case-sensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Azeri_Latin_90_CS_AI_KS_WS    1254    0x0000042C    0x00000002    Azeri-Latin-90, case-sensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Azeri_Latin_90_CS_AS    1254    0x0000042C    0x00030000    Azeri-Latin-90, case-sensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Azeri_Latin_90_CS_AS_WS    1254    0x0000042C    0x00010000    Azeri-Latin-90, case-sensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Azeri_Latin_90_CS_AS_KS    1254    0x0000042C    0x00020000    Azeri-Latin-90, case-sensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Azeri_Latin_90_CS_AS_KS_WS    1254    0x0000042C    0x00000000    Azeri-Latin-90, case-sensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Turkish_BIN    1254    0x0000041F    0x00000000    Turkish, binary sort
Turkish_BIN2    1254    0x0000041F    0x00000000    Turkish, binary code point comparison sort
Turkish_CI_AI    1254    0x0000041F    0x00030003    Turkish, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Turkish_CI_AI_WS    1254    0x0000041F    0x00010003    Turkish, case-insensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Turkish_CI_AI_KS    1254    0x0000041F    0x00020003    Turkish, case-insensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Turkish_CI_AI_KS_WS    1254    0x0000041F    0x00000003    Turkish, case-insensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Turkish_CI_AS    1254    0x0000041F    0x00030001    Turkish, case-insensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Turkish_CI_AS_WS    1254    0x0000041F    0x00010001    Turkish, case-insensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Turkish_CI_AS_KS    1254    0x0000041F    0x00020001    Turkish, case-insensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Turkish_CI_AS_KS_WS    1254    0x0000041F    0x00000001    Turkish, case-insensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Turkish_CS_AI    1254    0x0000041F    0x00030002    Turkish, case-sensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Turkish_CS_AI_WS    1254    0x0000041F    0x00010002    Turkish, case-sensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Turkish_CS_AI_KS    1254    0x0000041F    0x00020002    Turkish, case-sensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Turkish_CS_AI_KS_WS    1254    0x0000041F    0x00000002    Turkish, case-sensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Turkish_CS_AS    1254    0x0000041F    0x00030000    Turkish, case-sensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Turkish_CS_AS_WS    1254    0x0000041F    0x00010000    Turkish, case-sensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Turkish_CS_AS_KS    1254    0x0000041F    0x00020000    Turkish, case-sensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Turkish_CS_AS_KS_WS    1254    0x0000041F    0x00000000    Turkish, case-sensitive, accent-sensitive, kanatype-sensitive, width-sensitive

and in SQL Server 2008:

Azeri_Cyrillic_100_BIN    1251    0x0000082C    0x00000000    Azeri-Cyrillic-100, binary sort
Azeri_Cyrillic_100_BIN2    1251    0x0000082C    0x00000000    Azeri-Cyrillic-100, binary code point comparison sort
Azeri_Cyrillic_100_CI_AI    1251    0x0000082C    0x00030003    Azeri-Cyrillic-100, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Azeri_Cyrillic_100_CI_AI_WS    1251    0x0000082C    0x00010003    Azeri-Cyrillic-100, case-insensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Azeri_Cyrillic_100_CI_AI_KS    1251    0x0000082C    0x00020003    Azeri-Cyrillic-100, case-insensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Azeri_Cyrillic_100_CI_AI_KS_WS    1251    0x0000082C    0x00000003    Azeri-Cyrillic-100, case-insensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Azeri_Cyrillic_100_CI_AS    1251    0x0000082C    0x00030001    Azeri-Cyrillic-100, case-insensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Azeri_Cyrillic_100_CI_AS_WS    1251    0x0000082C    0x00010001    Azeri-Cyrillic-100, case-insensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Azeri_Cyrillic_100_CI_AS_KS    1251    0x0000082C    0x00020001    Azeri-Cyrillic-100, case-insensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Azeri_Cyrillic_100_CI_AS_KS_WS    1251    0x0000082C    0x00000001    Azeri-Cyrillic-100, case-insensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Azeri_Cyrillic_100_CS_AI    1251    0x0000082C    0x00030002    Azeri-Cyrillic-100, case-sensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Azeri_Cyrillic_100_CS_AI_WS    1251    0x0000082C    0x00010002    Azeri-Cyrillic-100, case-sensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Azeri_Cyrillic_100_CS_AI_KS    1251    0x0000082C    0x00020002    Azeri-Cyrillic-100, case-sensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Azeri_Cyrillic_100_CS_AI_KS_WS    1251    0x0000082C    0x00000002    Azeri-Cyrillic-100, case-sensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Azeri_Cyrillic_100_CS_AS    1251    0x0000082C    0x00030000    Azeri-Cyrillic-100, case-sensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Azeri_Cyrillic_100_CS_AS_WS    1251    0x0000082C    0x00010000    Azeri-Cyrillic-100, case-sensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Azeri_Cyrillic_100_CS_AS_KS    1251    0x0000082C    0x00020000    Azeri-Cyrillic-100, case-sensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Azeri_Cyrillic_100_CS_AS_KS_WS    1251    0x0000082C    0x00000000    Azeri-Cyrillic-100, case-sensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Azeri_Latin_100_BIN    1254    0x0000042C    0x00000000    Azeri-Latin-100, binary sort
Azeri_Latin_100_BIN2    1254    0x0000042C    0x00000000    Azeri-Latin-100, binary code point comparison sort
Azeri_Latin_100_CI_AI    1254    0x0000042C    0x00030003    Azeri-Latin-100, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Azeri_Latin_100_CI_AI_WS    1254    0x0000042C    0x00010003    Azeri-Latin-100, case-insensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Azeri_Latin_100_CI_AI_KS    1254    0x0000042C    0x00020003    Azeri-Latin-100, case-insensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Azeri_Latin_100_CI_AI_KS_WS    1254    0x0000042C    0x00000003    Azeri-Latin-100, case-insensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Azeri_Latin_100_CI_AS    1254    0x0000042C    0x00030001    Azeri-Latin-100, case-insensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Azeri_Latin_100_CI_AS_WS    1254    0x0000042C    0x00010001    Azeri-Latin-100, case-insensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Azeri_Latin_100_CI_AS_KS    1254    0x0000042C    0x00020001    Azeri-Latin-100, case-insensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Azeri_Latin_100_CI_AS_KS_WS    1254    0x0000042C    0x00000001    Azeri-Latin-100, case-insensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Azeri_Latin_100_CS_AI    1254    0x0000042C    0x00030002    Azeri-Latin-100, case-sensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Azeri_Latin_100_CS_AI_WS    1254    0x0000042C    0x00010002    Azeri-Latin-100, case-sensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Azeri_Latin_100_CS_AI_KS    1254    0x0000042C    0x00020002    Azeri-Latin-100, case-sensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Azeri_Latin_100_CS_AI_KS_WS    1254    0x0000042C    0x00000002    Azeri-Latin-100, case-sensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Azeri_Latin_100_CS_AS    1254    0x0000042C    0x00030000    Azeri-Latin-100, case-sensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Azeri_Latin_100_CS_AS_WS    1254    0x0000042C    0x00010000    Azeri-Latin-100, case-sensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Azeri_Latin_100_CS_AS_KS    1254    0x0000042C    0x00020000    Azeri-Latin-100, case-sensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Azeri_Latin_100_CS_AS_KS_WS    1254    0x0000042C    0x00000000    Azeri-Latin-100, case-sensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Turkish_BIN    1254    0x0000041F    0x00000000    Turkish, binary sort
Turkish_BIN2    1254    0x0000041F    0x00000000    Turkish, binary code point comparison sort
Turkish_CI_AI    1254    0x0000041F    0x00030003    Turkish, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Turkish_CI_AI_WS    1254    0x0000041F    0x00010003    Turkish, case-insensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Turkish_CI_AI_KS    1254    0x0000041F    0x00020003    Turkish, case-insensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Turkish_CI_AI_KS_WS    1254    0x0000041F    0x00000003    Turkish, case-insensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Turkish_CI_AS    1254    0x0000041F    0x00030001    Turkish, case-insensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Turkish_CI_AS_WS    1254    0x0000041F    0x00010001    Turkish, case-insensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Turkish_CI_AS_KS    1254    0x0000041F    0x00020001    Turkish, case-insensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Turkish_CI_AS_KS_WS    1254    0x0000041F    0x00000001    Turkish, case-insensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Turkish_CS_AI    1254    0x0000041F    0x00030002    Turkish, case-sensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Turkish_CS_AI_WS    1254    0x0000041F    0x00010002    Turkish, case-sensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Turkish_CS_AI_KS    1254    0x0000041F    0x00020002    Turkish, case-sensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Turkish_CS_AI_KS_WS    1254    0x0000041F    0x00000002    Turkish, case-sensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Turkish_CS_AS    1254    0x0000041F    0x00030000    Turkish, case-sensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Turkish_CS_AS_WS    1254    0x0000041F    0x00010000    Turkish, case-sensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Turkish_CS_AS_KS    1254    0x0000041F    0x00020000    Turkish, case-sensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Turkish_CS_AS_KS_WS    1254    0x0000041F    0x00000000    Turkish, case-sensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Turkish_100_BIN    1254    0x0000041F    0x00000000    Turkish-100, binary sort
Turkish_100_BIN2    1254    0x0000041F    0x00000000    Turkish-100, binary code point comparison sort
Turkish_100_CI_AI    1254    0x0000041F    0x00030003    Turkish-100, case-insensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Turkish_100_CI_AI_WS    1254    0x0000041F    0x00010003    Turkish-100, case-insensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Turkish_100_CI_AI_KS    1254    0x0000041F    0x00020003    Turkish-100, case-insensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Turkish_100_CI_AI_KS_WS    1254    0x0000041F    0x00000003    Turkish-100, case-insensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Turkish_100_CI_AS    1254    0x0000041F    0x00030001    Turkish-100, case-insensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Turkish_100_CI_AS_WS    1254    0x0000041F    0x00010001    Turkish-100, case-insensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Turkish_100_CI_AS_KS    1254    0x0000041F    0x00020001    Turkish-100, case-insensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Turkish_100_CI_AS_KS_WS    1254    0x0000041F    0x00000001    Turkish-100, case-insensitive, accent-sensitive, kanatype-sensitive, width-sensitive
Turkish_100_CS_AI    1254    0x0000041F    0x00030002    Turkish-100, case-sensitive, accent-insensitive, kanatype-insensitive, width-insensitive
Turkish_100_CS_AI_WS    1254    0x0000041F    0x00010002    Turkish-100, case-sensitive, accent-insensitive, kanatype-insensitive, width-sensitive
Turkish_100_CS_AI_KS    1254    0x0000041F    0x00020002    Turkish-100, case-sensitive, accent-insensitive, kanatype-sensitive, width-insensitive
Turkish_100_CS_AI_KS_WS    1254    0x0000041F    0x00000002    Turkish-100, case-sensitive, accent-insensitive, kanatype-sensitive, width-sensitive
Turkish_100_CS_AS    1254    0x0000041F    0x00030000    Turkish-100, case-sensitive, accent-sensitive, kanatype-insensitive, width-insensitive
Turkish_100_CS_AS_WS    1254    0x0000041F    0x00010000    Turkish-100, case-sensitive, accent-sensitive, kanatype-insensitive, width-sensitive
Turkish_100_CS_AS_KS    1254    0x0000041F    0x00020000    Turkish-100, case-sensitive, accent-sensitive, kanatype-sensitive, width-insensitive
Turkish_100_CS_AS_KS_WS    1254    0x0000041F    0x00000000    Turkish-100, case-sensitive, accent-sensitive, kanatype-sensitive, width-sensitive

but all of the code page values looked right!

Hmmm.

I went back and asked for more information, and then I got the clues I should have seen the first time:

• They called it a regression;
  
• The collations in the two lists were mutually exclusive.

And with this information I stumbled on a fact that I did not realize even though it was right in front of me.

The fn_helpcollations function wasn't returning deprecated collations!

It would have helped for me to look at the SQL Server 2008 version of the topic, which had the following note:

Deprecated collations that are no longer supported or installed in SQL Server 2008 cannot be queried with fn_helpcollations.

Aha!

though the older collation is actually still there; you just can't enumerate it. I was able to use it in T-SQL just fine for several operations.

And then I ran the following query:

SELECT
  Name, 
  COLLATIONPROPERTY(Name, 'CodePage') As CodePage,
  CONVERT(binary(4), COLLATIONPROPERTY(Name, 'LCID')) As LCID
FROM
  (VALUES ('Azeri_Latin_90_BIN'),
          ('Azeri_Latin_100_BIN'),
          ('Azeri_Cyrillic_90_BIN'),
          ('Azeri_Cyrillic_100_BIN'),
          ('Turkish_BIN'),
          ('Turkish_100_BIN')
  ) AS Collations(Name);

which returned the following results (incorrect ones that do not match the SQL Server 2005 values in red):

Azeri_Latin_90_BIN        1251    0x0000042C
Azeri_Latin_100_BIN       1254    0x0000042C
Azeri_Cyrillic_90_BIN     1254    0x0000082C
Azeri_Cyrillic_100_BIN    1251    0x0000082C
Turkish_BIN               1254    0x0000041F
Turkish_100_BIN           1254    0x0000041F

And there's the bug they were talking about.

Well, that and the documentation bug that claims that these deprecated collations aren't installed. I think what they were trying to say was that they were not available in setup to choose as the default collation, and it is just bbeing described in a weird way -- since clearly the old ones are still round (and would have to be or no database using them could be upgraded properly!).

and if you use Unicode columns then this bug won't matter.

But if you use CHAR, VARCHAR, or TEXT columns with this collation, your results will be wrong -- very very wrong....

Let's look at the code pages themselves to see how wrong, and where. First code page 1254:

and then code page 1251:

and you can see that while the Latin Azeri will be missing some characters, the Cyrillic Azeri will be missing pretty much all the ones you need.

Okay, a serious bug -- one that has to be fixed.

The easy fix is to change to the newer Azeri collations, but this can be complicated by anything that tries to map the characters through Unicode first since the data they have on the mappings is wrong here!

Not to mention users who notice the problem and just use the wrong collation that works correctly -- fixing the bug could even break a few people....

Now you may (if you are a regular reader) remember 100% roundtrip ASCII? 100% roundtrip ANSI?, which talks about missing characters in the two code pages.

In this case, code page 1254 is missing seven bytes:

0x81  0x8D  0x8E  0x8F  0x90  0x9D  0x9E

which if they were on the Cyrillic code page would be:

• U+0403 -- Ѓ (aka CYRILLIC CAPITAL LETTER GJE)
  
• U+040c -- Ќ (aka CYRILLIC CAPITAL LETTER KJE)
  
• U+040b -- Ћ (aka CYRILLIC CAPITAL LETTER TSHE)
  
• U+040f -- Џ (aka CYRILLIC CAPITAL LETTER DZHE)
  
• U+0452 -- ђ (aka CYRILLIC SMALL LETTER DJE)
  
• U+045c -- ќ (aka CYRILLIC SMALL LETTER KJE)
  
• U+045b -- ћ (aka CYRILLIC SMALL LETTER TSHE)
  

While code page 1251 is just missing one byte:

0x98

which if they were in the Latin code page would be:

• U+02dc -- ˜ (aka SMALL TILDE)
  

Once again, it is indeed the Cyrillic code page that suffers the most, though in the case of improperly stored data it might have been the easiest to detect being on the wrong code page, if these letters were being used in Cyrillic Azeri (which according to pages like this one they don't really seem to be!).

In the end, this bug becomes just another incident where due to the fact that Unicode is what is being emphasized, bugs in non-Unicode code paths slip through.

This might be another good reson to support Unicode, everyone!

I'll probably talk more about deprecation here, another day. This was an interesting and not-wrong (in my opinion) design direction that deserves some real understanding and elucidation....

This blog brought to you by the eight above mentioned non-Azeri Unicode characters that have been pulled into some degree of Azeriness...

The question yesterday was:

For date formatting and parsing APIs, is there a difference between lower- and upper-case Ys? (I.e., “yyyy” v. “YYYY”.) The articles on MSDN seem to be inconclusive. Errr.

That's an interesting question, I think.

Whether one looks at the Custom Date and Time Format Strings used by things like DateTime.ToString(string) or the strings in GetDateFormat or the strings in GetTimeFormat, it is clear that some characters (like H and M) have specific meaning attached to the cased variants.

In my book, since only the lowercase y is used here, the uppercase behavior is undefined -- if it works then have fun but you really shouldn't; if it fails then you kind of asked for that failure....

And more importantly, if the meaning changes in the future then that is also something requested in the improper use. and the nature of H vs. h and M vs. m kind of underscores this.

In practice, if the stuff works now could it really be changed later if the letter is needed for something else? Hard to say, really.

This is the kind of decision that I really hated when it was on me, though in that position I'd likely say maintain and document the behavior. Although the temptation to spank the violators would always be there, the fact that the actual spanking would hit the customers of the violator rather than the violator and the fact that Microsoft would be blamed would keep that from ever going beyond temptation....

And if I were designing the formatting/parsing tags from scratch, I'd lock them down from the beginning. :-)

Note that this problem is made slightly more complicated by the fact that some of the localized Regional and Language Options tags (also ref:[Localized] Date/Time format tokens) actually change case during the localization process. This would not change code or code callers, but it would confuse understanding in some other languages!

This blog brought to you by Y, y, and ʏ (U+0059 U+0079 U+028f, aka LATIN CAPITAL LETTER Y, LATIN SMALL LETTER Y, and LATIN LETTER SMALL CAPITAL Y)

Previous blogs in this series of blogs on this Blog:

• Part 0: The intro, sans content
• Part 1: Getting the obvious out of the way
• Part 2: A&P of a 'linguistic character'
• Part 3: It starts with cursor movement (where MS simultaneously gets better and worse)
• Part 4: Talking about the ask 
• Part 5: What's on the Next Level?
  

Now continuing on from prior blogs in the series, I thought I'd quote a bit from a recent email thread about a very similar issue to some of the prior discussion like especially Part 3, related to caret stops, aka the points where you can put the cursor as you navigate the string.

The thread had wandered a bit (as threads tend to do!), and then colleague Jerry Dunietz (an architect I have worked with before on issues related to Unicode and cultures and locales and such) offered a great description of many of the issues that I have discussed here in this series. With his permission, I will quote from his response:

Logically, a Unicode string is a sequence of UCS-4 code-points.  (I intend to carefully distinguish between “code-point” and “code-unit” in the text below.)

There are several way to encode a Unicode string in memory.  Ignoring for today the existence of a byte-order-mark (and of byte-ordering variations), there are three ways that one could represent such a string in memory:

As a sequence of 32-bit code-units, each representing a single UCS-4 code-point.  (UTF-32)
As a sequence of 16-bit code-units.  Some UCS-4 code-points are represented by single code-unit, and some represented as a “surrogate pair” of two code-units.  (UTF-16.)
As a sequence of 8-bit code-units.  The code points from U+0000 to U+007F are represented as a single code-unit, but all other code-points are represented by a longer sequence of code-units (UTF-8.)

Now imagine that you’re a programmer, and you want to pick representation in memory for a Unicode string.  UTF-32 seems like the easiest to work with.  (But it is the fattest encoding of the three for any real-world corpus of text.)  If you want the fifth code-point in a string, you just array-index to the fifth code-unit in the string.  If seems (but wait) that if the user hits backspace after inputting a string, you just back up one-code-unit.  Is seems that if a user hits an arrow key to move a caret from one character to the next, you would just advance the caret position by one code-unit. (Given all of this apparent simplicity, there seems to be a compelling argument for defining C++’s wchar_t to be 32-bits long.)

But it turns out that the stuff I wrote using the word “seems” is an over-simplification.  Unicode has code-points that correspond to combining characters.  Such characters combine with a previous code-point (or sequence thereof) to present to the user what appears to be a single character.  Section 2.11 of the Unicode 5.0 spec provides lots of examples of different combining characters, in different languages.  From a font-rendering glyph selection point of view, or from the point of a view of a user attempting to move a text cursor from character to character, a single glyph or character may correspond to a sequence of multiple UCS-4 code-points, and thus multiple UTF-32 code-units.  Given that such situations exists, an internationally-robust program working with a UTF-32 string must be prepared for the concept that multiple-code code-units or code-points correspond to the user’s concept of a single character.

But if our program needs to deal with the possibility of a user’s concept of a character corresponding to multiple code-units, then the apparent advantage to the programmer of using a UTF-32 representation instead of a UTF-16 one goes away. (And given the real-world size advantages of UTF-16, it now seems that making wchar_t be 16 bits is a better choice than making it be 32-bits.)

Whether you choose UTF-16 or UTF-32 encoding, suppose you want to spec a file format that can be easily displayed, but for which a viewer program can easily support text selection.  You could ask the viewer program to build in lots of smarts to determine where a logical caret-stop can occur.  Or you can ask the program that built the file to encode the caret-stop information within the file itself, reducing the burden on the viewing program.

This is really a great summary of the issues surrounding the UTF-16 vs. UTF-32 debate, as well as the whole UCS-2 vs. UTF-16 one I have been covering already. I can't really take full credit for his knowledge in this area since I was just one of several sources, but Iike now I'm in there somewhere and it is always good to know when one has been helpful in influencing an influencer (which indirectly puts me in mind of both the influence vs. impact issue and the fact that Ms. Phair might be wrong about the influence of an Ant in Alaska, sometimes!).

Now as to whether storing the information about the carets stops explicitly versus calculating them via a StringInfo-type technique (StringInfo is something I have talked about previously, in blogs like this one) is an interesting one.

Really in the end it depends on whether you think the pure "derived from Unicode data" answer is sufficient or whether you have additional sources of information. In the particular case Jerry was thinking of, they rely on much more sophisticated methods, such that constantly calculating on the fly could really impact performance. Since the data itself in his case is read-only there was never worry about the potential need for recalculation so the calc-once obnly and store the information was a no-brainer for them.

But even in a read-write situation, providing a sorted array of indexes for easy enumeration that allows for

• enumeration in either direction (for cursor movement and selection operations), and
  
• insertion (for the insertion of text), and
  
• deletion (for the deletion of text), and
• an initial population that will be already sorted (for a string with no data that needs to build up its initial cache)
  

is really just an ordinary interview question in disguise, and one that is pretty easily solved, too. Were it even vaguely "international" I'd say we should solve it here, but it's not so I'll leave that as an exercise for whoever feels that they need the exercise. :-)

Though of course I'll point a few issues to keep in mind for anyone who would want to implement such a cache....

Obviously one has to be ready to re-order the items after the point of change for the insertion/deletion case. this could be expensive, depending on where the action is happening and how mush text follows it (when to move to slightly more complex schemes then the doubly linked list that the initial problem suggests is also left as an exercise!).

But less obviously, in the area immediately preceding and following the insertion point, one can potentially need to recalculate caret stops due to changes in the text. For example if one has the letters "abcdef" then one will have seven indices (covering the points before each caret stop as one moves across the string, plus one for the end):

{0, 1, 2, 3, 4, 5, 6}

Then if one decides to add a combining umlaut after the initial "e", the new string is "abcdëf" and the indices would now have to be:

{0, 1, 2, 3, 4, 6, 7}

and so on. And in other cases a formerly combining character before or after might now not be....

From there the exact methods used for calculation of caret stops and how to integrate them together comes into play, as does how the different methods [might] interact. It really can become a rather fascinating technical question, in the end. Though mostly not for this blog, except more on the various methods eventually. :-)

Getting back to the series for a second, there are a few points left to cover, such as the actual means of support and then the whole UTF-8 question (which is still fair game here!).

I'll cover these in upcoming blogs....

This blog brought to you by ឿ (U+17bf, aka KHMER VOWEL SIGN YA)

Warning: although slightly technical, this blog is mostly non-technical, and/or technical about stuff related to the iBOT. If the technical issues related to SQL Server and/or PASS interest you then they will probably show up in future blogs...

Prior blogs in the series here and here and here and here.

It's kind of funny.

I just spent last week at SQLPASS 2008, and had a really great time. I was there as an "Expert" in the ATE (Ask the Expert) area, and I talked officially about migration, though I got a bunch of upgrade questions since customers don't tend to distinguish that much between migration (moving from any non-SQL Server database to SQL Server) and upgrade (moving from an earlier version of SQL Server to a later one).

Plus I got many questions about Unicode (especially both UTF-16 and UTF-8), collation support in 2000/2005/2008, and also rich text support in SQL Server reporting services (drawing some on my typography knowledge, and my work with the RS team). It was really a lot of great conversations with customers and colleagues and partners and friends.

Did I mention that I had a great time? Well, I'll say it again. I had a great time.

And not just because I got to see friends and colleagues from years prior and find out what they are up to like Debra Dove (now a Group Program Manager!) and Tom Casey (now a General Manager!) and a whole bunch of others too numerous to mention, not to mention all the people I met in SQL Server marketing. I mean that was very cool, but that wasn't it.

And not just because in my heart of hearts I'm still a databases person, just like I was way back almost in the beginning for me when Ashton-Tate's DBase II would spit out the "30 days hath September" rhyme when you gave it an out-of-range date (the first software "Easter Egg" I ever remember finding, running on an Osborne 1!). That was cool too, but that wasn't it.

And also not just because the evening parties and after-parties are superior to any other conference I've been to, though they are (it just seems like SQL folks work hard and know when to play harder, and especially when to not talk about work). This is one of the many reasons I have not been blogging; to be honest I wasn't even sleeping much last week. Every developer should go to a few database conferences like SQLPASS. :-)

The cool part I am referring to isn't that, either.

As you can guess from the blog title, it was really because this was my first technical conference with the iBOT.

I won't say that I was nervous, exactly. Though I admit I took the charger since I did not want to be stuck in Seattle with no way to get home!

As one of the "Elite 300" ATE folks, I had $25,000 in SQL Bucks to give away to people who asked me questions. But I realized early on that iBOT questions couldn't count -- if they did I would have required a bailout from the program organizers since I'd otherwise be broke within half a day!

To be fair there were a lot of people I talked to about SQL Server issues, too (as I mentioned before).

But even some of those conversations started with a few iBOT questions -- people were just interested and curious....

There was even one conversation with some of the WIT (Women in Technology) volunteers about the predictable way that iBOT questions tended to split on gender lines even when they were kind of the same question (e.g. "how does that chair stay upright?" from men versus "how do you keep from falling out?" from women).

I was even prepared now, armed with the answer to the ultimate question, the one John McConnell (yes, the one who talked about When will we support Rongo-Rongo) asked weeks before, at a group mixer.

You see, the most common question people would ask is some variation of "How is that thing balancing?"

My iBOT shtick eventually developed more fully as the week went on, some based on the boilerplate in the iBOT FAQ's How does the iBOT® Mobility System work?:

A revolutionary mobility system from Independence Technology, L.L.C. the iBOT® Mobility System utilizes our patented iBALANCE® Technology. It is custom-programmed and calibrated to the owner’s center of gravity. Reach forward in Balance Function to shake hands, and your iBOT® 4000 Mobility System moves with you. Lean back and it moves with you as well. It is subtle and responsive in a way no other mobility device can match.

but it always eventually made it to the marketing point, about how the iBOT, unlike the Segway, uses six gyroscopes to do its work.

This always impressed people though I suspect that was because mainly they don't really know much about gyroscopes. So there was no follow-up question along those lines.

But John is one of those really smart guys who does know about things like gyroscopes, so he asked the very reasonable next question -- how does it use six gyroscopes?

Now I even do know a bit about gyroscopes, but it had never occurred to me think of that next question, let alone ask it.

To John I had to admit I never asked, but that I would find out.

So I asked the iBOT folks.

Perhaps not entirely surprisingly, the front line of the IBOT's helpdesk didn't know either. :-)

But they tracked down a better answer, which I will give in a moment.

It's funny (minor segue for a moment), over the course of the week at SQLPASS, I was reminded of the Underpants Gnomes from South Park, and their three step business plan:

• Phase 1: Collect underpants
• Phase 2: ????
• Phase 3: Profit!

The thing I noticed over the course of the week was that in the old days there would be some women who would be waving to me, looking at me with interest. They would almost invariably be looking at someone behind me or nearby, rather than at me.

Now with the iBOT, they actually were looking at me. Well, technically they were looking at my ride, my iBOT, but they still wanted to talk about it. But at least they weren't looking at the person behind me, right? :-)

Of course while it clearly appears that this "second phase" represented some form of progress, I really didn't know how to get to the third phase.

In part because I'm not sure what it would be (I suppose having them being interested in me? I'm not sure!), but it still is easy to put it in phases and be curious about how to get to the third phase!

Anyway, that answer.

First, I'll borrow an image from Wikimedia to define Pitch, Roll, and Yaw in much less time than a verbal description ever would:

If you absolutely must have words then the Wikipedia Flight Dynamics article can probably help...

It has those other pictures that can help for people with the concepts, like these:

Pitch

Yaw

Roll

Now the gyroscopes on the iBOT are there to look at the movement of the chair, measure these forces.

The computers in the iBOT generally work under the principle of three independent subsystems so that if any one of them is getting different results then what one would expect, you will either see corrections being made or end up in a warning state if none can be made. But that does not mean that the gyroscopes are all there to provide nothing but redundancy.

Like with the Segway, They are laid out a bit more cleverly than that....

Three of them are placed on the main axis of the chair, in a straight line from front to back.

And the other three are off axis at various strategic places to handle detection of various other kinds of movement.

Combined with the data of the chair's mode and factors like the acceleration and turning being applied, the computer system can determine three things:

• where it is;
  
• where it is moving;
• where it is trying to move.

With that information, it can also know if its progress is being blocked or hindered or accelerated in some way, so it can use its internal governor to speed up or slow down, to correct an incorrect situation, or to trigger a fault condition if it cannot recover from a problem without user assistance.

The gyroscopes themselves are used to work together to provide the data rather than just having two items trying to get the same answer redundantly to re-check the answer -- because if the computer knows by how much the two gyroscopes should be different, then it can know when something isn't right. The combination is described on many sites online, like this one that talks about the Segway but which partly applies here too (the main difference being that the iBOT is not trying to get data on steering from the movements of the passenger, it is trying to get data to keep the passenger upright, in an inverted pendulum kind of system.

Other sites like this one also have other somewhat useful descriptions:

Like the Segway HT, the iBOT contains patented dynamic stabilization (iBALANCE) technology, an integrated combination of sensor and software components and multiple computers that work in conjunction with gyroscopes. Gyroscopes are motion sensors that help maintain balance. When the gyroscopes sense movement, a signal is sent to the computers. The computers process the information and tell the motors how to move the wheels to maintain stability. This electronic balance system is custom-programmed to the user's center of gravity, to monitor and respond to subtle changes in motion. Reach forward to shake hands, and the iBOT moves with you. Lean back and it moves away as well. The iBOT constantly realigns and adjusts its wheel position and seat orientation to keep the user upright and stable at all times, even when driving up and down curbs or inclines. In addition, the iBOT includes built-in triple redundant backup systems, as well as auditory and visual signals to provide even more safety and assurance. With input from the rider or an assistant, in "Stair Function" the iBOT utilizes gyroscopes and adjusts to the driver's center of gravity, climbing stairs by rotating wheels up and over each other. The iBOT can allow riders to stand up to the same eye-level as colleagues. The "Balance Function" of the iBOT can raise the rider to eye level for any number of business or social interactions. It lets the rider see over counters, and reach a high shelf in the office, kitchen or supermarket, safely and easily.

And this page from Silicon Sensing, one of the component providers, gives some more info:

The Segway PT is instantly recognisable across the world as a unique and alternative means of transport. Its press launch in December 2001 attracted enormous attention. But, by the time of its launch, Silicon Sensing had been working with the inventor of the Segway PT – Dean Kamen – for several years helping to develop and deliver its key design element – the balancing technology. This close relationship stemmed from our role in providing the balancing gyros for the IBOT®, the equally novel balancing wheelchair, from the same inventor.

Technically speaking, the Segway design is classic implementation of the 'inverted pendulum control theory' – balancing a broomstick on your fingertip is another example of the same thing. But to enable an automatically-balancing system based on this theory demands the availability of sensing, processing and actuation, all of which are fast and accurate enough. And for a commercially-viable product to emerge, this further demands the availability of these technologies at affordable prices, with sufficient robustness and reliability, and being of a suitable size.  The overall system concept demanded that the Segway PT could always continue to balance if a component fails, whilst providing alarms and reversionary action to ensure that the rider is able to dismount safely.

Being involved from the very early days, Silicon Sensing were able to propose and develop an innovative design, to be called the Balance Sensor Assembly, in which the size, reliability and affordability criteria were met through use of our VSG3-based silicon MEMS gyro technology. A key requirement was at least dual redundancy in balance sensing – and the desire for triple redundancy in at least the pitch axis. Although not immediately obvious, the other two axes of yaw and roll also required to be sensed for the situation in which the Segway PT is balancing on a slope.

The resulting solution is ingenious. Rather than providing dual and triple redundancy on each axis separately, the gyros are set at angles such that, by applying trigonometry to any pair of gyros, it is possible to deduce pure pitch, roll or yaw in more than one way. In summary, the solution provides three ways of measuring pitch and two each of measuring yaw and roll. To complete the module, two dual-axis liquid tilt sensors are included which sense the true 'down' direction and thus the pitch and roll angles. Processing within the BSA – again duplicated both electrically and physically – continuously checks the sensor data and monitors for any failures.

And then this document from BAE Systems, another one of the component providers, gives some more technical info on the nature of how the results of multiple gyroscopes are combined:

Using Maths to support design engineering
Ideas from maths are also important in engineering. The Segway HT makes use of a simple but ingenious bit of maths to reduce the number of silicon sensors used in the balance sensor assembly (BSA).

Directions of motion
There are three kinds of rotational motion that the Segway can experience; pitch, roll and yaw. These can be detected by a gyroscopic sensor.

Pitch Stand up and lean forwards and backwards
Roll Stand up and lean from side to side
Yaw While standing upright turn from your left to your right

For safety reasons each direction of motion needs to have two sensors, apart from pitch (the motion used to control the Segway HT) which needs three independent sensors. So, seven (rather expensive) sensors in all should be needed.

Geometry to the rescue
Cleverly, the BSA has one independent sensor (sensor 1) measuring just pitch and a set of four other sensors, angled such that each has two jobs. Sensors 2 and 3 both measure pitch AND roll. These are physically arranged so that positive pitch motion will cause both sensors to give a positive output signal. A positive roll motion will reduce the signal from sensor 2 and increase the signal from sensor 3.

Sensor 2 measures pitch minus roll. We can write this as Sensor 2 = P – R.
Sensor 3 measures pitch plus roll. We can write this as Sensor 3 = P + R

If we add these two equations together like this:

Sensor 2 = P – R
plus
Sensor 3 = P + R
Sensor 2 + Sensor 3 = 2P (the +R and the -R cancel each other out)

If we subtract these two equations from one another like this:

Sensor 2 = P – R
minus
Sensor 3 = P + R
Sensor 2 – Sensor 3 = P – R – (P + R)
Sensor 2 – Sensor 3 = -2R (the +P and –P cancel each other out).

The steering is done via a Joystick, and its way of doing the steering combines the way a boat would be directed (for the lateral directions) with forward and backward movement handled via forward and back in the joystick in a way that does not exactly match any that I have seen (usually this is handled elsewhere, in a throttle), thus leading to the not entirely intuitive backward movement that I am still learning.

Usually I rely on the zero radius turns so I can just go forward; this is definitely slowing down my "backwards" learning. :-)

Anyone want to take a guess on the difference between the Segway and the iBOT, with the former needing five gyroscopes and the latter needing six, in particular tp how the slightly changed mission of the one unit changes the way the gyroscopes are laid out? I give some of the answer away above in the Segway descriptions and the requirements of the two different units but not all of it.

Anyway, no one ever asked that question John did, which in truth seems like a very reasonable question to ask in response to the six gyroscopes answer. No one else has yet managed to get that far....

This post brought to you by ♿ (U+267f, a.k.a. WHEELCHAIR SYMBOL)

The title of this blog is an allusion to Coppola's Apocalypse Now, and eventually I'll be quoting a bit of the Herr-provided narration (those are the pieces Martin Sheen read)...

It all started with a seemingly innocent question the other day. It went something like this (product and component names removed to protect whatever might deserve protecting):

We are hitting an issue where surrogate pair characters do not display correctly on localized builds, but display correctly on English builds. This appears to be because the MS UI Gothic font used in the localized builds doesn’t “automatically” do the correct font linking.  (This can be verified by e.g. opening Wordpad, setting the font to MS UI Gothic, and typing some surrogate pair characters—you just get squares.  If the font is something else, e.g. Arial, the font linking works correctly.)

Is this a known issue with the MS UI Gothic font face? We are currently using one function to obtain the desired font face. Should we be calling a different function instead of this, or in addition to this?

Now as it turns out, there were several different issues going on here.

It start with the involvement of GDI font linking and Uniscribe font fallback, discussed previously in blogs like Font Linking vs. Font Fallback.

First and foremost was the fact that this was what they call a tester scenario. Because of this,the actual supplementary, CJK Extension B characters in question were not ones that are in any version of JIS (including the latest JIS X 213), which is why they were seeing notdef glyphs (aka square boxes).

Uniscribe largely stays out of the world of CJK (Chinese, Japanese, and Korean) text, allowing GDI font linking to so most of the work here. Usually this will guarantee that some ideograph will make an appearance, because as long as it is in one of those core CJK fonts, it will be on the screen.

But there is one time when Uniscribe is completely involved and GDI font linking is not -- and that is supplementary characters.

And Uniscribe is not quite as sophisticated in its efforts here -- it will see if the current font claims to support the Unicode supplementary ideographic plane (which contains e.g. CJK Extension B). If it does then the font will be used, even if there turn out to be some missing characters.

For the Japanese fonts, such as MS Gothic:

and MS PGothic:

and MS Mincho:

and MS PMincho:

and Meiryo:

each font is actually pretty much limited to the 300-some CJK Extension B characters in JIS X 213.

If you pick one of these fonts to display any other random Extension B ideograph, then you will get a square box.

And if you pick a font with no Extension B support at all, then it will pick one font to look in, based on its algorithm and system locale settings -- thus if you choose Arial or Tahoma or Microsoft Sans Serif or Segoe UI, then you will possibly also get an ideograph!

Korean does not have Extension B in any of its fonts.Given the gemneral tendency toward de-emphasis of Hanja in South Korea and the virtual illegality of it in North Korea, this is hardly a surprise (though this could change in the future if the customer demand drives change here).

And for the most part Chinese has the widest support. Because whether one uses the Simplified Chinese SimSun-ExtB font:

or the Taiwanese style Traditional Chinese font MingLiU-ExtB:

or the Taiwanese style Traditional Chinese font PMingLiU-ExtB:

or the Hong Kong style Traditional Chinese font MingLiU_HKSCS-ExtB:

one has a much larger number of ideographs to choose from.

The ranges are of course based on preferred glyphs in the PRC GB18030, Taiwan CNS11643, and Hong Kong HKSCS standards, respectively -- kind of the ultimate exercise of using a code page as a repertoire fence (something I have discussed before).

But the bug did not quite end there.

You seem it seems that the application had its own custom font choosing behavior, which in this case happened to be preferring the newer ClearType Simplified Chinese Microsoft YaHei font.

A font that also has some Extension B in it.

Eight CJK Extension B Ideographs, in fact:

These eight ideographs are:

• 𠂇 (U+20087)
  
• 𠂉 (U+20089)
  
• 𠃌 (U+200cc)
  
• 𠦝 (U+2099d)
  
• 𡗗 (U+215d7)
  
• 𢦏 (U+2298f)
  
• 𤇾 (U+241fe)
  
• 𧾷 (U+27fb7)

So far, these eight characters as a set seem to have no special relationship in China, Taiwan, Hong Kong, Macao, Singapore, Japan, Korea, or Vietnam, those being the major places where ideographs either are in use or have been within the last 1000 years.

If the characters spelled something special, I'd assume it was some kind of Easter Egg in the font (imagine the challenge if coming up with such an egg that relied on eight Unicode characters displayed in code point order -- talk about a fun word challenge in any language!

I am reminded of a bit from Apocalypse Now where Martin Sheen describes a report about Col Kurtz. Specially modified for the current situation, for the conspiracy theory minded:

Late Summer-Fall 2008:
The proper glyphs for ideographic text in the supplementary
planes show up fine in Vista. Then in November in one font
is noted the presence of eight specific ideographs. Two of
them are in JIS X 213, three are from a list of Hong Kong
Cantonese, one is from some from China. The number of
Extension B ideographs visible in the application in China
drops off to nothing. Guess they must have picked the
wrong eight characters.

Kind of a stretch obviously. But still fun to write (had I time to really draw this one out it would have been as much fun in my opinion as that Matrix one!

Whatever the reasons, their presence (due to the Uniscribe design here) can really break Extension B display support if someone is using the cool font with ClearType support.

If I had to guess, I'd wonder whether they were in there as part of an experimental effort at looking at ClearType Extension B support that just never got taken out (why would they? It's not like they are wrong, except in the meta sense of their effect!). But again that is just a guess. Probably more likely than my Apocalypse Font scenario above! ;-)

An interesting situation, in any case....

This blog brought to you by 𠂇 𠂉 𠃌 𠦝 𡗗 𢦏 𤇾 𧾷 (U+20087, U+20089, U+200cc, U+2099d, U+215d7, U+2298f, U+241fe, and U+27fb7)

This blog is as off topic as you can get without a prescription from your doctor....

Sometimes when one doesn't get the answer one wants, one can feel somewhat bitter about that fact.

Technical problems with computers can cause a person to be particularly susceptible to that kind of reaction, actually.

Though there can also be more to it, sometimes.

Case in point: a response to a blog from almost two years ago Vista turns on everything, which explains how you can't turn of the Text Services Framework anymore, like you could in the old days of prior versions.

Admittedly not great news for people who wanted to turn it off for application compatibility reasons.

Anyway, the response that Luke sent on (with no return address):

Useless as ever. You are nothing but a fool. This post is less useful than a broken key. I come here wanting to learn how to turn off advanced text services, and you take up several paragraphs to say "You can't". Don't ever attempt at helping anyone you useless 9 year old.

There is something particularly hateful about these words that really gives me pause.

It could be simple frustration leading to an emotional over-reaction, one that the seemingly anonymous nature of the Internet only encourages.

And some of the words such as "I come here wanting to learn how to turn off advanced text services" though clearly the title doesn't even suggest that the blog is about Advanced Text Services at all. tend to clearly suggest a man who found Vista turns on everything via a Google search (as I have to admit so many do).

And someone who found the blog by searching specifically for how to turn off TSF in Vista who read the whole post might get very frustrated about the "waste of time" and all.

And the conclusion of the comment (Don't ever attempt at helping anyone you useless 9 year old) certainly does display a certain amount of impatience and frustration. The kind that makes people lash out in perhaps strange ways that seem vaguely inappropriate.

Though there is something else in those words and others like "you are nothing but a fool", something that does not fit the picture -- it is not just the one blog that has Luke so unhappy with me. There really is something more going on here, running much deeper than a momentary frustration at not solving one single problem.

And then the initial bit of the comment (i.e. useless as ever) really doesn't seem to match here either, and makes no sense in the context of someone who had never been here before and (after the mistake of visiting the one time) would never visit again.

This is someone who doesn't like me, or maybe my online "persona", or maybe after having met me in person. Someone who just really finds no use for me whatsoever.

It's funny, I think that some of the people who hate me the most spend more time dissecting my words for inappropriate meanings to prove their beliefs than the people who are actually fans. This Luke may be one of them, one of the people who just really doesn't care for the taste of my brand of chai.

I used to talk with my friend Liz about this, and I have talked about it with Andrea too - in fact I've had this conversation with both of them long before I even had a Blog, nay before Blog was even a word. And both of them have pointed out that if I wanted to reverse everything I could, but that I speak with a very distinctive voice and would probably have a very hard time changing that since it mirrors the way I think about things.

I gave up three decades ago (in the third grade) trying to please everybody, and have never had cause to think I made the wrong decision back then.

The Blog is perhaps a megaphone, but not one that is changing what I say or how I say it all that much. I use it (and occasionally even abuse it!) in the same way that I would have done in any book or website or email or newsgroup post or presentation or conversation. I can name both people I have maddeningly frustrated and people I have ecstatically delighted. And I think I do serve a "net positive" purpose with what I do -- for myself, for my group, for Windows, for Microsoft, etc.

And of course you do have a choice here -- you could just not read me if you don't like me, or what I say, or both.

So Luke (or whatever your name actually is), if you want to come out from where you are hiding and tell me what your actual concerns with me are then I'd be happy to hear them or even discuss them. Or if you'd rather hide grudges or hatreds behind anonymous venomous messages then I suppose that is okay too.

Though the likelihood of having either influence or impact is much greater in the former approach than the latter. A friendly suggestion. :-)

This blog no sponsor, just as this sentence no verb.

Warning: Excessive PASS puns follow; if you don't like that sort of thing, then do not PASS...

It was one of those fun conversations that I find myself in from time to time.

Over IM, I was talking to my friend Rachel (a very smart ASPNET developer I know who rather delightfully has a properly spelled last name Appel rather than those more fruity types of names out there, if you know what I mean).

We were just kind of PASSing the time.

I asked her (in PASSing) whether she was going to PASS (I was referring to the upcoming conference put on by the PASS, the Professional Association for SQL Server).

She told me that due to the dates in question, she was pretty sure she'd have to PASS this time.

I explained that I was given a PASS for PASS, working in the Ask the Experts area.

She couldn't PASS up the opportunity to ask, "When is PASS?"

The dates were, I explained, "The PASS Summit is November 18th-21st in Seattle."

"Oh, I'll definitely have to PASS, I have client work I have to do then."

"You're going to PASS on PASS?" I asked.

"I think i'll have to PASS on PASS, even if you could get me a PASS to PASS."

"Bummer. I probably shouldn't try to get you a PASS to PASS anyway. People might think it inappropriate -- like I was trying to make a PASS at you or something!"

"Nash, they know me. I could probably get my own PASS to PASS, if I didn't have to PASS on PASS. I do have that client work to do."

"Yeah, plus those boarding PASSes don't pay for themselves."

"That too."

"Okay, as excuses go it is a pretty good one. I'll let it PASS."

It went on a bit longer with PASS puns, though we never made it to the NONE SHALL PASS scene from Monty Python and the Holy Grail, though I suspect that was because I didn't think of it until later.

Though as a point of fact, unlike the black knight, way over 3000 people will PASS, to go to PASS -- and not PASS on PASS.

It is an awesome SQL Server conference and I promise that I have all of the PASS puns out of my system now. I promise. :-)

If you will be there, be sure to look for me on my iBOT with the I'm a PC stickers on each side....

This blog brought to you by ⎑ (U+2391, aka PASSIVE-PULL-DOWN-OUTPUT SYMBOL)

Microsoft tends to get criticized, no matter what they (by which I mean we) do.

They (by which I mean customers) hate that the default install the additional IME, keyboard, font, and code page files(ref: What isn't in the default install for NLS).

But of course someone else (customer again) would be disturbed when we (Microsoft) "fixed" this, concerned that The fonts directory is freaking huge in Vista.

And then of course other people (customers again) are installing more fonts then any human could really want and trying their best at blowing their font cache. Obviously if several hundred fonts is too much then several hundreds of hundreds of them would be stratospheric.

Anyway, when we (Microsoft) remove the instructions to install files conditionally since we (Microsoft) no longer need to install them, people (customers again) point out the problems of the smaller intl.inf.

And then just the other day, someone else noticed you could install code page files anymore, and asked urgently:

...customer has an application that uses iso 8859-7  (WinXP and Win2000); after upgrading to Vista business he sees no option for elot-928 (iso 8859-7 ) and he is saying that the codepage has been removed and has been looking for it and ways to set it on Vista.

Ah, the concern was that because someone noticed that the additional code page install option had been removed, and they assumed this meant we had removed the code page.

I suppose we could have cluttered the user interface up with the note that you don't have to install the code pages because they are there, but I think the right call was made there.

But it does seem challenging sometimes to do the work to address a customer concern....

The happy note is that for the application in question the behavior will be better. Though clearly this does not always impact customer behavior. :-)

This blog brought to you by 𧾷 (U+27fb7, a CJK Extension B ideograph that is not on any legacy code page)

Lest you have any doubts, I speak here for myself and only for myself, not for Microsoft or for any person, group, or division within Microsoft. This statement is so simple that anyone can get it, right?

Now although I work for Microsoft, for everything I am about to discuss I am just a user of the technologies, not someone who even knows who owns it or works on it. So for this particular blog within the Blog, think of me as an outsider....

At the end of last month I got a mail that many other people got as well. The mail went:

Dear MSN Groups Customer, As a valued MSN Groups or MSN Communities Web Folders customer, we want to notify you that the MSN Groups service will close on February 21, 2009 and you will have the opportunity to move your group to our new partner service, Multiply. We understand the importance of keeping your group together, so we partnered with Multiply to create a migration process that moves your group to their service to preserve your online community and its history. Read on to find out about how to kick off the automatic migration of your group to Multiply. We realise this may be unexpected, so before presenting your options we want to briefly share why we've made this decision. Why? Because we are dedicated to providing our customers with the most current and user friendly technology available today we made the difficult decision to close the MSN Groups service. This decision is part of an overall investment to update and re-align our online services with Windows Live. In the long term we believe that closing the service is the best way to continue to offer innovative and effective services that help you stay in touch with the people you care about. We plan to launch a new Groups service in the coming weeks, but unlike MSN Groups, Windows Live Groups will focus on offering a place for small groups to collaborate. Multiply is available now, making it your best option today for continuing to share and communicate together online. Options for moving your group to a new service We've listed some options and resources below to help you decide what to do with your group. Option 1: Automatically move your group and its data. We have established a partnership with Multiply, an online group and media sharing service so our users can choose to migrate their group to Multiply's service. Choosing this option is free and easy to use: Multiply will move the Group's content on your behalf and invite members to re-join your group in its new location. To begin the migration click here. Option 2: Start again on another service. You can start from scratch and create your group on a different service but we recommend having your Group moved automatically by Multiply. This will enable your Group to transition easily and continue to enjoy the community you have created. Option 3: Start again on Windows Live Groups. To further expand our mix of communications and sharing services, Windows Live will launch a new service this autumn, Windows Live Groups. We plan to launch Windows Live Groups to the public in the coming weeks as a service that helps small groups or clubs collaborate online.   Options for MSN Communities Web Folders users If you use save files to the MSN Communities web folders (also known as "My Web Sites on MSN" or the web folder "My Groups"), these services are part of MSN Groups and will therefore will also be closed on February 21, 2009. We recommend that if you store files online using MSN Communities web folders that you back up these files locally, then upload them to another online storage service such as Windows Live SkyDrive. For more details on how to find and move files saved to your web folders, visit the MSN Groups Resource Center. Your Next Steps We have sent this letter to each MSN Groups user, whether member or manager. If you are: A member or user of MSN Groups: Check with your group manager to determine whether they plan to migrate the group. A manager: Visit the MSN Groups Resource Center to learn more about your options and consider soliciting feedback from your group members about what they would prefer to do, when and how. The Resource Center also provides a sample splash page you can use to notify your members that the group will move. If you're ready to move the group now, click here.   What to Expect between now and the closing date Between today and February 21, 2009 the MSN Groups service will remain the same as it is now. We will remove the option to add more storage to your group but other features will remain until the service is shut down and you can use it the same way you do today until the date of closure. Where can I learn more? You probably have more questions, and that's why we created a website to address them. Please visit the MSN Groups Resource Center at any time for the most up to date answers to common questions, information about migrating your group to Multiply, contact information for our support staff, and important dates. Our support staff are equipped to answer your questions and guide you through issues that may arise as you decide what to do with your group. They are ready to help so don't hesitate to contact them at MSN Groups Customer Support with your questions. We thank you for using our services and regret any inconvenience this may cause. MSN Groups, Microsoft Corporation Microsoft respects your privacy. To learn more, please read our online Privacy Statement. Microsoft Corporation, One Microsoft Way, Redmond, WA 98052

Now the groups I belong to are pretty much limited to the VOLT and WEFT groups, and I don't own any groups myself.

The whole situation seemed eerily familiar, though.

It was several years ago, in the CompuServe forums.

Microsoft had a huge presence there -- for betas, for product support, for product insiders.

Suddenly, they were moving out -- everything was moving to the new (non-replicated) NNTP servers that Microsoft put up.

There was a new Microsoft provided newsgroup reader that was in Beta, it had a code name of Athena, I believe. Though the goddess would undoubtedly smite the folk who though it was ready to handle the traffic in question, and the users in question (many of whom had never been in a newsgroup, some of whom had never been on the Internet outside of a closed client like CIS).

Several products in beta kept their CIS forums after people made a strong push to explain that this move could risk their product ship dates....

I remember a few months later talking to a product manager I knew who remarked how impressed he was at the level of sophistication of the questions being asked in the new Microsoft newsgroups, as compared with the old CIS forums. I had to break it him by pointing out that the reason was that the move was so poorly done that most of the customers had gotten lost along the way.

An effective way to improve the sophistication of your audience, that.

Reminds me of an old joke:

A man takes his wife to the doctor because she is ill.

The doctor explains that he hasn't run all the tests yet and it will take him several days to do so. But in the meantime he has narrowed it down to either Alzheimer's Disease or AIDS.

The man is horrified. "What do I do until the test results come back?" he asks, fearfully.

The doctor responds: "That's simple. Take her to the mall and leave her there. If she comes home then don't sleep with her."

Now this joke is truly offensive, yet in its own way this is kind of what was done to a whole bunch of customers.

And given the differences between Athena/all later Microsoft newsgroup clients and the clients that were already out there, many issues with differences in the way the MS clients work still plague the newsgroups community to this day -- phenomena like fully quoting old posts by default, top posting, etc. Microsoft managed to make itself even less popular with a large group of people that really didn't like them much anyway, and they managed to lose a bunch of their own customers too. MVPs like me went from posting thousands of responses a month to low hundreds -- and if I skipped a month, I lost no sleep. And I was not unusual in this regard -- many regularly posting experts disappeared or massively decreased their support due to the real annoyances with the client software. And they never came back.

That product manager I mentioned figured that Microsoft should put up a white paper explaining how to get to the newsgroups, which prompted to ask him where to put it up. "On the Internet!" he exclaimed, not even realizing the irony of the response....

Now as it turns out, the scuttlebutt of the CIS to newsgroups migration (reportedly) had to do with a limited time offer that Microsoft had to get out of it contract with CompuServe, which they jumped at even though their migration plans were not fully ready. It might be total fiction and I have no evidence that this is the case but since it kind of explains all of the facts I am willing to take it as the most likely hypothesis of the many I heard.

What am I to make now of this new announcement that the MSN Groups are shutting down, and what would otherwise be the most obvious intended replacement (Windows Live Groups) are not being provided a migration path like the Multiply option. What's up with that?

Now I am not a group owner, so I can't say whether they informed owners first or if they truly told everyone at the same time and told group members to ask your owners who may not have even heard about the plans. But I think this is probably just stupidity in the mail and not the plan -- I assume they sent an earlier mail to the owners and just didn't mention it in case people got offended that they were not given as much notice.

Primarily, I'm annoyed that they are doing all this before the replacement is ready -- it looks like the CIS thing all over again. And I suspect that lot of people will get lost in the shuffle, either intentionally because they go somewhere else (perhaps Google Groups, as one person in the VOLT group joked -- I wonder if Google is going to add a migration plan of their own to pick up some of these folk) or unintentionally because they just got lost on the way somewhere.

With MSN Groups and Windows Live Groups apparently targeting two different audiences, and with no replacement coming from Microsoft for some of those who will now be disenfranchised unless they do go to some other company entirely (such as Multiply, this looks a lot more like Microsoft getting out of a market (one they themselves took advantage of given groups like VOLT and WEFT) without being willing to admit why (the non-specific "Because we are dedicated to providing our customers with the most current and user friendly technology available today" implies that Microsoft thinks itself unable to provide those customers with something current or user-friendly? Surely that is not the message they intended here?).

Given that Microsoft did this kind of thing before (in the forum to newsgroup debacle), the whole thing doesn't really even seem all that innovative, to me. More of a "same shit, different group" kind of thing. Or, since the CIS thing happened back in the 90s, a "same shit, different decade/century/millennium" kind of thing. :-(

This blog brought to you by 𒁁 (U+12041, aka CUNEIFORM SIGN BAD)

Regular readers might recall a long ago blog entitled New in Vista: What's your name? Who's your daddy?, which talked about the new name-based NLS API functions, intended to wean people off of their use of LCIDs. Because let's face it, LCIDs suck.

Anyway, it turns out that in one case at least, bugs suck more.

Maybe people recall the even earlier blog entitled New in Vista Beta 1: more use of the word 'linguistic', which described (among other things) the NORM_LINGUISTIC_CASING flag -- a flag to do proper casing for Turkic languages.

Turns out there is a problem getting these two features to work together properly....

Well, take the following code in C# (it is a Win32 bug not a C# bug, but this lets us look at the managed case and the native one, which is sometimes relevant; plus in this case, more people can test it out themselves!):

using System;
using System.Globalization;
using System.Runtime.InteropServices;

class  test {
    static unsafe void Main() {
        Console.WriteLine("Turkish by name (native)");
        Console.WriteLine("131, \u0049 = " + CompareStringEx("tr-TR", 0x08000010, "\u0131", -1, "\u0049", -1, null, null, 0));
        Console.WriteLine("130, \u0069 = " + CompareStringEx("tr-TR", 0x08000010, "\u0130", -1, "\u0069", -1, null, null, 0));
        Console.WriteLine("\u0069, \u0049 = " + CompareStringEx("tr-TR", 0x08000010, "\u0069", -1, "\u0049", -1, null, null, 0));
        Console.WriteLine("English by name (native)");
        Console.WriteLine("131, \u0049 = " + CompareStringEx("en-US", 0x08000010, "\u0131", -1, "\u0049", -1, null, null, 0));
        Console.WriteLine("130, \u0069 = " + CompareStringEx("en-US", 0x08000010, "\u0130", -1, "\u0069", -1, null, null, 0));
        Console.WriteLine("\u0069, \u0049 = " + CompareStringEx("en-US", 0x08000010, "\u0069", -1, "\u0049", -1, null, null, 0));
        Console.WriteLine();
        Console.WriteLine("Turkish by LCID (native)");
        Console.WriteLine("131, \u0049 = " + CompareStringW(0x041f, 0x08000010, "\u0131", -1, "\u0049", -1));
        Console.WriteLine("130, \u0069 = " + CompareStringW(0x041f, 0x08000010, "\u0130", -1, "\u0069", -1));
        Console.WriteLine("\u0069, \u0049 = " + CompareStringW(0x041f, 0x08000010, "\u0069", -1, "\u0049", -1));
        Console.WriteLine("English by LCID (native)");
        Console.WriteLine("131, \u0049 = " + CompareStringW(0x0409, 0x08000010, "\u0131", -1, "\u0049", -1));
        Console.WriteLine("130, \u0069 = " + CompareStringW(0x0409, 0x08000010, "\u0130", -1, "\u0069", -1));
        Console.WriteLine("\u0069, \u0049 = " + CompareStringW(0x0409, 0x08000010, "\u0069", -1, "\u0049", -1));
        Console.WriteLine();
        Console.WriteLine("CultureInfo Turkey");
        CultureInfo ci;
        ci = new CultureInfo("tr-TR");
        Console.WriteLine(ci.CompareInfo.Name);
        Console.WriteLine(ci.CompareInfo.Compare("\u0131", "\u0049", CompareOptions.IgnoreCase));
        Console.WriteLine(ci.CompareInfo.Compare("\u0130", "\u0069", CompareOptions.IgnoreCase));
        Console.WriteLine(ci.CompareInfo.Compare("\u0069", "\u0049", CompareOptions.IgnoreCase));
        Console.WriteLine("CI en-US");
        ci = new CultureInfo("en-US");
        Console.WriteLine(ci.CompareInfo.Name);
        Console.WriteLine(ci.CompareInfo.Compare("\u0131", "\u0049", CompareOptions.IgnoreCase));
        Console.WriteLine(ci.CompareInfo.Compare("\u0130", "\u0069", CompareOptions.IgnoreCase));
        Console.WriteLine(ci.CompareInfo.Compare("\u0069", "\u0049", CompareOptions.IgnoreCase));
    }

    [DllImport("kernel32.dll",CharSet=CharSet.Unicode)]
    static unsafe extern int CompareStringEx(String strLocale, uint dwCmpFlags, String str1, int count1, string str2, int count2,
        char* version, char* reserved, int param );   

    [DllImport("kernel32.dll",CharSet=CharSet.Unicode)]
    static unsafe extern int CompareStringW(uint Locale, uint dwCmpFlags, string lpString1,
        int cchCount1, string lpString2, int cchCount2);
}

The results?

Turkish by name (native)
131, I = 3
130, i = 3
i, I = 2
English by name (native)
131, I = 3
130, i = 3
i, I = 2

Turkish by LCID (native)
131, I = 2
130, i = 2
i, I = 3
English by LCID (native)
131, I = 3
130, i = 3
i, I = 2

CultureInfo Turkey
tr-TR
0
0
1
CI en-US
en-US
1
1
0

The results that are the bug are in red.

Basically, the NORM_LINGUISTIC_CASING flag feature added in Vista does not work if you use the name-based NLS collation API functions added in Vista.

Not as bad as the whole IsSortable() == false? Well, sometimes it may be lying.... situation since in this case at least it was two different people.

However, that just lets the two people feel a little better; it doesn't really do anything for someone hit by the bug.

Thus on a scale of 1 to LAME, as mitigations go, this one is kinda lame. :-)

This is fixed in Windows 7, so I guess that's why we have new versions.

And I mentioned it here, so I guess that's why we have blogs. :-)

This post brought to you by İ (U+0130, a.k.a. LATIN CAPITAL LETTER I WITH DOT ABOVE)

Cheekheon asked via the Contact link:

Hi,

I have a NEC Desktop Computer running Windows XP Home Edition SP 2 with an ATI X300 (RV370) display card.

A few months ago, I did a security update from Microsoft and was advised to update my ATI display driver. So, I updated it accordingly to Version 8.6 (Display driver only).

However, the language of the system boot up display and the Windows Advanced Options Menu changed from English to another European language after the driver update.

The language is similarly changed when booting up a Linux LiveCD although the same LiveCD would display correctly in English when it is being booted up on another computer.

It may be pertinent to note that only the language of the system boot up display has changed.  Windows XP and Linux are still in English after the booting.  The BIOS Setup Utility is also still in English.

I have googled for a solution for the past few months without any success.

I would appreciate very much your expert opinion on what could be the problem and how to resolve it.  Thanks.

Regular readers may already know what to do here to get the situation fixed up....

I described the technique you use indirectly in

• Updating the keyboard list in the logon dialog and 
• The hacks of yesterday become the workarounds of today when corroborated by the features of tomorrow
  

and more directly described the problem in

• UI language of the LocalSystem account (which almost never shows UI) 

Now I have no idea why any kind of install would change this setting, but it is easy enough to change it back using this technique. :-)

This post brought to you by ꃆ (U+a0c6, a.k.a. YI SYLLABLE MUP)

So anyway, Kim's other recent blog, entitled Making a StreamWriter usable even after given garbage characters, highlights an interesting difference some of the methodology between the way that Windows and .Net handle encoding and codepages.

In Windows (in contrast to the behavior of most NLS API functions, as I have mentioned previously), the WideCharToMultiByte and MultiByteToWideChar functions will use the target buffer up until the point of failure, so that in the case of failure you may be able to do something with the partial results.

Now without a length indication the options of what can be done are more limited, but if nothing else then at least subsequent calls will not be affected by their predecessors.

.Net, on the other hand, has a default behavior here when you write to the stream that causes the StreamWriter to be useless.

The description in Kim's blog did not fully explain the problem, so I'll fill in the blank to it. :-)

She said:

For example, on an attempt to write U+DFC9, which is only half of a Unicode character (not a complete surrogate pair) an EncoderFallbackException was thrown

Now we have a stream here, so why is the stiry iver? Isn't the point of the stream thing that you can do it in chunks? Why would this be unrecoverable?

Well, the problem is that U+dfc9 is a low surogate.

See The basics of supplementary for a glossary update here!

As I mention in Why do the high surrogates have the low numbers? and other places, a surrogate pair is a high surrogate followed by a low surrogate.

A lone high surrogate is recoverable because it is incomplete.

But a lone low surrogate with no preceding high surrogate has no place to go, nothing to do -- it is toast unless you have a fallback plan in place, as Kim mentioned.

Though to be perfectly honest, after situations like that described in The torrents of U+fffd, I would much rather have had the default fallback plan be the U+fffd insertion.

I'm not a fan of the whole U+fffd thing, as I pointed out many times before. But given the huge push to change behavior from "drop illegal sequences" to "replace illegal sequences with the replacement character", I think behavior that did not throw in this case would have made for a better default....

And yes, I know there is a backcompat question here for the behavior, but since behavior was being changed anyway in this "in a service pack" change, there was a good opportunity to take a hard look at changing that default (since even already compiled applications were going to change their behavior!

This post brought to you by � (U+fffd, a.k.a. REPLACEMENT CHARACTER)

People have been misusing the word neutral in the whole area of internationalization of Microsoft products for quite some time now, a fact that I have discussed previously in blogs like Neutral? I do not think that word means what you think it means! and How ConvertDefaultLocale sorta broke backward compatibility in Vista, and why and Using full locales rather than the neutral ones? and Behold the Table Driven Text Service, Part 5 (All about the language, baby!).

As these various technologies mentioned in the above blogs tortuously abuse the nature of the word neutral over and over again, one is left wondering how one could remain neutral about the meaning of neutral given how much the meanings conflict!

Though to be honest, there is a competition for most over-used term in this area, and the other contender is the word default, which has been misused/overused in a lot of situations too, as I previously discussed in blogs like The ever-misleadingly incorrect usage of the word DEFAULT and Neutral? I do not think that word means what you think it means! again, and so on (I could cite more but it would just depress me if I did).

And of course the term locale has to be thrown in there too -- has any term been overused as much as it, really? And misused when language was what was really needed? And don't even get me started on culture here. From The ever-misleadingly incorrect usage of the word DEFAULT again and so on, locale nd the other two are way overused, and just as often misused as the rest (I'd add more cites but I might collapse in moral despair if I did!).

Then yesterday, my BCL friend and fave and fan Kim Hamilton wrote in her blog about What does the NeutralResourcesLanguageAttribute do?, and managed to avoid most of the difficulties in the above and in what managed code brings in to further confuse the issue. As she mentions:

NeutralResourcesLanguageAttribute marks the neutral culture for an assembly. That sounds self-referential, but a full description would require another blog post. To avoid getting bogged down, think of neutral culture roughly as the default language. (Fingers crossed that Michael Kaplan doesn't flame me for that oversimplification.)

Well Kim won't need to wear the flame retardant britches to lunch tomorrow, though I may enlist her at some point to help flame the people who made it so complicated and weird and terminologically retarded in the first place.

The confusion here obviously predates her attempt to explain things, and at least this simplification is self-consciously aware that it is walking on eggshells.

The fault, such as it is, lies in the shoulders she is standing on here. :-)

Do you notice how .NET now has yet another meaning for neutral that they threw in the ring -- they are using the term neutral to refer to their ultimate fallback (something I haqve talked about a bit before with the managed/native differences thereof in Random irreverent thoughts about the Ultimate Fallback).

That would have been a good [over-]use of the word default, to be honest, especially given the strong meaning that neutral has in .NET when it comes to neutral cultures, which are heavily used in resource loading.

When you consider that the .NET Framework was once so paranoid about using the same term to mean two different things that they introduced Ordinal for binary comparisons because they were paranoid that people would confuse the other use of binary in .NET (binary serialization/formatting) elsewhere, the fact that they would overload the word neutral in the same technology does suggest a small lapse in judgment somewhere.

Were the consistent terminology nuts over there napping that day? :-)

Consider the confusing legacy of the term ordinal which, while more consistently used then all of the above terms, has the disadvantage that all of them have the potential to be intuitive, something that ordinal will never have going for it.

They could have called it binary and binaryignorecase and been intuitive!

This blog brought to you by ⫻ (U+2afb, aka TRIPLE SOLIDUS BINARY RELATION, which is slanted even though not italicized)