March, 2010

Regular reader and collation hero of days past Santhosh Pillai asked over in the Suggestion Box:

Question: Does Windows 7 provide OS level support for any non-Gregorian support, such as switching your calendar?

Also, are there any online calenders that you know of that support non-Gregorian other than (Google Calendar)?

Of course I am assuming the question goes beyond the calendars mentioned in Calendar Identifiers:

Calendar identifier		Meaning
1	CAL_GREGORIAN	Gregorian (localized)
2	CAL_GREGORIAN_US	Gregorian (English strings always)
3	CAL_JAPAN	Japanese Emperor Era
4	CAL_TAIWAN	Taiwan calendar
5	CAL_KOREA	Korean Tangun Era
6	CAL_HIJRI	Hijri (Arabic Lunar)
7	CAL_THAI	Thai
8	CAL_HEBREW	Hebrew (Lunar)
9	CAL_GREGORIAN_ME_FRENCH	Gregorian Middle East French
10	CAL_GREGORIAN_ARABIC	Gregorian Arabic
11	CAL_GREGORIAN_XLIT_ENGLISH	Gregorian transliterated English
12	CAL_GREGORIAN_XLIT_FRENCH	Gregorian transliterated French
23	CAL_UMALQURA	Windows Vista and later: Um Al Qura (Arabic lunar) calendar

But I'm actually not sure what else there might be.

Switching the current calendar is as easy as a SetLocaleInfo call with the appropriate LOCALE_ICALENDARTYPE from the table above.

Beyond that? The pickings are pretty slim.

I mean, it was over five years ago that I pointed out the whole Calendars on Win32 -- just there for show.... thing.

So all of these calendars, they don't have full OS level support, which means parsing and formatting and using even if it's not the current calendar....

If that is what the question is about, the answer is a simple no. Because no such support exists....

Well, wait.

Technically, that isn't completely true.

There are some functions added to do calendar-type stuff, though they are marked deprecated and no header/lib file information is provided for them. You can see them in the National Language Supporty Functions list:

Function	Description
AdjustCalendarDate	Deprecated. Adjusts a date by a specified number of years, months, weeks or days.
ConvertCalDateTimeToSystemTime	Deprecated. Converts a specified CALDATETIME structure to a SYSTEMTIME structure.
ConvertSystemTimeToCalDateTime	Deprecated. Converts a specified SYSTEMTIME structure to a CALDATETIME structure.
GetCalendarDateFormatEx	Deprecated. Retrieves a properly formatted date string for the specified locale using the specified date.
GetCalendarSupportedDateRange	Deprecated. Gets the supported date range for a specified calendar.
IsCalendarLeapYear	Deprecated. Identifies whether the specified year is a leap year within the given era for the specific calendar.
UpdateCalendarDayOfWeek	Gets the day of week that corresponds with a specified day and populates the DayOfWeek field in the given CALDATETIME structure.

That last one is deprecated too, you can see if you look at the topic; they just forgot to mark it in the table like the others.

Hmmmm. Interesting set of functions, huh? :-)

These functions were added in Vista, as part of the work to support the updates to clock and calendar.

And presumably they could, by their existence, provide a way for .Net to support the ability to synthesize a calendar if Win32 supported one that .Net didn't know about. Though this might be something of a theoretical thing for now since as stagnant as calendars may be, it is clearly .Net that has gotten more new stuff, more recently....

These functions are supported in Windows 7, by the way. And in Server 2008 R2.

They provide a means to avoid one of the only other ways to do calendar stuff on the non-default calendar -- to switch the calendar.

For now, I'd say this is as close as you can get to the full OS support, though of couse parsing is still needed for "full" to be full enough.... 

Now my modest proposal about fonts  obtained in this blog lacks either the brazen absurdity or the sustained irony of Swift's A Modest Proposal, but maybe that could be countered by having the folks on the typography team eat the glyphs contained in their fonts they manage? :-)

The other day (well, it was Monday, by which I mean yesterday) I was giving a presentation to a bunch of engineers about font selection and how to do it an appropriate, localizable way.

I think I managed to make it interesting, which can be a challenge, sometimes.

Not everyone gets excited about that sort of thing, so you have to make an extra effort sometimes.

Anyway, at one point I talked about the importance of not using font face names to add font attributes.

I mean, it doesn't really make sense to make a font name show up in a string table with a name like Tahoma Bold.

Just for the pleasure of perhaps some Russian localizer calling it Tahoma Полужирный?

Or a Hungarian localizer to have to change it to Tahoma Kövér?

Or a Danish localizer to have to change it to Tahoma Fed?

Or a Norwegian localizer to have to change it to Tahoma Halvfet?

Or a Swedish localizer to have to change it to Tahoma Fet?

Or a Dutch localizer to have to change it to Tahoma Vet?

Or a German localizer to have to change it to Tahoma Fett?

Or an Italian localizer to have to change it to Tahoma Nero?

Or a Spanish localizer to have to change it to Tahoma Negrita?

Or a French localizer to have to change it to Tahoma Gras?

And so on. You get the idea.

Can you see why this would be a bad idea?

I mean, thinking about the following issues raised in Ready... set... Reboot Redux and Ready... set... Reboot Redux, part Deux!, it means all of the following is going on:

• the actual language that might be used here would ideally be based on system locale in the registry at the time the current session is started (a setting with no documented querying technique)
• it may be up to two session initializations before that answer can be deterministically expected, even if it were easy to get
• the string table language containing the font will be based on user interface language of the current session

It means the likely possibility of not getting the desired font effect is really a problem in a bunch of cases.

Now I have been railing about the best practices issues here for a while now.

But code keeps getting written, by Microsoft developers and the ones outside Microsoft. And people keep on making this mistake over and over again.

For giggles keep the whole related East Asian Font Names.... issue in mind as well - note how they actually did some work to solve that problem. By registering both names.

Maybe that is what should be done here, as well.

My humble proposal:

Any time a) the system locale setting that the session level font cache uses and b) the user interface language of that session, and/or c) the actual default system locale do not match, the font names for all of the fonts under these 2 to 3 languages should be registered.

This would guarantee that the fonts would work no matter what setting/configuration the developer might be using to try to get the right name.

I have almost succeeded in emptying out the Suggestion Box!

It has taken a while, but there are just a few left.

One recent one, from Andy:

I happened across the Tibetan keyboard layout and wondered about its design.

Each key has five letters assigned to it, two of which can be entered by pressing the plain key or combining it with Shift. So far so good.

Two more can be reached via dead keys m or Shift+m, the only Latin letter on the layout, which seems somewhat strange. To reach the last one, though, one has to hold Ctrl+Alt+Shift all at the same time while pressing the key. The layout has no AltGr, so yes, Ctrl+Alt really do have to be pressed separately.

This is bizarre. Why doesn't it make use of Ctrl+Alt without Shift? Why is there no AltGr? And why not use SGcaps rather than dead keys? And if dead keys are necessary, was there really no appropriate Tibetan symbol instead of Latin 'm'?

If you are a long-time regular reader, you may vaguely remember from way back early 2005 when I wrote Does MS pull new keyboard layouts out of their @!#$%? which referenced an FAQ entry that was written on the very topic the year before of where keyboard layouts come from:

Q - "I get the feeling Microsoft just makes up these keyboards by themselves.  Why don’t they represent my language the way I expect them to?"

A - New keyboards for a market always get tested in their respective market.  A great deal of research does go into the keyboards shipped with the system, with feedback from linguists, government officials, other internationalization experts, and local software providers. Often it is the case of Becker’s law applying (that is, for each expert, there is an equal and opposite expert), unfortunately. 

Now this is a nice, dismissive answer, to be sure.

And it generically talks about all keyboard layouts, rather than giving specifics on any one of them in particular.

This layout, which came from some of the Tibetan scholars in China, was the first effort at producing a layout that we had to have in order to ship the locale. It was based at a struggling attempt to build a standard of some sort, one that I seriously doubt existed on any actual hardware keyboards at the time (or maybe since):

Now there is a lot in this keyboard that is weird to my line of thinking, which you can see for yourself if you load the existing keyboard up in MSKLC and validate it.

Like the fact that the "spacing" form of the dead keys aren't included as the last item in the dead key tables, or the fact that the caps lock is defined for all of the shift state entries despite the fact that there isn't a case variation there.

Though it isn't like you would want an m or an M in your Tibetan, and perhaps defining the CAPS LOCK that way lets you switch without using the shift key so much (though defining it that way for the Alt+Control -- Shift+Alt+Control case as well might have been nice. As Andy pointed out those are a pain to type!).

This does partially work around one of Andy's actual concerns, about the pain in typing some of the characters. Though the lack of documentation about the "feature" makes it kind of unusable.

The complete lack of AltGr characters defined was on purpose and due to the general problem I first mentioned in "To start press the ALTGR key." Hmm... where's the ALTGR key? and have discussed many times since -- the way the helpful shortcuts in Word will make certain keystrokes untypeable. Those folks in China had been testing their layout in Word and wanted to avoid that problem. With a vengeance!

Their goal was to provide six "shift states", though two of them were being provided via the dead key mechanism.

I can't claim it as a keyboard I would be happy with myself (I wasn't at the time, either!), but I was not the one who needed to be made happy here.

The keyboard was largely taken as is, in any case. Since we didn't have anything else available to use, and had to have something. Tibetan was one of the specific languages listed in that very first phase of GB 18030 support as a requirement, and when in doubt about requirements one can seldom go wrong by a government's line of thinking if you follow the information given to you by some part of the government....

Now they did a bunch of testing later and there was even interest in a slightly different layout being put in, something like this:

Well they moved from the m  to the h and shifted some keys around based on some usage data in text, but it was mostly the same layout.

But they hadn't provided a .KLC file (just a bunch of tables of information) and in the end no one pushed it further.

In fact neither ever seemed to make it so far as being defined as a standard anywhere. I'm just saying....

I can't say I would have done it any differently had I been them - the delight of building your own keyboard from scratch is usually over by the time you have defined just a few keys. Building a whole layout can be torture (in fact it was a mild amount of torture for me to build this mock-up for the screen shots above, only slightly mitigated by the Perl script I wrote that used those tables of data!).

Mine validates, at least. But theirs might not have since I did not do the caps lock enabling stuff and they might have rather had that part done now that I think about it.

Since it is shipping precisely nowhere I didn't want to jump up to perfect it for the sake of some screen shots. :-)

There is also the fact that the space key was not designed on either layout to type anything but a space (U+0020), as opposed to some other desired characters instead:

• U+0f0b, aka TIBETAN MARK INTERSYLLABIC TSHEG
• U+0f0c, aka TIBETAN MARK DELIMITER TSHEG BSTAR

This is something that standards developed for other Tibetan script languages like Dzongka use as a matter of course (something we worked to fix in MSKLC 1.4 to allow these other characters used for SPACE-like actions to be used).

I do not have a layout with the Dzongka keyboard to add those screen shots, so you'll have to use your imagination.

In the end, I don't think either the "m" layout we shipped or the "h" layout we didn't is particularly useful for a native Tibetan user - in addition to all of the other problems described above including the lack of typical native spaces, the complete lack of punctuation symbols pretty much requires one to use multiple keyboard layouts to do even the simplest things on the computer.

It isn't a keyboard I would be satisfied with if it were my language, even within the framework of what simple keyboards on Windows can do.

But it is good enough for government work....

This is not a blog about my social life or love life. I'm just saying....

Over in the Suggestion Box, Ian Boyd asked:

I'm short on DATETIME.

Is there, or why is there no, LOCALE_SSHORTDATETIME?

There's a formatting string for short dates, LOCALE_SSHORTDATE (e.g. 3/18/2010). There's a formatting string for times, LOCALE_STIMEFORMAT (e.g. 9:08:13 AM). But there's no locale formatting string that combines a date and time together.

Is it safe to construct a full Date+Time as the date, following by a time, separated by a single space? It this true in all locale's, in LTR and RTL, etc?

There's several questions in there, I'll take them one at a time....

To answer the first question first -- there is no such combinatorial beast of date and time in Win32.

Dates and times are handled separately, despite the almost religious fervor about avoiding overlapping tokens in dates and times (that fervor is mostly an effort to avoid confusion and in English really only affects MINUTES and MONTHS).

And to answer the third question second, about whether it is safe to combine two of the existing ones, yes it is "safe" in that you won't physically hurt anyone by combining the two (that is all .Net does when it combines things, for what it is worth).

One could occasionally confuse users a tad given random factoids like the mostly left-to-right-ness of the Hebrew short date, but by and large they'll know what is going on, so it's fine.

I'll have to cover that issue about Hebrew another day!

And finally, to answer the second question third, the question of "why not?", there is an obvious and easy answer....

Since Win32 has supports for dates and times in two different functions, there is no place one could use such a thing as a LOCALE_SSHORTDATETIME! Thus if it existed it would just be there to make people ask where they could use it since there is no function that can take it as input!

Could a new function have been added? Of course.

But then it begs the question -- if one wants this for a log file or whatever, one can just concatenate the two values, it isn't like the new function would be doing any kind of unique work here that blocks developers, right? :-)

Now if you stretch the "why?" question back to wonder why things were designed that way in the first place, you'd have to ask a bunch of people who made decisions nearly 20 years ago. Only one of those people reads this blog as far as I knod, and even she probably only does so occasionally. Either way, I doubt any of them would really remember the why....

For what it is worth, the times that .Net creates formats like this (e.g. the "f", "F", "g" and "G" formats among the Standard Date and Time Format Strings) it builds them by concatenating various date formats and time formats. So if there are any that display oddly for you, then you are in good company -- every version of the .Net Framework will do equally well/poorly!

In response to This is not yet my take on DirectWrite, Ravi Chhabra asked (in a comment):

Hi Michael,

I note that you interest lies in:

"what's missing -- what scripts, what languages, what scenarios"

In that light I wanted to know if you could shed some lights on why Myanmar Font is mentioned as system supported in the DirectWrite documentation, while in fact it's not there in the system? Thanks.

and in the Suggestion Box:

There is a slide by Anantha Kancherla:

http://download.microsoft.com/download/5/E/6/5E66B27B-988B-4F50-AF3A-C2FF1E62180F/GRA-T515_WH08.pptx

On slide number 44 it list the scripts supported by DirectWrite, which clearly includes Myanmar. But on slide 45 it shows all the supported scripts in their native names such as देवनागरी etc. Myanmar is missing on this slide though, and most of the text seemed to be rendered with Segoe (Body). Could you shed some lights on how DirectWrite supports Myanmar Script, and yet is not included in Windows 7?

http://msdn.microsoft.com/en-us/library/dd371554%28VS.85%29.aspx

The MSDN also list Myanmar as available as a system font. Thanks.

Ravi.

Well, the ability to support scripts defined in OpenType can exist even in the absence of fonts on the system, of course. :-)

Though I suspect (without proof or genuine knowledge) that there was originally a plan to have a font there, a plan which ended up not working out for whatever reason, but no one was going to take out shaping engines or code to support the script.

If that's true, then anyone who has a font can see the script.

I have a few fonts on this machine that support it, but they were added later; they didn't come with Windows 7.

Now one thing that should have happened, and maybe someone could edit now, would be to take that entry for Myanmar on the Introducing DirectWrite page and add the asterisk meaning

"For scripts marked with an *, there are no default system fonts. Applications must install fonts that support these scripts."

There are all kinds of reasons US companies have to be careful about support for Myanmar -- political and legal and geopolitical reasons. So avoiding these kinds of slips in docs would be in everyone's best interests as specs and requirements and shipping features change....

Over in the Suggestion Box, Ivo asks:

I am working on a software that has to support all languages available for Windows. When I install Traditional Chinese the system reports the language as "zh-HK" (using GetThreadPreferredUILanguages). So I added "zh-HK" to my list of languages and everything worked fine.

However few days ago a user of mine complained that it doesn't work on his system because his language is "zh-TW".

What is the difference between "zh-HK" and "zh-TW"? Is there really a zh-TW version of Windows, and if so, why isn't that language pack available for download?

I am going to be very precise in my language as I try to answer Ivo's inquiry.

To start....

The list of languages into which Windows is localized is much (much!) larger than the list of MUI Language Packs.

I mean, just as an example that I bring up often, there are the Language Interface Packs. By the time the ones in prior versions were done, they outnumbered the fuller localized versions!

Most languages can be in both kinds of packaging, and of course there are some languages that cannot be anything Microsoft ships at all, in any form, given current events and situations.

For this specific case, one can go to Microsoft's Taiwan home page (here) and perhaps click around a bit, watching the graphics if one does not know the language, to see that there are a wide array of Microsoft products, including Windows.

Now with all that said, the differences between zh-HK and zh-TW are not huge.

Some would argue that last statement has a flaw in it, that the real sentence might have been more accurate as:

Now the differences between zh-HK and zh-TW are not huge enough.

(This would be on the same basis as the lack of en-US/en-GB differences.)

Really the differences are minor and relate primarily not to actual language differences that would ordinarily suggest localization differences, but rather a little bit to the issues discussed in Sometimes, tech companies cannot take sides, and a lot to the general lack of effort to duplicate things that are only mostly duplicated more than anything else.

Because there is a certain importance in being both cost-effective and apolitical if one wants to sell products across as many borders as possible. And the actual business case that has to make for each type of download Microsoft puts up.

In the end, not every language will be available in every way, for all of these reasons and maybe even more.

Getting back to the original question...

Wait, what was the original question? :-)

When Ivo mentioned:

I am working on a software that has to support all languages available for Windows. When I install Traditional Chinese the system reports the language as "zh-HK" (using GetThreadPreferredUILanguages). So I added "zh-HK" to my list of languages and everything worked fine.

Was he referring to a desire to localize the software into every language into which Windows is localized?

If so, then let's not forget the LIPs, which will be streaming out kind of semi-continuously for the next year and beyond.

The other differences between zh-HK and zh-TW are more to do with the locales themselves, and those differences (not to mention the obvious fact they are both on the list!) are on every machine. Windows is being sold both places, and customers are hopefully getting what they are looking for.

So as long as you are doing the right thing in regard to globalization support users will see the correct results on their machines no matter what language version they have installed.

Other differences? Well, different sorts, different focuses on character support (HKSCS vs. CNS11643, etc.) in relatioin to font choices, and such. I have talked about these things in the past.

Some differences are missing from qall versions of Windows that might be useful to provide; for example there is not quite the Cantonese support (important for Hong Kong, not so much for Taiwan) that I would personally like to see, but I'm doing a little something there for a starter, and there are genuine companies stepping up with professional efforts as well.

The most important thing to take away from all of this? There can be:

• some languages that are only in Windows base SKU languages but not Language [Interface] Packs, and
• some languages that are only in Windows Language [Interface] Packs and not in base SKU languages, and
• some languages that are in both Windows base SKU languages and Language [Interface] Packs, and
• some languages that are neither in Windows Language [Interface] Packs nor base SKU languages.

Anyway, I hope that answers Ivo's question; if not I expect there will be a comment, presently.

Over in the Suggestion Box, William Krick asks:

Hi Michael,

I'm having a strange issue at work that I believe is character set and/or font related but I'm not totally sure which is to blame.

We have one XP SP2 (don't ask) system (our development system) and we use various text editors notepad/textpad to view X12 data files which contain hex characters as delimiters.  The hex characters used are: 1C, 1D, 1E, 1F.  On our development system, notepad/textpad both display the hex characters as small black squares.  This is the behavior we actually want, which is probably the opposite of everyone else on the planet.

Our other system (non-dev) also has Windows XP SP2, but it has MANY more fonts installed (due to Office 2007, I assume).  When we view the same X12 data files in notepad/textpad on this system, the hex characters do not display at all.  No black square, nothing.  The characters adjacent to the hex delimiters just runs together.

Any idea what's going on and if there's any way to fix it?

Interesting to see how people are using Windows sometimes, isn't it? :-)

Interestingly, this is the same issue discussed previously in blogs such as What happens when you involve an unenabled Uniscribe with vertical text, given that Uniscribe doesn't handle vertical text? and IsComplexEnoughForYou? and IsCrAndLfComplex or what?.

When Uniscribe is enabled for text (i.e. when the Install files for complex script and right-to-left languages (including Thai) check box in the second tab of Regional and Language Options is checked), then you will not see square boxes for these or many other characters. But that level of intelligence about Unicode is something that does not occur if the check box is checked....

In the long run, William will want to find a different way to enable these users do their work since the functionality is always on in Vista and later. As I point out in We need to be optimizing for more than just the simple cases, it is best not to put one's performance tests, or Notepad usage, in the "simple" case.

Because users may have very good reasons for the complex cases, too....

Over in the Suggestion Box, dmelliott asked:

I read what I could find about fonts installed in the system not being installed in IE, but I could not find anything about how to solve the problem.  I would assume registry hacks?

Now I am not 100% sure what the question is here, since Internet Explorer doesn't install fonts (and even when it did it was just a few extras for Win9x and NT4 machines).

Maybe it is fonts not showing up in the per-script UI in Internet Explorer, in which case I have talked about the issues of how both choice is determined and selection is stored in blogs such as Where are the IE plain text fonts? and The importance of Tagalog to Burmese, aka "Of course I'd lie to you, I'm a font!".

Though without any information on either Internet Explorer or Windows version it is hard to guess further than that, or suggest solutions/workarounds/hacks.

Hopefully dmelliott is still around and can provide the additional info....

The other day, JC Ahangama mentioned in the Suggestion Box:

I suggest the topic "Romanizing paired with Orthographic fonts route for Indic".

I have successfully made a round trip conversion between romanized Sinhala (inside ISO-8859-1) and Unicode Sinhala.

It allows intuitive phonetic data entry that gets displayed in the native script if you have an orthographic font for the language created using Open Type 'ligq' features. Otherwise, it gracefully falls back to Latin-1, which is readable too.

Collation happens easily (I extended JavaScript sort() for it). This is a much simpler solution than the struggle we are going through with double-byte Indic.

For the record, I am completely, entirely, and unreservedly against this. UNICODE remains the answer, not new pseudo-solutions, be they new code pages, font hacks, or any other such item.

But I thought I'd answer the question in a slightly better way, one that provides a solution to the issue that does not rely on such parlor tricks or special cases.

And with that in mind....

Recently Scott Hanselman pointed me at something that I thought was pretty interesting (another Microsoft employee pointed it out to him, if memory serves).

It was a tool for inputting Indic text.

Now I have done my fair share of random work in various languages of India like Tamil, Hindi, and Bengali.

And I've been getting a fair amount of mail from people who wanted to tell me about their exciting input method that would change everything (and usually they proceeded to point how cheaply Microsoft could buy it!).

Often not very exciting for me since usually they are not that exciting.

But this time was going to be much cooler.

This time it was the Microsoft Indic Language Input Tool!

Now this thing is incredibly cool in both its web-based and desktop versions, it really is.

In fact, it caused me to wonder how Goldie was doing in her new job....

Because I took her name -- the one that was painstakingly spelled in Learning to spell in Bengali (when one doesn't know the language) -- and simply typed it in English using the English transliteration that she had known her whole life (Goldhuli Chaudhuri).

I typed it in straight, no pauses, and after I saw what happened I did it again taking screenshots as I went:

And that's right....

গোধূলি চৌধুরী

without needing to select an alternate candidate or even select the initial one (when I hit the space it committed the word). Typing was as fast as typing it in English.

Well duh! I was typing English. :-)

I have to say, on a more personal note:

I must admit that I am on the whole happy that this tool wasn't available when the events described in Learning to spell in Bengali (when one doesn't know the language) happened; had it been, it might well have fundamentally altered the nature of our relationship at the time, what with that whole initial little story involving Tagore and poetry and music and us and her parents and Bengali and a big email thread and lots of IMs. Had none of that happened, a-la- screwing up the Back to the Future kiss between Lea Thompson and Crispin Glover? The horror!

Well I'm not sure, but I imagine that no props would have been given to me if it were typing two words and we were done....

But I also have to say, on a more professional note:

I am very glad the Microsoft Indic Language Input Tool exists now.

My other experiments with it (e.g. random words and phrases I know in Bengali and Tamil) were also impressive, as was my (re)creation of the experience with Hindi and How would Harry Potter have pronounced शहिवाख़्‍ का दर्पण, anyway?. And I was once again very impressed.

I admit the one place the tool failed was in creating that unpronounceable word, but considering how amazing of a job it did at words that do exist I'm willing to live with its trouble on the one that doesn't!

So check out the tool, check out their blog, and keep your eyes on these folks.

Indic input? It just got easier for a whole lotta people.

And unless I am misunderstanding the registry keys the tool added, it is using the Text Services Framework, which makes it very "Windows friendly" to my way of thinking.

Is someone working on getting this stuff in the box some day? :-)

Okay, today we're gonna do a twofer!

First, regular reader and "friend of SIAO" Jan Kučera asked, over in the Suggestion Box:

Greetings! What references (and advices) would you suggest to someone considering implementing a complex script rendering engine? (on a platform where there is no such available of course :-))

Jan

Yikes!

I suppose you would want to hear me recommend Imitrex for the migraines such work will surely cause? :-)

The only advice I can really give is to look at the specs for fonts like the OpenType specification; in parts like the various appendices that go into the kinds of things that fonts can expect from the various shaping engines.

That will be a huge effort.

Heck, it is a huge effort, in Uniscribe.

Maybe if you don't care for needles you can try to use Motrin or Tylenol, instead. There will be many headaches in the future, there....

Okay, moving on to number two. :-)

Over in the Suggestion Box, alexcohn asked:

I would like to find a way to enable Uniscribe bidi support in Windows Mobile. The available solutions by 3rd parties are expensive and use dangerous hooks all around the system. Furthermore, they are not standard, and it is a burden on aplication developers to look for glitches with each one of these tools.

What does it take to build Uniscribe - enabled smartphone? If it requires special licensing, who may be a relevant point of contact in Microsoft?

Well, the main thing it takes is a CE platform on the device that includes the Uniscribe component.

For example you can see the CE 5.0 instructions in the Creating a Complex Scripts-enabled Run-Time Image topic, with the 6.0 version of that same topic right here.

That topic even has a list of supported locales, a table that is the same in both versions:

This suggests that this is a port of the XP RTM version of Uniscribe to the CE platform that is in 5.0 (the Uniscribe updates for Bengali and Malayalam came in XP SP2, asnd the ones for other languages such as Oriya came in Vista -- not sure if those were ported).

Now as a special note, CE Uniscribe does not do surrogate support like its desktop ancestor does, as discussed in Working With Unicode Surrogates. If that is what you were looking for, you won't need Uniscribe after all....

But the problem if you aren't building your own CE for a device is that it seems like none of the OEMs ever chose to include this component.

Unless someone knows of a device that chose to include it?

Now note that none of this applies to the upcoming version, which has had no solid information on this topic discussed....

Nothing technical in this post - if my multiple sclerosis does not interest you then feel free to skip this one, there will be another by morning!

Since this is not a political Blog, I am going to make sure this is not a political blog either. I am going to keep politics out of this one, and just talk about my situation over the last few decades. I'll be quick to delete anyone who tries to against this too much in the comments under the legal theory that the cross-examination will be exceeding the scope of direct! You have been warned!

Throughout the debate on health care, people have asked me various questions about how it would impact someone such as myself.

I figured I'd better explain.

My initial diagnosis, un-diagnosis, and re-diagnosis (details described in When is a question about prognosis a soup question?) was actually done as a self-pay kind of thing.

Interestingly, it was at first treated as a worker's compensation case, as the initial symptoms happened while I was working.

This was fine by me, as that was a part time job I was doing while building my consulting business into a full-time job, and I had no health insurance.

They quickly found it was not work-related, but the differential diagnosis included some pretty nasty potential stuff, from brain tumors on down. So there was not a ton of time to count the cost.

I was already doing a bunch of work as an independent contractor and/or a consultant (not enough that I would call it a career but it was a start), and I had piled up a bit of money doing it.

A bunch of that money got spent on MRIs with and without contrast, Visual Evoked Potential tests, Brain-auditory Evoked Potential tests, Somatosensory Evoked Potential tests, Corticosteroid infusions, and multiple visits to a neurologist (the one  who ordered all of those tests and treatments over the course of days and weeks and  months).

All in all, I spent at least $15,000 in that first year or two.

At the first juncture, my opinions about health insurance were made clear by the fact that there was no plan I could sign up for, since most would consider this lifelong illness to be a pre-existing condition, and the rest would be prohibitively expensive.

My neurologist pointed out that I may not need to be spending more money at that time (this was before Copaxone, Rebiff, Avonex, and Betaseron -- those $1000 a month drugs that have for so long since then become the staple for some neurologists), so now that I had paid for the expensive "diagnosis" part that, as long as I was mobile and had no debilitating symptoms, I could just go quiet and not worry about it so much.

Still I was advised to think about the future, as this was a ticking bomb of sorts, this fact that I had no health insurance and couldn't get any health insurance....

So there I was, off the grid. I would have exacerbations from time to time, and I would get better.

I bought the frames for glasses without a prescription (I wouldn't need prescription glasses for years, yet) because I would have some occasional visual symptoms and I wanted the excuse of I forgot my glasses if I was having trouble.

I bought a cane at a Rite-Aid without a doctor's recommendation because I was occasionally falling in the daytime and I decided I would rather people think I was a gimp than a drunk when it happened.

The latter was a terrible idea, by the way -- as I never got any training with it, and I was using the cane incorrectly!

The cane and the glasses were pretty cheap, by the way. So still self-pay, but not such a hardship.

In the middle of those years I had an attack of Trigeminal Neuralgia that was not assisted by medication, and I flew back to Connecticut to get the surgery done (all of that was previously described in Its been almost four years, Sherry....).

I still had no health insurance.

But Sherry was a resident (being paid by the hospital) and Dr. Poletti (who I did work for years prior) waived his surgical fee when I explained my insurance situation.

So I was "only" out the hospital fees, travel expenses, pre-op and post op medications, MRIs done before the travel to rule out more serious Janetta-esque illness mimicking MS symptoms, and follow-up care with a neurologist.

By the time the dust had settled, I was out a little over $25,000 that time.

All of this, I paid out of the (by this point) larger bit of money I had set aside.

After my post-op visits, I was off the grid again until my friend Tamra came to my apartment that I had not left for three weeks as I was having trouble walking and dragged me to the hospital.

I was now officially back in the system, and have been pretty much ever since.

Among other things, this included that whole "Copaxone, Rebiff, Avonex, or Betaseron -- pick your $1000 a month poison" thing. Plus new MRI scans, and new tests, time with physical therapy, and so on.

No one was waiving anything, so I went through another $40,000 or so over those next few years from all of that stuff.

Eventually, I started as a full-time employee at Microsoft, and one of the big conditions I had (knowing that my MS had cost me over $80,000 up to that time) was that the health insurance wouldn't exclude pre-existing conditions.

Microsoft's health insurance (in my case Premera Blue Cross as administrator, Microsoft as the paymaster) had no rules excluding pre-existing conditions.

Thus, by a fluke of circumstance I was finally be back under insurance again, just a decade or so and more or less $80,000 too late.

Now for many years that insurance has covered MRI scans and the $1000 a month drugs and later the six Novantrone infusions and eventually an iBot (which most insurance companies routinely deny and which even mine denied until after the first appeal -- as discussed in Cogito ergo cathedra... (I think, therefore IBOT...)).

I was prepared to pay for it myself if it was ultimately denied, after having gotten the hint of how empowering it would be (this was before later circumstances proved it to be such a game-changer that it was practically a world-changer). But it was nice to be back on health insurance, since the years I was covered had expenses that far outweighed/outspent the years I was not.

If Microsoft had no health insurance (rather than having such a superior one that they do!), I probably couldn't afford to work for them.

Now, looking back....

In some minor way, getting actual glasses sooner (rather than using "fake" glasses to hide visual issues) and training on the cane sooner (rather than using it in a way that increased the amount of fatigue) would have had symptomatic benefits for me, for over a decade that I had no such benefit.

And in a more major way, there is statistically significant benefit to overall disease course in going on one those drugs (Copaxone, Rebiff, Avonex, or Betaseron) earlier rather than later.

It will haunt me (and some of my friends, and maybe even an ex-girlfriend or two) for the rest of my days how much better I could have been, had I not been forced off the grid of insurance coverage by the rules about pre-existing conditions that made me think of clever solutions to avoid going bankrupt.

My choice, to be sure, but I really didn't have the money in those earlier days, and then since I was "off the grid" I had no doctor to encourage me to do anything different, or to go on some study, or whatever.

I will take the blame for the decision (since it was mine) and I definitely have the consequences of it to deal with -- I might still be walking or dancing or running or skiing now. Something I would gladly give up my iBot-esque "life on two wheels" for. In a heartbeat.

So getting back to the original issue, how do I feel about the health care bill passing?

Well, leave aside the way Johnson & Johnson dropped the iBot (discussed in From I SCOOT to IBOT, #10 of ??: The good, the bad, and the ugly (IN THAT ORDER)) in large part due to the refusal on the part of insurance companies to pay for it. I say leave it aside since even Medicare and Medicaid and the VA¹ were often refusing to pay for it, so perhaps it would be just as hard to get in a few years as it was back then.

Looking past that, my feelings?

Well, it is too late for me.

But luck saw me through in spite of that, during the early days.

However, the next person who has a similar situation does not have to rely on the luck of circumstance to be able to get more effective (and professional) short-term and long-term care.

Because no matter what the republicans get repealed or the states push back, that one bit of poison (the pre-existing condition exclusion clause) is unlikely to ever make it back in. For me it is the single most important part, and all of the rest is just trying to get more people covered. The pre-existing condition exclusion blocks even the people who could pay for insurance as I was from being able to get in on it.

1 - The one vereran who had an iBot that I met (at the Halloween Party at the Playboy Mansion, a charity gala for Wounded Warriors) did not get the iBot through the VA, it was through donations and other assistance after an injury in Iraq left him paralzed with a full spinal injury. He was one of the last -- possibly the last -- person to get an iBot....

Over in the Suggestion Box, PerryN Newton asked:

Michael,

Your blogs are very interesting reads and very informative - thanks!

Observation - you seem very informed on calendars and date functions, and have written several posts relating to them.

Question:  Have you ever tried to import Visual FoxPro table data that simply lists U.S. Presidents and their birthdates into SQL-Server?

Suggestion: When I last tried this, using VFP 6.0 and SQL-Server 6.0 I would always get an error and discovered it was a limitation of the beginning date recognized in SQL-Server and I wondered if this limit has been since documented (KB) or modified by subsequent SQL-Server versions.

Thanks for all you do and share!

Excellent question!

Now if you look at the latest SQl Server docs, you will get hints as to what might have been going on here originally.

Specifically, the table comparing the relevant datatypes in Date and Time Data Types and Functions (Transact-SQL):

Date and Time Data Types

The Transact-SQL date and time data types are listed in the following table.

Data type	Format	Range	Accuracy	Storage size (bytes)	User-defined fractional second precision	Time zone offset
time	hh:mm:ss[.nnnnnnn]	00:00:00.0000000 through 23:59:59.9999999	100 nanoseconds	3 to 5	Yes	No
date	YYYY-MM-DD	0001-01-01 through 9999-12-31	1 day	3	No	No
smalldatetime	YYYY-MM-DD hh:mm:ss	1900-01-01 through 2079-06-06	1 minute	4	No	No
datetime	YYYY-MM-DD hh:mm:ss[.nnn]	1753-01-01 through 9999-12-31	0.00333 second	8	No	No
datetime2	YYYY-MM-DD hh:mm:ss[.nnnnnnn]	0001-01-01 00:00:00.0000000 through 9999-12-31 23:59:59.9999999	100 nanoseconds	6 to 8	Yes	No
datetimeoffset	YYYY-MM-DD hh:mm:ss[.nnnnnnn] [+|-]hh:mm	0001-01-01 00:00:00.0000000 through 9999-12-31 23:59:59.9999999 (in UTC)	100 nanoseconds	8 to 10	Yes	Yes

Now obviously there are some datatypes that would not handle the full range of presidents, and others that have way more detail than one might need.

But one can find one or two that are just right!

No, this is not a log about my misuse of the language of Shakespeare and Chaucer. Or about my use of the words of curse, which has been on the decline recently anyway, thank you!

One of the downsides of working principally on Microsoft technologies is that you find yourself speaking a dialect that not everyone else will readily understand.

This is even worse if you work for Microsoft, since the insulation from all that is not Microsoft will just naturally be thicker.

And if you work for a specific product whose primary focus is on a single version, like Windows, then the shielding can be legion. If you know what I mean.

I'll give you an example.

In Vista a feature was added, a feature that most people feel was made much more useful/usable in Windows 7.

The controversial feature known as UAC (User Account Control?) or LUA (Limited User Access?). Basically a way to keep you from doing things that you can only do as an administrator unless you really intended to do so.

As a choice bit of irony, I found this to be much worse in Vista than Mac OS X but much worse there than in Windows 7. But that is a blog for another day. Or maybe never....

Anyway, I was going to talk about UAC/LUA for a minute.

There are many overt acts that have come up in the past like running the setup for App Locale or installing MSKLC 1.3 or adding a Table Text Service TIP to Windows.

And when you do so, it is either easier or in some cases only possible if you are running from a command prompt that has received this approval from the user -- this approval to be like an administrator type person.

Now what this is generally referred to by the aforementioned subset of people I mentioned as an elevated command prompt.

I have used this phrase many times, as you can see in a simple search here.

Now in almost every case my intent was that people use the simple UI you get by right-clicking a command Prompt shortcut:

You know, the one that says Run as administrator, which will cause there to be an elevated command prompt.

If you click on it, you will be prompted to make sure you really wanted to do that. If you say YES then the command prompt will indicate its "Admin-ness":

Anything you run from within that command prompt will now be run as if you are an administrator. So you can do these varous tasks I was talking about....

I had been calling it an "elevated command prompt" since Vista was called Longhorn so it never occurred to me that people might no know what this meant.

But if you look in the comments to some if those blogs, people often just skipped that bit and proceeded on, not realizing that this step was crucial for the support to work.

It was made a bit harder in the case of Table Text Service Tip installation since it would appear to work but the registry keys were put in a virtual store that was not used for the component. :-(

I don't know what the whole virtual store thing is really supposed to work for, but for these kinds of machine-wide, cross-process, permanent settings, it falls down rather quickly....

In fact, I think it is why good people like Andrew West are having trouble in Windows 7 (see the comments of Can't I pick the candidate list font if I don't speak fluent square box? for context) in some cases...

Anyway, from now on I will recommend to people when I need them to use an elevated command prompt to point to this topic, or even better to give more detailed instructions that don't assume people will know my bizarre dialect of English....

I get a lot of email.

Like at least 400 and sometimes as many as 1500 pieces of email a day, not counting spam (if I accept the premise that email is communication then I won't consider spam to be something I will call email).

Th organization system of rules that files stuff is staggeringly complex and allows me to scan through mail I need to see very quickly, and to triage items as they come in.

Not sleeping very much helps too. :-)

Sometimes people send me mail to report issues or ask questions about a problem they are having.

I will wade in and look into the problem and help them if I can or point them in a more hopeful direction if I can't.

Like the other day when someone reported a problem that didn't seem like his bug or even a bug in the platform he was using (Delphi), but on the OS underneath it (Windows, or Windows 7 to be precise).

I looked into it and sure enough his analysis was spot on.

But he didn't have the backstory, so I thought about it and figured it would be worth a blog to talk about the problem he is seeing. Maybe it will help someone else....

So there was this bug.

Wait, I should start at the beginning.

Like, before the bug.

Sound good?

You know how there has been this bug push that started in Vista and has continued unabated since that time to move people off of LCIDs and into Locale Names?

Like some of what is behind blogs I have written like Your LCID sucks?

Well under the covers, one of the big changes was to have the name stored in the registry and not just the LCID, right here:

Now obviously you should not depend on this LocaleName any more than you used to depend on Locale. Which some people did occasionally, even though this wasn't documented or supported, even though there have been documented and supported ways to do this since the earliest days -- functions like GetUserDefaultLCID.

Anyway, this new value was added, this LocaleName. And all of the code inside was changed to be name based, so that none of the NLS code was depending on the LCID anymore.

Everyone knew that some people were still reading that darn Locale registry value, so the value was not removed. But the Locale value was only there and being kept in sync with the LocaleName value for backward compatibility for these not entirely well-behaved apps that really were doing something that has been wrong since the earliest days.

All well and good, everyone can be happy, right?

Well, not entirely.

Like I said, there was this bug.

It turned out that in Windows 7, in some RUNAS/Impersonation scenarios, where one runs as another user, loads their HKCU registry, and does some work (as in the Impersonation and NLS blog I wrote back in 2005) to change the user locale, that the work to keep the LocaleName and Locale registry values in sync was not happening.

So you get something different; something like this:

(00000809 is the en-GB locale.) 

Maybe slightly different than this case, but the two values would definitely not be in sync.

So in this one obscure case, the effort to support this unsupported scenario was not, in fact, being supported.

A few early cases that were reported, I myself gave advice that they update the application, because even if/when this was fixed it wouldn't change the fact that what they were doing was wrong -- so their fix should happen and it should be independent of the Windows fix.

Now soon after that, as later cases proved, it turns out that the impersonation scenario was not entirely 100% obscure, as many OEMs using tools to image their machines with specific locales were using impersonation this way.

But there wasn't such a good reason to bend over backward to make this all work, since no supported code in Windows was using the Locale registry value. It was still these outlier cases.

There were some complaints, but the customers who were complaining would readily admit that they were doing the wrong thing.

So it seemed like the case was kind of closed....

Or maybe not.

Because although all the NLS and Regional and Language Options code was no longer using HKCU\Control Panel\International,Locale in any of its initialization routines, that there was at least one bit of supported code that was still [indirectly] using this registry value, at the time of session initialization for the user.

That one function is GetThreadLocale (the one I mention in blogs like Why I think the thread locale really stinks).

At session initialization (as in at logon) for a user, this registry value gets put in the TEB, and unless you change the user locale and change it back, both HKCU\Control Panel\International,Locale and GetThreadLocale will contain that wrong value in this scenario.

And not as a coincidence; one is causing the other.

Strange how unsympathetic I was to people hitting the unsupported registry value and how sympathetic I am to people calling a documented function!

This bug causes some of the random problems Delphi users are seeing like this one and this one -- and probably others if you look; although I find the thread locale to be in its own way evil it may not be the best policy to punish people for using the evil we're giving out just because it is evil.

I've been calling the thread locale evil for years and rally haven't scared very many people off of it yet. Certainly Delphi never got the memo (they really shouldn't be using the thread locale anyway, even if this bug was not happening).

Anyway, not sure yet what will happen here eventually.

For now, the officially sanctioned workaround (via Shawn):

1. Open Regional and Language Options
2. Under "Format" pick anything else. (eg: English (United States)).
3. Press "Apply"
4. Under "Format" pick your desired locale (eg: English (United Kingdom)).
5. Press OK.

But if I hear anything else, I'll holler....

So the other day, the question was:

Hi All,

I am seeing different behavior for currency symbol for Turkey based on regional settings.

We are creating culture like this:

var culture = new System.Globalization.CultureInfo("tr-TR");

If Turkish is selected in Regional settings then Culture.NumberFormat.CurrencySymbol is set "TL" which is correct.
However if English is selected in regional settings then Culture.NumberFormat.CurrencySymbol is incorrectly set as "YTL" which is an old currency symbol.

What would be the expected behavior?

Thanks

The first part of this is explained in .NET is too busy being consistent with Windows to be consistent with itself.... - if Windows and .Net say something different, then when you create a culture based on the same locale as is in Regional and Language Options, you will get whatever Windows has.

Now which of Windows and .Net will say TL here and which ones will say YTL will really depend on the version of each.

And that gets us to the second part.

Now the Turkish folks did something interesting a few years ago.

Probably not as some type of revenge for a certain nameless employee of a company some people consider evil who owns a blog about internationalization smuggling a cat into the country. Because there is really no proof that such a thing ever happened, as both my stated and actual purpose for wanting to be in Ankara was to see a show near the tail end of Jethro Tull's Little Bit of Light Music tour, and nothing untoward happened at or around the time the concert did. The passport itself has been replaced and there is doubt that reliable witnesses could be found to claim I was even in the country. Plus, smuggling a cat on a flight that long would have involved many things beyond capabilities that any normal person would ascribe to me, like intricate knowledge of anesthetic effects on felines not to mention willingness to risk being quite literally sent to a Turkish prison -- the sort of crapshoot I'd typically avoid. Does that sound like something I would do?

Obviously not. You people know me better than that.

I hope.

Anyway, where was I?

Oh yeah, that interesting thing not involving Nikola's cat that the folks in Turkey did.

You can get the skinny on Wikipedia's article on the Turkish lira, in particular the bit on the second lira, excerpted here:

2005–2008

In the transitional period between 1 January 2005 and 31 December 2008, the second lira was officially called "new lira" in Turkey. Coins were introduced in 2005 in denominations of 1, 5, 10, 25 and 50 new (yeni) kuruş and 1 new (yeni) lira. The 1 new kuruş was minted in brass and the 5, 10 and 25 new kuruş in cupro-nickel, whilst the 50 new kuruş and 1 new lira are bimetallic. All coins show portraits of Mustafa Kemal Atatürk.

Since 2009

From 1 January 2009, the "new" was removed from the second lira, its official name in Turkey becoming just "lira" again; new coins without the word "yeni" were introduced in denominations of 1, 5, 10, 25, 50 kuruş and 1 lira.

Now of course the 2005-2008 currency sign was something they wanted to be YTL (that word mentioned above) and then the 2009 change was to take it back to the TL like it was before that, though with the zeroes still gone.

Now Microsoft had put out a Euro update to help update versions of Windows that did not know about countries that switched to the Euro, something I have talked about before in blogs like Hey brother, can you spare a Euro?, a tool that is no longer available under its old name (though you can find it in updates that will even get the latest countries added like Slovakian, Slovenian and Maltese in KB960680 for XP, Server 2003, and other platforms.

I just noticed that the original page for this tool is still up, with the old information, at Microsoft Turkish Lira Currency Symbol Converter Tool. By the time you read this, it may be gone.

Published: November 9, 2004

The Turkish Lira Currency Symbol Converter Tool is designed to update the current Turkish monetary symbol (TL) to the New Turkish Lira symbol (YTL). While the tool will update the currency symbol as soon as it is run, the Turkish government will officially change from the current Turkish Lira to the New Turkish Lira symbol on January 1st, 2005. Therefore, we suggest you run this tool after December 31st, 2004.

Note that this tool's download link (ytl-install-en.exe) points to something that is no longer there, and that the KB article that mainly talks about the Euro's latest updates also updates the Turkish lira issue.

And a few years prior, they first released a LiraConv tool that did the same thing to change the TL to a YTL (to wit: the very download page I just cited!), so it makes sense to change it back.

Now this tool just updates the current locale, and each version of Windows and .Net has shipped with the version of the Turkish currency symbol matching the law of that point in time, so if you consider the versions of each that shipped over the years and the people who have run this tool, whether the Turkish currency sign would be right anytime you didn't allow user overrides or Turkish wasn't your default user locale would be a crapshoot.

On the whole the Turkish people were not very well served by the update, either time, given that confusion. Developers aiming at the market were bound to present an even worse story....

For what it's worth, Nikola's cat passed away a few years ago, at the age of 18 (which is fairly reasonable for a cat if you keep track of such things). 

And the only jail/prison I have ever spent time was not in Turkey and was only for a few hours in Missouri and an honest mistake involving unrealistic speed limits, a 1979 Dodge Diplomat, and a road that was straight as the eye can see where you could literally see 3-4 miles in front of you. It only cost me my AAA card to get out, and somehow it never ended up on my record.

That was a crapshoot too, for what its worth....

Maybe some of this offtopic blather should have been footnoted, or perhaps just skipped entirely? Sorry, what with jury duty and all, last week (which for me writing this blog would be this week) was a really weird week and a this week (which for me writing this clause would be this week since I am adding it minutes before the blog is going live) hasn't made anything seem more normal as of yet....