July, 2008

A question came up the other day that some regular readers might find vaguely familiar:

We have been using the CultureInfo.CurrentUICulture to get the default UI language of the client OS. It works for locales like en-US (which is the default) and for other language locales like de-DE, es-ES, fr-FR, it-IT, ja-JP, ko-KR, pb-BR, ru-RU, zh-CN etc., but NOT getting the locales which are other flavours of English like en-AU etc., instead returning en-US in those cases.

The language locales are picked based on client operating system for the language or if not from the regional settings languages etc.,.

For example, if we set the locale en-AU everywhere in regional settings and in also in MS Office, its still picking up only en-US.

We are trying to get the locale from our Add-in in MS Office using the .NET 1.1.

Please throw some light on how we can get the locales which are other flavours of English.

Ah,. the hazard of being UICulture-based running on Microsoft products!

So maybe you remember one of the following blogs from the past:

• Using full locales rather than the neutral ones?
• About that Portuguese localization question...
• It speaks my language (well, almost!) 
• It's in Spanish? What kind?
• They speak English in other places, too

and more.

The problem is the same -- despite the commonly known and understood fact that these various dialects are not all 100% mutually intelligible, companies like Microsoft, in an effort to save money, tries to enforce a single language version of many different products.

The exceptions to this are few and far between, e.g.:

• Chinese
• Portuguese
• Norwegian (where we actually skipped a Nynorsk LIP, as I described here)

And despite all the lip service (pun intended) that people pay to the need to support "local experiences", despite complaints from former MSFTies like Mike Williams or not-yet-quite-former MSFTies like me, despite the work of cartoons like Darby Conley's Get Fuzzy with the multitude of cats who visit from non-US English speaking places like Manchester that many can't understand and even more random references, no one thinks the problem is bad enough to bother with.

No one wants to "get" the problem here though.

When I think about the nightmares associated with time zones and all the brave efforts to fix longstanding problems that only were able to get traction when they directly impacted people at the executive level in Redmond, I wish some similar solution were possible here -- like localize all of Microsoft's products into UK English or even better Australian English and have these versions on the computers of every executive, technical fellow, and Distinguished Engineer at Microsoft.

How many days would they have to use products while they struggle to understand the words before it would become a mandate to care about local experiences in all of the other places that English, Spanish, French, German, Italian, Chinese, etc. are spoken besides the few places we localize to....

But to be honest I don't see how it could be accomplished. And since no one gets made an executive by finding ways to have stuff cost more money, the problem perpetuates itself.

This blog brought to you by z (U+007a, aka LATIN SMALL LETTER Z)

Just got home a little while ago, and I am definitely BWI (blogging while intoxicated). This is something that a friend of mine warned me about, but I think it'll be okay.

The only real risk is that sometimes you can have an idea that seems like a really good one at the time (since you are drunk) but once you are sober you realize was not as good as you thought it was.

Kind of the beer goggles approach to blogging for software developers in that weird place that I call:

Not too drunk to write code, but way to drunk to be checking code in.

Many developers (as well as some testers and program managers) who have similarly blurred work/life balances will know what I am talking about.

Anyway, I was with some friends at a club and we ended up having a very random conversation.

We were at the bar, so those things can happen.

Someone else who was getting a drink was telling her friend that she had seen What Women Want the other day (that movie I talked about back in I am 20 out of 21 and flexible on the capital punishment issue) and she was asking her friend if she ever was sitting with someone who they suddenly found themselves making out with.

They got their drinks and left, but the question lived on with us.

I pointed out it has happened to me, though I don't usually remember initiating anything -- I am much more of an "almost initiate but back off at the last minute" kind of guy, which allows it to either happen eventually or never happen, depending on the preference of the other person.

While still taking the first -- potentially embarrassing -- step of admitting interest.

The other times that spontaneous make out sessions happened, it was either completely the other person, or maybe no one initiated and it just happened. Like spontaneous combustion or something.

And no, this is not a "being drunk" kind of thing -- I learned years ago to keep it holstered when drinking; it is just better for everyone.

For example, I didn't make out with anyone tonight. :-)

It reminds me of an incident from nearly two decades ago -- I was at a party at Johanna's house, the last party where I ever seriously drank beer. I had way too much, and I was sitting with Johanna out on the stoop. Suddenly I realized something:

Michael: Jo, can I ask you something?
Johanna: Sure.
Michael: I have to throw up now. Should I go (a) go inside to the toilet, (b) go behind the bushes over there, or (c) do it right here on the sidewalk?
<<pause while Johanna, who was also pretty drunk, thinks about this>>
Johanna: If you do it out here then people will get it on their shoes later when they leave. But if you try to go inside you may not make it due to lines. I'd go with the bushes.
Michael: Very sound reasoning. will you excuse me for a moment?
Johanna: Certainly.
<<pause while I go off to throw up>>

Now what was most odd about this was the way that a pressing need/want to do something came up, yet there was a surrealistic pause to analyze the issue and weigh options. Who the hell takes the time, or at least spend the time they have in that particular way?

But have you ever found yourself in one of those situations where you are irrevocably committed to that kiss that you know will become a make-out session in a real Liz Phair Why Can't I? sort of way (ref: here and here), yet you take the time beforehand to analyze it with the other person?

I am not sure what the hell this is -- I mean the only thing you can really accomplish here is to talk yourself or the other person out of it. Maybe it is intentional auto-sabotage? But it has happened, and it kind of makes me think that these rare spontaneous make-out sessions weren't my idea (since if they were I wouldn't trying to be talking anyone out of anything).

Though it isn't like this is such a common occurrence that I have a real statistical universe from which to draw conclusions.

But I was having lunch with a friend the other day, and I remember at a previous lunch she related something like this happening at a party. Though she hadn't mentioned any attempts to talk anyone out of anything so I suspect that maybe this is just me (or anyone who thinks themselves generally unworthy?).

The decision of the group of people (my friends who were there and the people sitting around us who got into the conversation) was that for most people it only happens when one or both of the people involved have been drinking, and there are seldom huge conversations beforehand. for normal people the spontaneous make out sessions are rare but when they do happen they are truly spontaneous make out sessions.

The secondary conclusion (based on my experiences and one other guy's -- a guy who was also a software developer) was that software developers are the only ones who would make the mistake of talking their way out of it, of snatching defeat from the jaws of victory and wearing it like a shawl. And that we should really try to work on that.

I agreed to take it under advisement (though the situation is not all that common these days so it feels like a fairly theoretical point....

So how about regular readers -- any spontaneous kissing with people you haven't kissed before?

And if so, is it really spontaneous or so you do your damnedest to talk your way out of it first?

And finally, are you a software developer? :-)

This blog brought to you by ? (U+003f, aka QUESTION MARK)

Previous blogs in this series:

• 0: A long journey begins with the zeroeth step
• 1: Do I know you, or some version of you at least?

This next part of the series is about a phenomenon that I hope most people are unfamiliar with, whether in their personal or professional lives.

In one's professional life, it is usually less than ideal to be in a pure do you want fries with that? kind of situation where there is really nothing beyond where you are being used and for how long. Thus even feeling the question is appropriate to ask is a sign that one might be in the wrong job.

And in one's personal life, it is always a net negative kind of a sign if one is being used, only made slightly less negative if one is aware of it (since one can then change the situation). But the ultimate low point would likely be feeling the title question bubble up.

I won't talk about the few times (very few times) this has come up in my personal life in this blog. Maybe some other day in this Blog. Though holding one's breath waiting might be a poor bet....

Ditto on the professional side of this atomic wedgie inducing fence.

But I'll talk about an aspect of the question that one wants one's code to be able to ask any time one wants to remove or replace a component.

I run across this issue quite a bit with keyboard layouts, and it is even harder to handle when one is dealing with fonts.

The latter is the subject of today's blog (the former might be the subject of some future blog).

So about asking a font whether anyone is still using it....

Ironically, the font is no better off than the two situations I hinted at in the beginning of this blog -- there is really no way to ask the question. There is, quite simply, no way to find out if the font is in use by the system. At all.

The files may not be locked (the various components that use fonts tend to open the file, cache some information like metrics of the font and glyphs, keep weak pointers to other information for easy lookup later, and then closes the file. This all happens very quick, so the odds of the file being in use at the time are not very good; your only real hope if it is in use is that it will not be asking for any glyphs or other information not already cached -- which includes new sizes or other attributes of the same font.

This causes two different somewhat intractable problems:

1) Obviously if one is removing a font that is in use one will have GDI or whomever pointing to something that isn't there. This is really quite unlikely to crash or blue screen because of years of hardening, but it is quite capable of causing strange problems with processes that assume the font is still valid.

2) If you try to install a font and an earlier version of the font has been determined to exist (see Part 1 for more on the version stuff), there is no way to know if the font is currently in use. If it is then there is no way to know the impact of replacing the font file due to orphaning all of those references to information in the font.

Paul Linnerud relayed in a document an anecdote that helps underscore the consequences of these problems here:

...were to replace the font file with an updated file, Windows would not be aware of this change and would still assume the cached pointers were valid. If an application attempts to access the font for more glyphs than are cached or withes to change size unexpected results will occur and they may not occur for some period of time. The first time I saw this, the font Tahoma was replaced and was in use by Microsoft Mail. About ½ hour after the font was updated, suddenly all the lower case characters using that font would not display.

There are even worse scenarios you could imagine here, I'm sure. I am quite bullish about the safety aspects here -- the code has been quite throughly hardened here to keep terrible results from happening. But I am not so bullish about potential appearance problems; in fact I am really quite bearish. And the problems are (ironically given the word usage) quite unbearable in many situations.

You may have noticed the interesting consequences of the problems hinted at here occasionally in the past without really attributing a specific cause. A reboot will obviously fix it so I suppose you could just attribute it to @#%&*! Microsoft software and/or operating systems and call it a day. :-)

But the deeper issue remains -- if you remove or replace an installed font file you have no real way to know the potential impact on anything that may have been using or may still be using (or even may plan to use) the font.

Again I will talk about some partial mitigations for all of this in a future blog in this series, as well as some "apparent" solutions that can actually be much worse.

At the core of it all is that any time one can reasonably desire to ask the question Are you done using me yet?, one is in a potentially bad and possibly weird situation until and unless one gets the opportunity to reboot....

This blog brought to you by Ƒ (U+0191, aka LATIN CAPITAL LETTER F WITH HOOK)

The question that came to a managed code alias was easy enough to see:

How can I either (1) detect that a system environment variable has changed, or (2) get the current system-wide setting of an environment variable (not necessarily the value of the variable as it was set when my app was launched)?

Windows apps are notified via the WM_WININICHANGE, but I’m looking for a way to do it in a .NET app.

I'll deal with the second paragraph first -- why is it that everyone wants to do stuff the ".NET way" even hen asking a purely Windows question that even when there is a 100% managed way one has to realize that under the covers there will be numerous native things going on, whether pinvokes or other. Is this a naive code purity thing, a desire to eat the steak without thinking about the cow?

Okay, we'll move past that. Someone ignored the manged-only piece and suggested they take a look at the WM_SETTINGCHANGE message, but there was a less than positive response to that idea:

Thanks for the response.

I don’t know how the... link will help---when I change an environment variable, the WM_WININICHANGE message is sent to all top-level windows (tested on Vista), and not the WM_SETTINGCHANGE message.

Also, one question I forgot to ask is: if I only can capture some notification that the system environment changed, then that’s not enough; how do I get the contents of the new system environment?

I laughed when I saw that one, I mean what with the following defined in winuser.h since the late 90's:

#define WM_WININICHANGE                 0x001A
#if(WINVER >= 0x0400)
#define WM_SETTINGCHANGE                WM_WININICHANGE
#endif /* WINVER >= 0x0400 */

Classic!

Like that old Bill Tush skit with the teenagers who argue because one loves the words but hates the lyrics, while the other insists that the lyrics are awesome but the words suck.

Only not contrived!

In fairness, the WM_WININICHANGE message and the WM_SETTINGCHANGE message don't go out of their way to help here -- they in fact seem designed to do the opposite, really.

You know, like they put on their Eagles hats and sing "Relax, said the night man, we are programmed to deceive" without realizing they have the words wrong; they don't realize the topics are writing checks that the header files can't cash....

Whether one looks at the amusing text in the WM_WININICHANGE message:

An application sends the WM_WININICHANGE message to all top-level windows after making a change to the WIN.INI file. The SystemParametersInfo function sends this message after an application uses the function to change a setting in WIN.INI.

Note  The WM_WININICHANGE message is provided only for compatibility with earlier versions of the system. Applications should use the WM_SETTINGCHANGE message.

With a claim that the wParam is not used and just a small message about the lParam:

A pointer to a string containing the name of the system parameter that was changed. For example, this string can be the name of a registry key or the name of a section in the Win.ini file. This parameter is not particularly useful in determining which system parameter changed. For example, when the string is a registry name, it typically indicates only the leaf node in the registry, not the whole path. In addition, some applications send this message with lParam set to NULL. In general, when you receive this message, you should check and reload any system parameter settings that are used by your application.

All of which is most amusing when you look at the very different text in the WM_SETTINGCHANGE message that gives numerous details about the values of both parameters.

Though ultimately the funniest part being the requirements for both functions:

Client	Requires Windows Vista, Windows XP, or Windows 2000 Professional.
Server	Requires Windows Server 2008, Windows Server 2003, or Windows 2000 Server.

This is something that makes very little sense given just the header file alone, let alone the pragmatic knowledge between the two topics and the past knowledge of many a Windows developer. :-)

The language of the documentation, it can be its own dialect of English at times....

No Unicode character wanted to support this blog except some of the old ones that were removed from Unicode as a part of the 10646 merger

In this Blog, some topics are harder than others.

It was born out of a question, left in the Suggestion box (at my recommendation) by keoladonaghy:

Could you possiblly create a topic on how to create an installer package for  fonts. I have one that works on XP using Installer Creator, but it doesn't work on Vista. I've read a few forums that talk about elevated status and such in order to be able to write to the Fonts folkder. Even when using an admin account and turned off User Account Control, the installer runs and tells you that the fonts have been installed, but when you look in the fonts folder they have not. I'm hoping to find something cheap (or free!) and simple as this is the only Windows installer I need to create.

[ref: About the Fonts folder in Windows, Part 3 (aka What changes in Vista?)]

A moment ago, when I was talking about topics that were hard?

Well, this is one of those.

So I am going to tackle it in a series of blogs, all inside this blog.

About the challenges, the issues, how to do it right by pointing out a lot of the wrongness that goes on, and so on.

Think of this blog as filler -- it accomplishes nothing, or maybe almost nothing.

This is, after all, a statement of intent to have more blogs on this topic.

Are you excited yet? I mean in the professional sense?

I should warn you that I am going to seem very impressive here, because I am going to standing on the shoulders of some really impressive people who have forgotten more about this topic than I (and let's be honest pretty much everyone else) will ever know.

So if there are mistakes along the way, they will indubitably be mine.

And anything great or cool or insightful or interesting or intriguing will be those impressive folks I was just alluding to.

There may be the odd personal reference too, because I'm just kind of built that way -- you can ignore those bits if they aren't your thing or saor them if they are.

You know where we are, right now?

We are at the beginning.

If you are reading this live rather than via a web search, then you got be here when it all started.

Welcome!

This blog brought to you by Ｆ (U+FF26, aka FULLWIDTH LATIN CAPITAL LETTER F)

So thinking about the design of MUI, some interesting thoughts came up in a conversation the other day.

Let's take NOTEPAD.EXE for a second.

If we move to the WINDOWS directory and hit ALT+ENTER we get it's Properties:

Let's move over to that Details tab:

Hmmm. I thought that we were all language-neutral now. Why is the file' version resource claiming that the file has a language of English (United States), anyway?

Let's look over in the language-specific directory and see what the NOTEPAD.EXE.MUI file looks like there:

Wow, no VERSION properties at all!

Wait, maybe that is an issue with the filetype. Let's copy the file and remove the .MUI extension:

 and try again:

Ah, there we go. So it has a VERSION resource and the language is tagged.

Of course this proves nothing -- the original file makes the same claim, even though all of its resources are gone (as we learned in Random irreverent thoughts about the Ultimate Fallback).

So let's look at some of the other language files, like Arabic:

or maybe Hebrew:

And maybe we could mix things up a bit.

Like looking at the Arabic file under a Hebrew user interface language:

or the Hebrew file under the Arabic user interface language:

Okay, so the language files are being marked.

The only weird thing left is that the language neutral file, the one that even Mark Russinovich discovered the hard way is language neutral (ref: Random irreverent thoughts about the Ultimate Fallback), is marked as having a language.

Well, it turns out that the language splitting functionality in the Resource Compiler (RC.EXE and RCDLL.DLL).

Now RC.EXE has several flags related to creating .MUI files:

/fm mresname

RC creates one language-neutral .RES file and one language-dependent (MUI) .RES file using script-file. This option must be used together with the /fo resname option. RC names the language-neutral .RES file resname.res and names the language-dependent (MUI) .RES file mresname.res.

/fo resname

RC creates a .RES file named resname using script-file.

If the /fm mresname option is also set, RC creates one language-neutral .RES file and one language-dependent (MUI) .RES file.

/g1

If /g1, is set, RC generates a MUI file if the only localizable resource being included in the MUI file is a version resource. If /g1 is not set, RC will not generate a MUI file if the only localizable resource being included in the MUI file is a version resource.

/j loctype

Localizable resource types RC places into the language-dependent (MUI) .RES file. If the /q option is also set, this option is ignored, and the information in the RC Configuration file takes precedence.

/k overtype

Overlapping resource types that RC places into both the language-neutral .RES and the language-dependent (MUI).RES files. The resource types that are specified by the /k option must be a subset of those that are specified by the /j option. For example, –J2 –J3 –K3 specifies that RC places resource type 3 in both the language-neutral and language-dependent (MUI) files. If the /q option is also set, this option is ignored, and the information in the RC Configuration file takes precedence.

/q Mui.RCConfig

An RC configuration file that follows the RC Configuration File format. The RC Configuration File format enables components to self-describe resource information such as resource versioning, MUI file path, resource types and items. This file specifies which resources go into the language-neutral .RES file and which resources go into the language-dependent (MUI) .RES file. This option, and the information provided in the RC Configuration file, override the command line options /j and /k.

Notice that in none of that talk of splitting resources does it ever make any claims about changing the language of the "neutral" .RES file it creates as part of the splitting.

Not that it wouldn't make sense for it to do that (since it took the time cause a de facto change to the language of the resource by removing all of the language-specific information), but that work item would have some interesting consequences, which I can talk about more some other time, perhaps....

This post brought to you by ຝ and ພ (U+0e9d and U+0e9e, a.k.a. LAO LETTER FO TAM and LAO LETTER PHO TAM)

Previous blogs in this series:

0: A long journey begins with the zeroeth step

One of the first things people do when they enter the room is make themselves known.

If everyone knows everyone else they don't even need names, otherwise they give their name.

If they are twins (and this has happened to me, within the last few years!) the twin who is there will identify which "version" they are.

The version thing is pretty important, as you can see.

Now version information is pretty straightforward in Windows....

Hell, I can't even say that with a straight face.

You can look at Raymond Chen's Blog, and blogs like this one and that one and this other one if you want a hint about how complicated it is.

If you are internal to MS you might have seen the dozens of small utilities put up by developers and testers over the years, almost all of which properly update at a maximum just one of the two items to update (the string version of the resources and the binary one), and some did not even get one of them right.

So version stuff on binaries is tough in Windows.

But tough as it is, it is actually a lot easier than the version information in fonts, unless you are writing an application that walks through the font file data ad its purpose in life.

Because in order to get the font version information, you have to walk through font file data.

Specifically, you are looking for the name table.

This table, this name table, has an entry in it that specifies the version.

The entry might just be 1.0. Or it might be 1.0.3. Or it might be 867.5309. Or even 3.1415926.

Crap, it could be Version 3.14. Or it could use the word Versio, or Versión, or Versione. You get the point.

Or really anything -- it is a string, after all.

Maybe we can hope for valid characters....

The exact format of the string can be anything, and a font vendor/foundry can even change the format they use between versions, if they want. If they do they might even have a good reason for wanting to do that. But even if they did not, they can still do it.

Hell, two different fonts from two different places can have the same name and the first of many comparisons you might do is with the name and the version. Just dig right in, like I did in Getting all of the localized names of a font but perhaps much more efficiently. :-)

If you are building something that has the job to install a font and you need to know if a font already there is a prior version or a later version, you have to be able to parse that information so you can compare the two (and decide which one is newer).

I find myself grateful about the fact that the twins I referred to previously were in two different disciplines - development and test -- and therefore I did not fancy heuristics to tell them apart; merely listening to what they opinions were would often quickly identify the one present, if the meeting itself did not!

No such boon can be assumed with fonts. Your generic awesome font installation code needs to assume the worst.

Since the worst can indeed happen.

Okay, this seems like a fair enough start, hinting at inner depths. Next time I'll dig into some of those depths a bit....

This blog brought to you by ｆ (U+FF46, aka FULLWIDTH LATIN SMALL LETTER F)

Remember those posters that said "Today is the first day of the rest of your life"? Well, that's true of every day but one -- the day you die. -- Lester Burnham (Kevin Spacey)

Every rule has an exception.

Like the general rule that it is a bad idea to hard code strings in a call to MessageBoxW -- you should always load the string from resources so that your localizer has the opportunity to translate the text.

This is a very good and sensible rule, and it was just yesterday that several of us were reviewing bug descriptions that were entered to the bug database by a tool that detects such problems.

But while looking through the valid reports of bugs that various developers were going to have to fix, we found one case that is the exception to this rule.

This good and sensible rule.

Can you guess what it might be?

The one time that this rule isn't true?

Some people might think it is an ASSERT function. Good guess, but no. That isn't it.

The exception is when the error is something like this one:

Catastrophic Error: Unable to load message.

And the call comes after the attempt to load the error string in question fails.

This is a great exception to the rule.

I mean, is there ever a better time to point that we could hardly expect a developer to load that message from a resource....

Though to be honest there is a flaw in this specific description of otherwise unassailable logic.

Can anyone guess what that flaw might be?

This blog brought to you by ⧮ (U+29ee, aka ERROR-BARRED WHITE SQUARE)

Dedicated to an Easter Egg poem, and its author...

Previous blogs in this series:

• 0: A long journey begins with the zeroeth step
• 1: Do I know you, or some version of you at least?
• 2: Are you done using me yet?

This blog in the series is a slight detour....

And I am going to go slightly out of order now.

The off-course jaunt started in my mind last time in a response to Part 2, regular reader Andrew West commented:

Surely the font installer does not need to know who exactly may or may not be using the font; it just broadcasts the WM_FONTCHANGE message and well-behaved applications will handle it and update their font information accordingly.

Now Andrew is quite right -- WM_FONTCHANGE is just the message that a responsible installer/uninstaller would send here.

Unfortunately there are many problems with the architecture -- some the fault of installers/uninstallers that are indirectly the fault of the difficulty of the task, and others the fault of the complex process that nevertheless has failed to keep up with some of the complexities of the underlying platform:

• Lots of installers out there tend to be about as responsible as Nathan Scott's mom when she's bombed;
• Even more uninstallers out there tend to be worse;
• Since install and uninstall have detectable consequences, forcing this extra step is a little unfair;
• Starting in Vista, the work to mitigate shatter attacks keeps a process from communicating with elevated processes -- meaning the more important an application is, the less likely it is to get the message;
• The message does not go outside the session, meaning any other session remains unaware of the change -- woe to those who make these changes on a Terminal Server or the world of Fast User Switching!

You get the idea.

The design here is quite sensible for a bygone era, but to be honest something better might really be considered.

But I digress.

The point is that even a "well-behaved application" that sends the message and a "well-behaved application" that receive it might not ever be able to communicate well enough to be told that something is up.

Despite the fact that the contents of a directory and the contents of a registry key were altered.

So in the end, the rules (if you work within the framework above) end up being quite simple:

1. If the font is there and the new one is determined to be the same font but of a lower or equal version , then you do not have to do anything;
2. If the font was not there, then you can add it, add the registry information, and send the message; everything is as good as it can be, but other sessions and elevated processes might have to wait until next time;
3. If the font is there and the new one is determined to be the same font but of a greater version, then you have to do all of the install work but it is much safer to require the reboot and do the actual file replacement at boot time.

So pretty much the only time that you never have to do anything is when the font in question is already installed. when you didn't have to do anything.

Yeesh. Talk about over-engineered! :-)

In the next part of the series we'll talk about the mechanics of the install and uninstall that must happen....

And in a later blog I will talk about my opinion of regular reader John Cowan's suggested "solution", here. :-)

This blog brought to you by ƒ (U+0192, aka LATIN SMALL LETTER F WITH HOOK)

Julien asks via the Contact link:

Dear Mr Kaplan,

I would like to display Japanese Characters in the reporting solution of my company. The DB is SQL Server 2000 and datasource is an ERP wish provide data in DBCS format. Default Collation is French_CI_AS on server and client machines.

Stored data in varchar format appeared on this way in Enterprise Manager :

à_áÉá?áZ áEáìáóá,án
<<strange characters removed due to reports that they break the feed>
Is it possible to display Japanese Character and how?

Thanks a lot for your help and information you broadcast on the web on this topic.

Julien, from France

Well, first of all, I'm not Mr. Kaplan. I was gonna say that my father is Mr. Kaplan but now that I was about to type that I realized that I don't even think of my father as Mr. Kaplan -- I don't even think of the VP name Kaplan as Mr. Kaplan since he sent mail not too long ago to all the Kaplans at Microsoft and he signed the mail with first name. Essentially, there are no Mr. Kaplans in my world at the moment! :-)

Julien, you can call me Michael; all the people I like do....

Okay, now back to the topic.

This is a rough situation.

Basically you have non-Unicode data that is stored in a column with a specific defined code page, in this case that would be COLLATIONPROPERTY('FRENCH_CI_AS', 'CodePage') or 1252.

And the data itself is actually in some other code page.

Offhand, the first problem is the one I mention in 100% roundtrip ASCII? 100% roundtrip ANSI?: that these code pages have byte sequences that are not valid in them. And since their "invalid" bits do not always overlap, one can lose data if one (using Unicode as a pivot) moves in and out of them.

So the only "fix" would be to take the data out as is (say via direct BCP transfer that does not go through COM or anything else that will try to convert it to Unicode), with a plan to later on bring it back in using preferably Unicode columns but if all else fails one that can handle Japanese data.

But the Unicode columns would be best in any case. If you look at all of the characters that the Japanese standards define as characters used by Japanese, only Unicode can store them all in a SQL Server  column. And the Japanese and I have in common a dislike for valid characters being converted into question marks, if you know what I mean.

And now we get to the next problem -- with the small sample Julien provided, I couldn't produce something that looked all that Japanese, testing with either shift-JIS or EUC-JAP. so someone would have to probably give some information about what encoding the data is in for sure, so it could then be converted.

Without knowing the encoding it is in, how could one ever hope for a computer program to read it?

This post brought to you by ¿ (U+00bf, INVERTED QUESTION MARK)

• 0: A long journey begins with the zeroeth step
• 1: Do I know you, or some version of you at least?
• 2: Are you done using me yet?
• 3: I'll love you forever, or at least for this session.
• 4: The easiest part is the addition!
• 5: The takeaway from that is...

Now last time I hinted that there was yet another problem related to font removal that has to be considered.

But let me back up for a moment.

It is a problem I hinted at in Part 1 when I mentioned in passing:

Hell, two different fonts from two different places can have the same name and the first of many comparisons you might do is with the name and the version.

I should watch the cussing with all of the minors lurking about, but ignoring that for a damn moment, the implication (or maybe it is more of an imputation) is obvious.

Behind each font is a filename.

A filename that might be the same as some identically named file.

As long as the "get rid of file extension" zealots keep not getting their way you can have some confidence that identically named file is also a font, but the facts remain the same -- someone might be squatting on your turf with their own file that callously has the same name as yours and which obnoxiously happened to be installed first. Bastards that these people were, they probably also killed Kenny...

But the problem remains -- when you install you have to deal with the fact that some other font file name is the same.

Now AddFontResource only takes a filename, but the font itself can have all sort of names in it -- the names that will be "registered" for the font so that it can be found later.

Aha, so if the file is already there and you determine that it is not some earlier version of yours to be replaced (see Part 1 for more information on that) then all your setup program has to do is rename the file and then you have none of the problems with someone who happens to have the same name.

So everything is good, right?

Well, kind of. With the system trying to maximize the chances of a font trying to get installed. Everything is good....

Until problems happen.

I return once again to Paul Linnerud, and an example explained by him:

...the system has Palatino installed as palatino.ttf. The user drags Palatino Linotype with file name palatino.ttf into the fonts folder. The fonts folder will determine that palatino.ttf is already used by a font and rename the Palatino Linotype file as pala001.ttf. Later if the installation program wishes to install Palatino Linotype with palatino.ttf it will really replace Palatino and we will end up with two Palatino Linotype fonts installed with different file names.

And yes, the font folder of some versions of Windows has that renaming code -- and this problem really can happen. Even if it didn't, a font installer can do the very same thing. What will your installer/uninstaller do if if finds that the file it is trying to remove/replace is not the file it is meant to touch but someone else's?

Go on, guess.

If your guess is anything along the lines of "not what you want it to" then you get a gold star. Since most won't....

The name of the file and the name(s) of the font itself are just entirely separate beasts -- even the updating operation can be dicey and you will want to really make sure it is your font.

So How can all of these problems be solved?, you may be asking.

Or Can these problems be solved? might be the question on the tip of your tongue.

Stay tuned and we'll start talking solutions, next time....

This blog brought to you by Ϥ (U+03e4, aka COPTIC CAPITAL LETTER FEI)

Yesterday in The keys are so hot, they're smoking!, I did something that you won't see me doing very often.

I waxed on at length about a feature as a really exciting thing for people to use.

I'll now admit that when I did that, it was mostly a setup.

Because today I am going to prove that:

• I wouldn't make the worst tester in the world
• Even Vista has a bug or two in it
• The "Hot keys for input languages" feature has weird bugs in it

So we will start by adding three keyboards under three different languages to my usual setup -- in this case Latvian, Lithuanian, and Luxembourgish (The "L" Word languages in Europe):

This will give us the following view in that Advanced Key Settings tab:

Because this is Vista we will get the three shift states and twelve character choices:

The two characters at the end are ~ (U+0073, aka TILDE) and ` (U+0060, aka GRAVE ACCENT).

We're gonna need a few more keyboards here for what I want to do. Let's go back and add three more, under those same languages (just for fun):

Okay, this will give us the list to die for:

Six entries plus the English I started with.

Okay now, to quote Antonio Banderas from Desperado right before he started shooting up the bar, Let's Play.

Among these six keyboards, let's use that Change Key Sequence... dialog, to add hot keys to each of them:

Okay, do you see What we have here?

• Ctrl + ~
• Ctrl + `
• Ctrl + Shift ~
• Ctrl + Shift + `
• Left Alt + Shift ~
• Left Alt + Shift + `

Now if you have a US keyboard you might notice the problem here.

Yes, that is right -- the tilde is the shift state to the grave's base state.

In other words, there are not six unique states here at all -- the user interface just thinks there is. All of the TILDE entries without a SHIFT in them have an implicit one built in, and all of the GRAVE entries with a SHIFT do not exist since the SHIFT changes the characters.

Oops!

Let's hit the Apply button now and see what happens:

Uh oh! Somebody silently removed our CTRL + GRAVE ACCENT that was on the Latvian keyboard.

But the rest were left.

Let's fix it and try again:

Okay, that one stayed fixed but now two others are gone.

Third time, will it be a charm? Let's see:

Um, not exactly -- not only is one missing, but another one is now duplicated on the list but without the hot key. And we can't add it back either as it is no longer on the list:

Weird.

And kind of broken, given that we have now broken the UI and half of the ones that are there are not going to work the way we want them to anyway.

If you keep doing this, eventually you hit a non-steady steady state of something like this:

Ick.

But let's pretend that this is not a problem for a moment.

And we'll look at some keyboard layouts:

All of this kind of underscores the point from Punctuation keys can make lousy shortcuts, because even if we do not know whether these hot keys are CHARACTER based or VIRTUAL KEY based, we know that neither the characters nor the virtual keys can be relied on for being in all keyboards or in the same positions even when the are present.

Sometimes they are there  but the VK values are different, sometimes they are on other keys, sometimes they are actually dead keys (and believe me the interaction when they are dead keys almost deserve their own bug report!).

So basically between half and all of these entries are going to be kind of broken, depending on the layout you choose.

And of course there are those duplicated entries to help us keep it all in perspective:

Then we can try Estonian instead since both the character and the VK are different there:

As this will give us more somewhat broken entries for hot keys, unless you do them this way:

Though even if you do them right for the English keyboard (to switch to Estonian), you may not have them right for the Estonian keyboard (to switch to English) which does not have them both there.

Not much room for gray area here -- there are just a whole heap of bugs here that have as far as I know never been reported and some of which have existed for any versions.

Ick.

Kind of provides a possible answer to Centaur's wondering:

I wonder what is the target audience for these hot keys.

Fictional users? :-)

I'll talk about the next problem tomorrow....

(Keep in mind we are keeping away from the IME topic, for now!)

This blog brought to you by ` and ~ (U+0060 and U+007e, aka GRAVE ACCENT and TILDE)

Yesterday in The situation was quite grave when I realized how "tepid" those hot keys were, I pointed out a whole bunch of interesting little problems with the "Hot keys for Input Languages" feature in XP and Vista. And I promised there would be more.

There is.

First, you have to take advantage of the ability to add a lot of keyboards. I took every single locale in either Europe or Russia or places surrounding, and almost ever keyboard listed under each.

I basically had 89 locales and 149 keyboard layouts - more than any human would likely ever need.

But I am trying to really show a problem and an extreme example is often the best way to do that.

Now first of all, the language bar is no longer the most usable thing in the world:

Notice that it has no scroll bar so all of the languages that are not there cannot be reached.

That is not the bug I had in mind, but this is also kind of, if not a  bug, then a real design limitation that you are allowed to create something that you cannot use.

But to get the real problem, let's look at the Advanced Key Settings tab:

89 locales, 149 keyboards. Remember?

Well, as it turns out, only the first 32 ever show up:

The other 117 cannot ever be set in the user interface.

Now there is a limitation of 32 hot keys that can be set, which is fine.

But in this case the user interface lets you add more than 32 keyboards, and it won't even let you choose between all of them!

Now the order of the list is another interesting bug/problem -- it is LCID (well really PRIMARYLANGID) based, which is kind of crazy in terms of usability. Tobe honest it is kind of another bug all on its own, come to think of it.

But for example in my case, I better not have wanted to include any Romanian or Russian or other keyboards lower on the list for hot keys!

There is a big difference to only being allowed to set 32 of something and only being given 32 options to choose from....

Maybe someone could teach that to this component. :-)

Workarounds here have real challenges blocking them, though. Covering them might be a good idea at some point (with the inability to easily select a keyboard in some cases, the ability assign a hot key becomes much more important!).

This blog (like the one before it) brought to you by ` and ~ (U+0060 and U+007e, aka GRAVE ACCENT and TILDE)

So I was at the Woodland Park Zoo tonight, where Aimee Mann was playing.

She has missed Seattle her last couple of times coming through this part of the country, so it was nice to her and hear some of the songs from the new album, live.

Though as venues go, it isn't my favorite.

There are just way too many people there who seem much more interested in going to the zoo with their kids and getting to take them to a show that they can get into free than people who are actually interested in listening to music, you know?

I don't mind kids. I used to babysit, I have worked in after-school programs and day care centers. I was even a nanny for a year!

But taking them to see an Aimee Mann show means that you have to be okay if the occasional curse word comes out of her mourh, since some of the words do....

Plus it was weird but the place I was sitting had a huge bunch of Marc Cohn fans rather than Aimee Mann fans, which was also a little distracting....

I ran into Kevin and Keith there and ended up sitting with them which was cool. Apparently we all like Aimee's music (it must drive Cathy nuts, now that I think about it; sorry, Cathy!).

I also learned something very important tonight.

You see I wanted to get the setlist for the show but I forgot a pen.

So I decided to use the facebook status updates from my phone to record them.

What I discovered was that I would hate Twitter -- I can't even stand updating the status line this often!

But it ended up being a good show. I'm going to have to keep listening to the new album to see if gets me; I really did like some of the songs live that I hadn't gotten into before....

The setlist:

• Stranger into Starman
• Looking for Nothing
• Freeway
• Phoenix
• Great Beyond
• Save Me
• Wise Up
• <an Elton John song I have blocked out>
• Goodbye Caroline
• Medicine Wheel
• Deathly
• Thirty One Today
• How am I Different?

There is potential here -- maybe I should head down to the Portland show where the venue will be less distracting to me, and see the show there....

This blog brought to you by no Unicode character in particular....

Previous blogs in this series:

• 0: A long journey begins with the zeroeth step
• 1: Do I know you, or some version of you at least?
• 2: Are you done using me yet?
• 3: I'll love you forever, or at least for this session.

So far in this series I have glossed over the actual installation and uninstallation tasks that have to happen.

This blog will talk about the installation half.

Basically there are two main groups of steps:

First of all, the font has to be placed in a nice shared location everyone/thing else on the machine can get to, and an AddFontResource function call has to be made on the font so that the font is available to everyone, and the WM_FONTCHANGE message should be sent around so that others within the session will know about it.

This takes care of the HERE and NOW.

For the moment we will ignore all the problems I talked about related to replacing existing fonts in Part 2 and with folks running on the machine outside the scope of the current session and security level that made up so much of Part 3. We'll talk more about them later.

Second of all, the font has to be listed in the appropriate registry key so that future sessions can pick up the font. The system will then call AddFontResource for all of sessions created in the future and they will all wok and everyone can be happy and joyful.

This takes care of the EVERYWHERE and LATER.

The registry location is:

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Fonts

on NT-based platforms, and

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Fonts

on Win9x-based ones.

And a pox on the crops of whoever decided to make these two keys different. :-)

Once upon a time, the best practices for what to put in the registry was something like this:

The registry key will look like this for Arial Bold “Arial Bold (TrueType) = arialbd.ttf’.  In the font, the face and style may be localized and that localized name is what should be used on a system with that given locale. When determining what string to use the setup program should look at the locale id of the system was localized to. The system function to get the system locale and user locale are changeable on a system. The actual locale stamped into gdi32.dll or another system component should be used to determine what language name string to use.

Though of course the issues of localized font names are much simpler now, as I mentioned in blogs like East Asian Font Names and Getting all of the localized names of a font and Is Font.FontFamily localized? and Getting all of the localized names of a font[.NET]. You no longer have to care as much about the name you put in the registry, as GDI now does the extra work to register multiple versions of the font under he different language names in the fonts themselves.

At this point, starting with Windows 2000, the name in the registry is not nearly as important as just having the font file name in there correctly.

Remember this point, we are going to talk about this later when we start looking into solutions in future blogs in this series....

Anyway, install that simple.

To the extent that the above is simple?

It certainly seems much less troubling than earlier parts of the series!

This blog brought to you by Ѳ (U+0472, aka CYRILLIC CAPITAL LETTER FITA)