March, 2008

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)!
Warning: This particular blog consists mostly of self-indulgent meta-blogging crap and is thus almost entirely ignorable.
Note: You know those times when almost anything you would say is going to be the wrong thing? This is one of those times. Thus, while comments are not disabled in this post, they are strongly discouraged....

Apologies for the X-Files allusion in the title, as softened by the accompanying eroteme.

The question about blogging about matters so closely tied to my work when my employer (or at least my management) wants it to be made crystal clear that this is a personal blog has become an interesting one, to me.

Yet if I only posted about personal stuff, then people might find hosting it on blogs.msdn.com might be inappropriate.

So clearly by some metric this blog is about my work and my job, even though the people who manage my job are denying me three times and all that.

Which is not to suggest that I am anyone's Jesus or anything; you know what I mean.

But in the aftermath of what happened and what is, I am left wondering what this blog really should be.

Or even whether it should be anything, while it sits on this server and while I sit where I do, job wise.

I have been largely coasting for the last few months on existing items I said I'd talk about and such, but without a real sense of direction.

Avoiding the question.

I could just say I miss Liz and leave it at that, but this isn't about her.

Well, not really about her, at least.

But once upon a time she did provide something of a passive compass for me that I only sometimes consulted but always knew was there, and I admit I'm having trouble figuring out where North is at the moment, without that compass.

Though I can't truthfully claim to be unhappy. In fact, I am not unhappy.

Professionally speaking I have been rather pleased and happy to find that even though my management (in a burst of wankitude/wingnuttery that I could calculate if my slide rule was not in storage in a box in Philadelphia) doesn't appreciate me, that a lot of other people and groups do -- it keeps me feeling good about Microsoft, at least. And of a lot of the people and groups in it. And I do believe on the work that happens in the group and in the building.

And I believe in a lot of other things too.

I could likely keep coasting with the blog posting for a while, as that huge mound-o-things on my To-Do list is still there. With probably a dozen written posts and twice as many half written, and then with three or four times as many that are almost fully composed in my head and all I have to do is dictate them and thereby insert the expected typos/speechos.

Though there is no rule that says that all of them, or even any of them, have to be posted.

When I think about how the Blog is aimlessly wandering without a theme, while each blog continues to have some point, I wonder. Do people notice that lack of overall direction as opposed to the original slowly changing overall direction, all in a see of for the most part directed slices of time?

The quote that pops into my mind: The assassin's gun may believe it is the surgeon's scalpel, but the assassin must know the task.

At the moment I am unsure. Not discontent with life, but perhaps a bit discontent with all of this. At the moment.

So I am left wondering -- what if I stopped blogging?

The 20 people who read regularly read would notice right away (adjusted for the fact that this is posted on a weekend), the regular but busy folk/the in-love ones/others more casual would notice within a week or three, and it might make for a great WEHT blog that someone else could write in a year or so, after looking into whatever did happen. Nothing would actually "happen" though, in the end.

That persona would be gone, and people who missed it would have to seek out me and a less passive type of connection than the blog enables for them. A terrible loss, for some.

And I might miss it, too. My soapbox and my pronouncements from on low. I could imagine missing that, some at least.

If the existing other unpublished posts never went live? Not that much of a difference, really -- perhaps the difference between the Blog being hit by a bus and the Blog passing on after a long illness.

I am reminded of how I miss Suzanne E. McCarthy's Abecedaria, which is now approaching its second anniversary of not being posted to. We miss them when they disappear, whether with bang or whimper, knowing the people might still be alive, just no longer in our lives, right? With the blog just sitting, dormant -- a testament to the disappearance without corresponding backstory.

What if SiaO just stopped? Like with no warning or indication, perhaps even in mid-

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

If you have been around for a while then you may have read my Sometimes it seems like everyone hates Microsoft, a description of a particular call to Dell Technical Support.

I got a lot of mail about it afterward, including four people who pointed out that they work in Dell Technical Support and were regular readers here, and all of them were pretty unhappy about their closet Windows-hating colleague. One example (personally identifying information removed):

I read your blog post today about your unfortunate adventures with dell tech support. I actually was at work at dell, in tech support, when I read it and was appalled. I myself would never say such things. I always feel it is proper to put blame where blame is due and to give credit where credit is due. Windows is EXCELLENT at compatibility, thats its strongest point in my opinion. However, there are some driver writers and hardware vendors who dont hold their own work to the high standard windows is held to. I can not believe he blamed everything on windows, and with a microsoft employee on the phone. That, and he made you re-troubleshoot everything all over again. I hope that this hasn't soured dells image in your eyes.

It definitely did not sour me on Dell Technical Support, as I knew it was a pretty exceptional case. But if that had not been true, all of their responses would have made me feel much better about Dell.

What I found interesting was how all of them sent mail from their personal email addresses, probably on their own time, in what really ended up being pretty good damage control for one regular customer. I was impressed that Dell inspired people who care about their work enough to be bothered by such a terrible approach to the job.

And then if I contrast that with my recent Comcast experiences -- Hello pot, meet kettle (aka No, I don't want Comcast phone service, dammit!) and More fun and games with Comcast (aka the A&P of my Cynicality) -- where I was contacted by Frank Eliason from Comcast -- not only in comments (you can see them in both posts) but by phone (he found my listed landline phone and called me).

Now both of them gave me the reminder that the big companies are made up of people, and both made me feel a little better about the respective situations.

But riddle me this.

Why did the Dell response

(which was unofficial, not by anyone with ability or authority or mandate to remedy matters, and left no avenue to improve the situation)

have a more powerful effect on me than the Comcast one

(which was official, by someone with ability and authority and mandate to remedy matters, with several avenues of potential improvement available!)

anyway?

The mind works in strange ways, I usually assume mine are stranger than most, to be honest. And both effects were ultimately positive for me, but in very different ways.

There are at least eight people who have directly told told me that this blog played a significant part of their decision to interview with and ultimately choose to take a position at Microsoft. Would their decision have been to not do it had they known how unapproved I am by the people I work for at the moment, no matter who thinks well of my contribution outside of that small non-ivory tower? Or would the "unofficial" nature of it had made their interest have been even greater? And if that changes, does the meaning of the words really change?

Perhaps it is just the optimistic bent of my ultimately cynical nature to delight in the small pockets of genuine passion in large oceans.

How would you have felt?

This blog brought to you by ﹖ (U+fe56, aka SMALL QUESTION MARK)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

A bit from Fast Times at Ridgemont High you may recall:

BRAD: May I help you?
BUSINESSMAN: Yes. This is not the best breakfast I ever ate {he gestures to the sign over Brad's head: TRY OUR 100% GUARANTEED BREAKFAST} and I want my money back.
BRAD: Well, I believe you have to fill out a form. There's a pad right around here.
BUSINESSMAN: No. I want my money back right now.
BRAD: Well, that's not the way it works, really. And you ate most of your food already, too...
BUSINESSMAN: See that sign? It says 100% Money Back Guarantee. Do you know the meaning of the word 'guarantee'? Do they teach you that here? Give me my money back.
BRAD: I can't do that. But if you wait a minute...
BUSINESSMAN: {as if talking to a child} Look. Just put your little hand back in the cash register and give me my $2.75 back. Okay? {looks at name tag} Please, Brad?
BRAD: I'm sorry, sir. Just let me find the forms here.
BUSINESSMAN: I am so tired. I am so tired of dealing with morons. How hard is it to...
{Brad looks up from under the counter. No amount of pay will make him take that kind of insult}
BRAD: Mister, if you don't shut up, I'm gonna kick 100% of your ass.
{The door to the Assistant Manager's office swings open, and Dennis comes hurtling out of the back)
DENNIS: Can I help you, sir? Is there a problem?
BUSINESSMAN: You bet there's a problem! Your employee used profanity and threatened me with violence! I'm shocked, frankly. I've eaten here many times and I've always enjoyed the service -- until today!
{Angle on bathroom door as it opens and Arnold starts towards the register. He quickly sees the incident with the irate Businessman and ducks back inside the bathroom}
BUSINESSMAN: All I wanted was my money back for this breakfast. It was a little undercooked. And this young man used profanity and threatened me with violence.
DENNIS: Did you threaten this man or use profanity in any way?
BRAD: He insulted me first. He called me a moron.
DENNIS: Did you threaten this customer or use profanity in any way?
BRAD: Well, yes.
DENNIS: You're fired. {to Businessman} I'm very sorry this happened to you, sir.
BUSINESSMAN Thank you very much.

I was thinking about this as I was reading over in TSA's Evolution of Security Blog, in particular at their TSA and Piercings blog (a response to the incident in Lubbock with a woman with piercings that had to be removed with pliers).

And I realized where public relations nightmares seem to happen most.

When no one either resigns or gets fired or gets transferred.

How different would the public perception of the incident be if the people involved (by which I mean not just the TSOs on the front line but the people in charge of the policies that drove them) had been either severely and publicly either reprimanded or fired, clear policy had been publicly stated, and sincere apologies been offered to the woman who was victimized here?

Hugely different.

I work for a company with over 80,000 employees, including a lot of executives like vice presidents, etc.

Years ago, my dad was the VP and managing tax officer of a bank as I mentioned before, and he explained to me that one of the reasons that companies have lots of vice presidents is so there are enough people of sufficient authority to act as fall guys when bad things happen.

Business units can have problems, lose even billions of dollars, and if you are a stockholder or an employee (or both) you want someone who is at a high enough level to either be fired or have them resign due to some kind of failure to do something. I mean, we know they have golden parachutes anyway, but the serious gesture means something.

It gives a sense of accountability.

It is all an illusion, but we just want someone to feel like a mistake was made, and that someone will pay the price....

Whether it is the person who cut loose and did not follow policy, the person who created or administered the policy, or the person who let the bad thing happen under their watch.

Now Microsoft is an at-will employer, which means if they want that they can fire you any time. By and large, conventional wisdom is that you won't get fired unless you are guilty of a) theft, b) kiddie porn, or c) both. For just about anything else they just kind of slowly push you out, or not.

But it is easy to feel dissatisfied from the outside if you deal with a company that does not make these gestures to indicate accountability.

You see the stars rewarded, which is half of that -- but you never see the villains suffer. That all get swept under the rug.

When people publicly choose to make the gesture, it means something; it makes a difference.

Like the VP who said he wouldn't take his bonus that year given how late Vista was. Very few people asked why he was getting a bonus at all, why anyone was that year. But a lot of people recognized that such a sense of accountability is a good thing, and a lot of employees who felt like victims of the politics of large products really liked when someone in authority over such a project would step up in such a way.

The public wants accountability, they want something or more to the point someone to blame when things go wrong. It may be unfair and scapegoats do not solve real problems.

But scapegoats put faces on problems, acting as gestures that the problem was noticed, is being dealt with, and will be dealt with.

And it solves a lot of PR problems when they come up....

Is TSA better off for publicly admitting the procedures had some problems in them, as they did in their Statement on Alleged Improper Screening at Lubbock, Texas?

TSA acknowledges that our procedures caused difficulty for the passenger involved and regrets the situation in which she found herself. We appreciate her raising awareness on this issue and we are changing the procedures to ensure that this does not happen again.

Yes, I suppose they are.

But on the other hand, I didn't hear much in the way of an apology in this "official statement of regret", or any word of what happens to the people who were so excessive in following the procedures as to cause the problems, or the people who authored or administered the flawed procedures in the first place. Speaking as someone who has been asked to stand along enough that I actually did fall down, those kinds of gestures mean something.

This blog brought to you by ℀ (U+2100, aka ACCOUNT OF)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

The question seemed so simple (as it usually does):

Could you please explain me: why when I have the Arabic (Saudi Arabia) locale then the full date time format (“dddd, MMMM dd, yyyy HH:mm:ss”) actually looks as “dd tt hh:mm:ss MMMM, yyyy”. And opposite if conversation was done with the "dd tt hh:mm:ss MMMM, yyyy" pattern the result starts with substring "dd MMMM, yyyy"? Thanks for your attention.

Luckily sometimes the answer is simple, too! :-)

Let's take a look at this locale in Regional and Language Options, like right here:

and hit that Customize this format... button to get to the detail dialog about date formats.

Believe it or not, this is expected -- it is basically something you can look at two different ways, both of which are technically valid (though the second is probably "more" valid!):

The format strings themselves are using Latin script characters, which are always going to be normally used in a left-to-right context, while in the case of Arabic the actual string that has been formatted and which includes Arabic text will be displayed in a right-to-left context. So what appears on the surface to be a "reversed" string is actually just at RTL formatted string based on an LTR format string.

The format strings themselves are in logical order, which is always left-to-right and independent of how the final strings would be visibly displayed, while the formatted string in a language like Arabic will be displayed in a visual order that will be right-to-left.

So nothing is really in the wrong order, and things are mostly behaving as they are designed to.

I'll talk about when things get a little weird in a new blog some day.... :-)

This blog sponsored by غ (U+063a, aka ARABIC LETTER GHAIN)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

My oldest active Suggestion Box question (from Stef) is:

Hey, I'm a IT Student and at home i work alot with access.

'i work alot with' means that i made 3 db's projects that are 'Finished'.

Now working on the fourth i'm getting smarter and better,...

I learnt to split my db, but this last thing realy bite me

I'm having huge problems updating my older be version to a newer be version.

I know that you will say now use the Sync method!!! of the replica object...

But the 2 db's one of the costumer and my own are not linked in any way, I can only walk to the place and insert my usb drive. and your website sais don't use it when your moving your db around since it will create another replica wich is very bad. So i didn't use it.

Now, what would be the best way to update my backend with let's say 1 new table and 5 new fields in that table and 100 rows of data on the older db all have to be transported to my new version automaticely with the push of a button.

aren't computers made for this? :d

Sorry it took so long to get to this one, Stef. It is just that I am a lot of years out of Jet replication work (my last active development project that used replication was at least eight years ago). :-)

But there are many problems with using Jet replication for this scenario....

The problem you were hitting with removable storage? That's the old illegal move/copy problem wth replicas.

Add to that Microsoft has never gone out of its way to get behind the technology at all (and now in Access 2007 Microsoft uses their new fike format option to disallow replication -- a definite indication that they want to de-emphasize it even further!

Anyway, I usually point people to solutions like Tony Toews and his Auto FE Updater and/or Bob Laraon and his Auto Updater for Backend Databases, instead. :-)

In other news -- Limonata and Fruity Cheerios: yet another great combination (and good for you, too!).

This blog brought to you by ␖ (U+2416, aka SYMBOL FOR SYNCHRONOUS IDLE)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

Over in the Suggestion Box, Daniel García Magariños asked:

Hello there!

My proposed topic is "Exclude dictionaries" in Microsoft Word.

http://support.microsoft.com/kb/902163

They are very useful but they only work in plain text.

This means that:

a) You can only use them with languages which match the encoding of the "Language for non-unicode applications" in XP. We cannot change this setting in our environment... :(
b) It does not work at all with unicode-only languages like Maltese (one of our official working languages here)

We are using Windows XP and Office XP. Is this featur Unicode-compatible in newer versions of MS Word?

Thanks!

Daniel

It is very true that this limitation exists, though luckily Word 2007 has something better in its new renamed exclusion dictionaries (what they are called in the help files and such, with no KB article to give hints about this!).

The steps are a little different than the KB article since you have to unravel the ribbon and all....

But open up the Word Options dialog and select the Proofing tab:

and then make sure to hit that Custom Dictionaries... button.

It will open another dialog:

Now you see that file path, right? The one with the UProof rather than Proof directory?

Let's click the New... button there for a second:

You can look at those .lex files -- right click them and try to do the Open With trick, open in Notepad, and do the Save As thing to see what encoding Notepad detects.

The files are indeed Unicode encoded!

This is very cool, though the fact that the previous five versions were not doing it right was not very inspiring. :-(

At least they got it right eventually!

This post brought to you by ಹ (U+0cb9, aka KANNADA LETTER HA)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

Over in the Suggestion Box, Marvin asked:

Puzzled by MSDN docs. IMLangConvertCharset is recommended for speed but there is no way to tell it what default (fallback) characters to use. Most maddeningly its Initialize function accepts a dwProperty param that can be MLCONVCHARF_USEDEFCHAR.

IMultiLanguage2::ConvertStringFromUnicodeEx has this ability but it is supposed to be slower. ConvertStringToUnicodeEx on the other hand has relevant parameters but says they are ignored.

What should a developer do (other that using 3rd party library)?

Background: I need MLang rather than MBtoWC/WCtoMB C API because I need continuable conversion, i.e. being able to to give parts of the input (possibly broken in the middle of a char) to the conversion routine. It appears that the only MS technology capable of doing it is MLang.

The IMLangConvertCharset interface does indeed claim to be pretty damn fast, but you have to keep in mind something that has been true since the earliest days of WideCharToMultiByte:

The WideCharToMultiByte function operates most efficiently when both lpDefaultChar and lpUsedDefaultChar are set to null pointers. The following table shows the behavior of the function for the four possible combinations of these parameters.

lpDefaultChar	lpUsedDefaultChar	Result
NULL	NULL	No default checking. These parameter settings are the most efficient ones for use with this function.
non-NULL	NULL	Uses the specified default character, but does not set lpUsedDefaultChar.
NULL	non-NULL	Uses the system default character and sets lpUsedDefaultChar if necessary.
non-NULL	non-NULL	Uses the specified default character and sets lpUsedDefaultChar if necessary.

The fact that the fastest interface does not expose these ways to slow down the operation even though the MLang equivalent of WC_DEFAULTCHAR exists -- the operation of replacement by the default character and the option of specifying a custom one are two very different methods that aren't going to be available in every method or even in every code page.

But many other of the conversion methods within MLang will support specifying a custom default character and continuable conversions in buffers or streams (as you noticed). Some of them even work!

I don't think the particular lack on a particular interface is a reason to decide there is nothing provided by Microsoft, though. :-)

As a general rule, the desire for ultra-fast does not mix terribly well with the desire for lots of options, and this case is definitely not an exception to the rule.

This blog brought to you by ヲ (U+30f2, aka KATAKANA LETTER WO)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

The story behind today's blog post started in Why do we call w 'double u' -- doesn't it look more like a 'double v'?, where I talked about the Swedish Academy's change to the way the letters W and V were to be handled in collation, and the impact on Microsoft software when this change eventually makes it to the point where it needs to be integrated.

In particular I discussed the implications for software such as Jet [Red] and SQL Server, which unify Swedish and Finnish collations together into one single collation, when the two collations become different. Obviously there are far-reaching implications here that have to be carefully considered.

And that one day this "theoretical" issue that is a punch line in a blog post from Raymond or I would have far reaching design consequences....

A couple of days later, I noted in The disunification of Norwegian and Danish sorting that there was actually already such a change in Vista -- a change in Norwegian collation that was not happening in Danish (two collations which had also been unified in Jet [Red] and SQL Server), a change that had already gone into Vista.

Obviously with each of these changes it is only a matter of time before the issue has to actually be addressed, but neither is theoretical -- like pregnancy, at some point the situation will force itself on people and cannot be ignored. And to continue the ridiculous pregnancy analogy for just one this one more sentence you are reading now, the Norwegian "baby" is much further along than the Swedish one....

Now on top of all of this, factor in Not all in sync quite yet (aka SQL and the CLR and Windows and .NET), which spent some time talking about the consequences of these various collation solutions that move further and further apart. And in particular looked more specifically at the Danish/Norwegian disunification issue and SQL Server, at the collations covering the two languages in SQLS 2005, and what to worry about going forward....

Now of course if you care about SQL Server, you probably noticed (though when I say this it is with full knowledge that at least one regular reader to whom it is crucial missed it!) my blog On changing the world, or at least the way people order things in it, where the Windows Server 2008 collation support in SQL Server 2008 feature had achieved line item status, and obviously the whole Danish/Norwegian disunification issue must have been solved!

And indeed it has.

If you take the query I used in this post:

SELECT
    name,
    COLLATIONPROPERTY(name, 'CodePage') as CodePage,
    CONVERT(binary(4), COLLATIONPROPERTY(name, 'LCID')) as LCID,
    CONVERT(binary(4), COLLATIONPROPERTY(name, 'ComparisonStyle')) as ComparisonStyle,
    description
FROM ::fn_helpcollations()
WHERE
    COLLATIONPROPERTY(name, 'LCID') = 1030 OR
    COLLATIONPROPERTY(name, 'LCID') = 1044 OR
    COLLATIONPROPERTY(name, 'LCID') = 2068

(this query that returns 20 rows in SQL Server 2005) and run it that CTP of SQL Server 2008 to see what it says.

And here it is:

So the old Danish_Norwegian collation still exists, but two new 10.0 collations (Danish_Greenlandic_100 and Norwegian_100) were also added, bringing the total number of rows this query returns to 56.

Now accidentally, the SQLS folks managed to work around an interesting conceptual backwards compatibility issue, too.

I'll explain. :-)

Now not every version of Windows will necessarily have an updated Unicode version that would require an updated default table. Which means that in theory had there not needed to be a new default table that as simple Norwegian collation would have been needed to be added.

If you look at the Danish_Norwegian collation, you will see how it is associates with LCID 0x00000406, aka MAKELANGID(LANG_DANISH, SUBLANG_DANISH_DENMARK), so the new collation could be added with the different LCID and the old one left right in place -- thus no change in behavior would be required for the existing collation!

But I imagine that the name (Danish_Norwegian) was likely based on the random chance of alphabetical order, as was the LCID they chose. So there was a 50% chance that this behavior-preserving change would not have been possible and that the same LCID would have required a behavior change while the newly added LCID wouldn't....

Luckily, Vista took a long time to ship and some pretty hefty Unicode updates made it in. So it was a non-issue, since two new collations ended up being needed irregardless of this issue.

Of course as it turns out they will get lucky again on the Finnish/Swedish disunification whenever that happens, since Finnish_Swedish uses 0x0000040b, aka MAKELANGID(LANG_FINNISH, SUBLANG_FINNISH_FINLAND) -- so the Swedish change in some unknown future version would just be a single additional set of collations, rather than two sets.

Two near misses, though. Though perhaps this is a good time to bring up that it may not have been the best design in the world to do all of this unification work, given how things can disunify over time. And perhaps one day the disunification will fall on the wrong side of random chance....

This blog brought to you by å (U+00e5, aka LATIN SMALL LETTER A WITH RING ABOVE)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

This post has nothing to do with any issue in the Middle East whatsoever. I suppose I have an opinion or two about that but it is unlikely to be a topic in this blog any time soon....

The whole using lyrics from a Neil Diamond/Jazz Singer song is a bit of a stretch, sorry. But I just had to get myself taken off a list talking about [Biblical ]Hebrew support in computers that turned out to be really really keen on the active proselytization of Jews (something I always thought many or perhaps most Christian churches oppose, to various degrees), and The Jazz Singer was on cable, and this question came up related to Hebrew, and everything just seemed to intersect.

I would likely have done better had Neil Diamond given me more to work with, but then I guess I usually feel that way when I think about The Jazz Singer.

But I digress.... :-)

Back on the 13th of this month over in the Microsoft Volt users community, aldinois asked:

is windows vista ultimate the only vista where i can use hebrew?

Now maybe it is just me but that takes me way back, all the way to Even every version of XP Home is fully internationalized...., in fact!

Because Hebrew will work on any version/SKU of Vista without even the need to enable complex script support, with only two caveats:

1. Hebrew as a UI language (i.e. MUI support) requires Ultimate or Enterprise, and
2. Microsoft Office behavior with Hebrew may be less than ideal if you do not enable support for Hebrew in their Language Settings tool.

But other that, עברית (Hebrew) will work just fine in any kind of Vista you can throw at it. Uniscribe is on, the fonts and keyboards are there, and everything is ready to go!

This blog brought to you by ע (U+05e2, aka HEBREW LETTER AYIN)

Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

You may have read Vietnamese is a complex language on Windows, which discusses some fixes that were put in for Vietnamese in Vista.

The nature of the bug and the fixes that were put in was discussed there, as well -- it amounted to some characters used in the language but not included in the Vietnamese exception table -- characters also missing from the keyboard and the code page, and a few inconsistencies in weights found.

Anyway, recently the person who originally reported the bug commented on the Vista behavior:

Hi Michael,

Not until today, I got a chance to test the new collation on Vista. It seems that those bugs have been fixed for the Unicode composite format but not for the precomposed. Moreover, the fixes seem to have introduced new bugs. The list below includes the Vietnamese characters in question.

 Reference  : aAàÀảẢãÃáÁạẠăĂằẰẳẲẵẴắẮặẶâÂầẦẩẨẫẪấẤậẬiIìÌỉỈĩĨíÍịỊoOòÒỏỎõÕóÓọỌyYỳỲỷỶỹỸýÝỵỴ

 Composite  : aAàÀảẢãÃáÁạẠâÂầẦẩẨẫăĂẪằấẤẰẳậẲẬẵẴắẮặẶiIìÌỉỈĩĨíÍịỊoOòÒỏỎõÕóÓọỌyYỳỲỷỶỹỸýÝỵỴ

 Precomposed: aAàÀảẢáÁạẠãÃăĂằẰẳẲẵẴắẮặẶâÂầẦẩẨẫẪấẤậẬiIỉỈĩĨíÍịỊìÌoOỏỎóÓọỌòÒõÕyYỳỲỷỶỹỸỵỴýÝ

Given the latest environment (in terms of OS, .NET framework), how can I get the correct sort?

Quan

Let's take a closer look at the values he gave, marking the one that do not match the reference in red (blowing them up and putting the reference in the middle to make visual comparisons easier):

 Composite  : aAàÀảẢãÃáÁạẠâÂầẦẩẨẫăĂẪằấẤẰẳậẲẬẵẴắẮặẶiIìÌỉỈĩĨíÍịỊoOòÒỏỎõÕóÓọỌyYỳỲỷỶỹỸýÝỵỴ

 Reference  : aAàÀảẢãÃáÁạẠăĂằẰẳẲẵẴắẮặẶâÂầẦẩẨẫẪấẤậẬiIìÌỉỈĩĨíÍịỊoOòÒỏỎõÕóÓọỌyYỳỲỷỶỹỸýÝỵỴ

 Precomposed: aAàÀảẢáÁạẠãÃăĂằẰẳẲẵẴắẮặẶâÂầẦẩẨẫẪấẤậẬiIỉỈĩĨíÍịỊìÌoOỏỎóÓọỌòÒõÕyYỳỲỷỶỹỸỵỴýÝ

The information here is useful in the sense of repirting that therereporting are problems, but ultimately not in resolving the problems.

Perhaps I should explain what I mean by that. :-)

• Some of the ones marked "incorrect" are done so in the absolute sense but relatively speaking some are correct -- but none of this is marked (in the end only a few of the last row entries are wrong in the relative sense);
• None of the ones on the composite row use the composite forms, meaning you cannot reproduce the same reported issue with these strings for the composite characters;
• None of the intermediate forms that sit somewhere between normalization forms C and D are represented in the composite row, either;
• These rows of characters are not entirely useful since they do not distinguish between primary, secondary, tertiary, and quaternary differences;
• None of the E or U characters are included in there, which suggests as minimum of 46 missing vowels -- and given the problem that happened by not looking at the full set, anything less than the full set is ultimately insufficient;
• Other letters like đĐ are also not in there, which lead to the same kind of potential problem;
• No references to dictionaries or government standards are given, which mean any actual contemplated fix would require that kind of supporting information.

Now ignoring all that for a moment and explaining a bit about the actual differences noted....

The nature of the problem here is twofold:

• There are still some letters that are being identified as being Vietnamese that are not in the Vietnamese exception/compression tables -- they are put in where they are in the default table, which is in a few cases incorrect.
• The identified diacritic marks were also "moved" in accordance with the expected weight results, even though some of the letters they combine with may not have been, given the above point (this causes the composite case to be better for the targeted letters though worse for some of the others.

As a rule, characters that are not used in a language do not tend to get moved along with the ones that are, but this particular discontinuity means that when such letters are sorted, their results may not be a 100% match for the precomposed/composite forms, which is why there is a difference between the two forms (the default table does match them, as do the identified Vietnamese letters; but when the moved diacritics combine with letters not in the known set, they will be moved out of matching their analogous precomposed forms).

This is a problem that cannot be fixed without a major version change, for Vietnamese, which would means a new version of Windows -- as I pointed out in 2001, a Correctness Odyssey (aka What's the matter with Ü?), it has been decided this kind of change cannot be done in a service pack....

So in any case, someone over in NLS has some investigation to do, for both repertoire and order, for a future version of Windows.

This blog brought to you by Đ (U+0110, aka LATIN CAPITAL LETTER D WITH STROKE)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

You may recall when I blogged 'It's an iBOT,' I would say. 'Yes you bought it, but what is it?' she would answer, the Colbert Report interview of Dean Kamen talking about the IBOT (sorry that the video is no longer available, that's a youtube thing, its not me).

Anyway, Dean was back on the show last week to talk about his water purifier, you can see the interview here, here, or here perhaps (look fast before they disappear, I guess?). The unit is pretty interesting it is a much cleaner looking version of a project he first talked about maybe five years ago, described in his Wikipedia article, here:

Kamen has worked extensively on a project involving Stirling engine designs, attempting to create a machine that would generate power while serving as a water purification system. He hopes the project will help improve living standards in developing countries.

It is staggering, and more than a little bit inspiring, to see people like Dean really trying their hardest to help solve problems like this.

Which reminds me -- sorry hon (you know who you are, dear!), though Dean is still single, he is married to his inventions. Maybe he would be willing to have dinner, though. He could pick you up in his helicopter. :-)

My appeal for the IBOT (ref: NOT MEDICALLY NECESSARY per, by their own description, non-medical people), is proceeding, and I now know of some people who have succeeded so I am feeling more and more hopeful about the process and an eventual good result. I'll post more about this soon. I have had an additional opportunity to try the IBOT out and for home, work, shopping, and traveling, it is looking more and more like a very powerful life enabler....

In other news, another great meal one can make with the help of San Pellegrino Limonata -- Limonata and Honey Nut Cheerios!

If this grosses you out, then try to stay strong. This was (while not as good as Limonata and Chrunch Berries), quite tasty!!!

This post brought to you by @ (U+0040, a.k.a. COMMERCIAL AT)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

I already had my own reasons for avoiding the shuffle, after the "random chance event" that led to me being really down on relationships (for more info see The shuffle? Be afraid. Be very afraid...). And now XKCD reminds me of yet another reason!

This is very good advice, yet another way to avoid being depressed (by the potential aftermath of someone spending intimate time with a blogger who has put identifying names of people the songs might remind him of).

Not that this ever happens, of course.

This blog brought to you by ซ (U+0e0b, aka THAI CHARACTER SO SO -- the character that really really wanted to sponsor this post, for reasons that are obvious, if depressing!)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

My blog Behind 'How to break Windows Notepad' was a pretty popular one, as were its follow-up posts -- More on that which breaks Windows Notepad and The Notepad encoding detection issues keep coming up.

Someone suggested to me that it was because they spent more time explaining why it was happening then composing fun sentences like Bush hid the facts and such. a popular pastime, to be sure, but not one that gets very far after that unless you know Chinese and manage to construct a string that is funny in both languages.

As a by the way, that would impress me very much if someone was able to come up with it! It is like Numerology on steroids to take the ASCII values for English letters and map them in one's ,ind to the Unicode code point values for Chinese in a way that you come up with such a word!

But that is not the subject of today's blog post. I just can't help saying that I am practically willing to fly to the person who comes up with this and buy them a glass full of whatever they are drinking! :-)

Today is about Erzengel's comment to the original post yesterday, here:

I can't repro in Vista. Fixed?

Well, that simply indicates that Erzengel has SP1 installed on his Vista machine. Because in Vista SP1 and Server 2008 have indeed taken steps to mitigate this issue (in other words, fix this bug!).

Now for the record this was not a change in the IsTextUnicode function, it was a change to Notepad, which now uses a slightly modified version of the algorithm I mentioned in The Notepad encoding detection issues keep coming up, one that is a bit more skeptical of the results of IsTextUnicode when its only basis for returning TRUE is IS_TEXT_UNICODE_STATISTICS and it has one of these kinds of strings

I might have argued that it should have been fixed in the function, but for most uses I think the IsTextUnicode results are no worse than problems behind the reasons that I don't like IsTextUnicode anyway. So in the end this probably okay to take the public concerns out of the problem.

And it is cheaper than fixing the freaking function I don't like, which no one felt like jumping up to do. I get tempted to do it myself, though to be honest I am most tempted to just try to get a development intern who would be given this job -- rehabilitating IsTextUnicode's functionality to make it a worthwhile function I'd be proud to not make fun of one day. Though maybe the PTB wouldn't want to assign resources that way, I think it would be a fun project if I were an intern. :-)

Anyway, the summary -- starting in Vista SP1/Server 2008, only IsTextUnicode thinks that Bush hid the facts. :-)

This post brought to you by � (U+fffd, a.k.a. REPLACEMENT CHARACTER)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)! 

Meanwhile over in the microsoft.public.dotnet.internationalization newsgroup, don rau asked:

I am trying to load a Korean string resource from a DLL.

I use LoadStringW after setting Thread.CurrentThread.CurrentCulture to Korean. 

The string I'm trying to receive should be comprised of the following 5 characters:
0xc6a9 0xc9c0 0x002f 0xd488 0xc9c8

however what I get is:
0x00a9 0x00c6 0x00c0 0x00c9 0x002f

Note the relationship between what I expected and what I received.

Anyone have ideas on what I'm doing wrong?

You might be able to see the pattern and know what is going on already. You might incorrectly be reminded of Behind 'How to Break Windows Notepad' but that isn't it since there is no guessing going on here....

They wanted "용지/품질" and they got "©ÆÀÉ/" instead!

Regular reader Mihai had the answer for don rau:

Thread.CurrentThread.CurrentCulture should not affect the result.
Most likely you declared that LoadStringW uses ANSI strings.
Take a look at [MarshalAs(UnmanagedType.LPWStr)]

That is one way to do it; the other way is to use something like the following p/invoke signature:

[DllImport("user32.dll", CharSet=CharSet.Unicode, EntryPoint="LoadStringW", ExactSpelling=true)]
static extern int LoadString(IntPtr hInstance, uint uID, StringBuilder lpBuffer, int nBufferMax);

Then you can let the .NET Framework do all the marshaling for you (I always prefer explicit entry points and exact spellings to save the real problems associated with the weird guessing that .NET does as part of its effort to be Putting the *backward* in backward compatibility and choosing a terrible CharSet default....

Which leads me to wonder, actually. If Visual Studio can default new C++ projects to Unicode (as they now do, ref: The Unicode train is leaving the station), why couldn't they change the default for new projects to use CharSet.Unicode and all of the related behaviors in a new version of .Net?

All they would need to do is add a new project level property for the "Default CharSet value to use" -- if it is not specified (like all pre-existing projects would be), then use CharSet.Ansi. And then have all of the new projects for the next version default to set that new property to CharSet.Unicode. And that's all you need to move everyone over to using Unicode.

Kim, are you reading this? Maybe you could help push this where it needs to go over there, I'll buy lunch if you will! :-)

This blog brought to you by U (U+0055, aka LATIN CAPITAL LETTER U)

Content of Michael Kaplan's personal blog not approved by Microsoft (see disclaimer)!

One never knows what one is going to overhear, does one? :-)

Maybe "Overheard" needs its own category. These are fun!

Anyway, it went something like this:

HIM: That girl in the background of the picture, she's cute.
HER: She's awesome... a very unique woman, never met anyone else like her. 
HIM: {nothing said. he just looked quite interested in what she had to say}
HER: We have similar temperaments, like to sleep lots and do the same stuff. A very unique woman. She is really cute though. If I were a guy....
[Pause for a "Joey Tribbiani" moment here to imagine what SHE meant here; HE didn't pause, though]
HIM: Maybe you should introduce us?
HER: Sure, she may come here to visit, soonish.
HIM: Great!
HER: But this is a terrible way to let me know you're not interested in me.

This is the kind of interaction that would almost definitely be ruined by knowing the context.

A full understanding of HIM and/or HER and/or THE PICTURE and/or THE GIRL IN THE PICTURE is almost guaranteed to be less interesting than what one could fill in with a bit of imagination, don't you think? :-)

If you happen to be HIM or HER or THE GIRL IN THE PICTURE (in the admittedly unlikely case that any of you are regular readers of this Blog -- my readership within the Seattle metro area is hardly large enough to make this terribly probable!), you of course are allowed to comment, though please help keep some of the mystery alive so that we can all live vicariously through the three of you and your interesting potential triangle....

This post brought to you by 𐨲 (U+10a32, aka KHAROSHTHI LETTER KKA)