April, 2006

Just over a week ago I was posting about Unicode? Zip don't need no stinking Unicode!

Well, as Heath points out in this post, Palm has re-released their update, sans the non-ASCII characters in file names.

Given the current limitations in ZIP and the additional complications I pointed out in that earlier post, I suppose I could hardly blame them....

It is funny how Microsoft can't really win in this kind of situation -- if we do nothing, then we are branded a bunch of ignorant provincials for the limitations in the platform; if we provide any kind of extension to compression, then we are evil for ignoring existing standards in favor of our proprietary solutions.

Of course those two groups are (mostly) different people, so I guess it makes sense.

But the fact that we need to keep things in ASCII is more than a little disappointing. :-(

This post brought to you by "®" and "™" (U+00ae and U+2122, a.k.a. REGISTERED SIGN and TRADE MARK SIGN)

I guess there is no denying the problem. As this post from over as year ago proves, I am not even as smart as a starling.

But that's okay, because as Mark points out on Language Log, no one else is either....

:-)

Oh, just to keep the balance going (I am behind again!):

)))))))))))))))))))))))))))))))

This post brought to you by "］" (U+ff3d, a.k.a. FULLWIDTH RIGHT SQUARE BRACKET)

Remember how I have talked in the past about the difference between two different purposes for collation (comparison vs. identity, or alternately the difference between CompareString and EqualString)?

(if not you can follow those links for some of the backstory!)

Over on the BCLTeam WebLog, Krzysztof Cwalina explains about Why IComparable<T> does not extend IEquatable<T>.

Basically the same issue -- in most cases, one does not need to answer both the "which comes first?" and the "are they equal?" questions when two objects of the same type are being looked at.

Even when they are fairly basic items like strings, when both questions are valid in different contexts, the use of the strings themselves often suggests that only one of them is appropriate at a given time....

This post brought to you by "＜" (U+ff1c, a.k.a. FULLWIDTH LESS-THAN SIGN)

You don't want to ever use DEFAULT_GUI_FONT. Just like the thread locale, AVICAP32.DLL, and SetLocaleInfo, it really stinks.

Raymond Chen talked about some of its history of this GetStockObject derived font last year when he answered the question What are SYSTEM_FONT and DEFAULT_GUI_FONT?

He even explained one of the big problems about it in international scenarios:

"...on a typical US-English machine, they map to bitmap fonts that do not support ClearType."

But beyond that, there is another big problem. The problem is that the actual font it resolves to is SKU dependent; it is not dependent on any of the various locale settings on the machine. It therefore becomes one of those "features" that keeps an MUI system with a changed system/user locale from being a 100% substitution for a localized system in testing.

And even in Vista, where the single world wide binary goal has really been largely achieved, the backcompat issues that come up with (for example) a Korean user who expects a particular legacy application's font settings to not change disallow 'fixing' this behavior. So we are stuck with this bit of cruft that will forever differentiate some particular language SKUs of what is otherwise a SWWB model system....

Don't forget that as Raymond talked a bit about, all of the GetStockObject derived fonts are no longer what the help topic claims to be -- they are not used by the system for anything, and remain only for the sake of applications that have depended on these ugly beasts.

So please stay away from DEFAULT_GUI_FONT; it stinks, truly!

This post brought to you by "ᇸ" (U+11f8, a.k.a. HANGUL JONGSEONG HIEUH-PIEUP)

Hugh may have unintentionally come up with a great new slogan for the Microsoft Locale Builder....

Though it probably would never make it past the lawyers.... :-)

A few days ago, I wrote a post entitled Why do we call w 'double u' -- doesn't it look more like a 'double v' ?

The post itself had nothing to do with the title; it was actually about the Swedish Academy changes to consider 'w' and 'v' to be separate letters, as well as some of the potential consequences for Microsoft products if such a change is reflected in actual customer usage that needs to be captured in the future.

(the title was actually a little experiment to see whether a catchy title would get more attention in comments than the substance of the post; you can look for yourself what was proven in that experiment.)

Anyway, in the process of my speculation on products and what they might do in the future, I forgot that Vista already has such a change, in a country that is quite near to Sweden and Finland!

It has to do with the use of "aa" in Danish and Norwegian.

Both languages have the same basic alphabet -- the 26 latin letters used in English and most places, plus æ, ø, and å. Though in earlier days the letters aa were used instead of å (in fact in Danish this letter was not added until a spelling reform that happened in the late 1940s -- it was already widely used in both Norwegian and Swedish).

Now the Danish and Norwegian sort in Microsoft products was using the special sorting of the aa as a unique letter after z that was basically equal to å.

But while this was a (relatively) recent addition for Danish, it has been in Norwegian for much longer. And feedback had come in from customers such as the following:

Feedback such as this led to some more investigation and the final assesment was made that the time had come to remove the aa entries from the compression tables for Norwegian.

"But Michael," you might be asking, "What about Danish?"

Well, the answer there is that it is still too common to expect it to be treated as a letter -- and there are way too many textbooks and websites that put an extra (aa) at the end of the alphabet. Danish could not make the same change that Norwegian needed.

So, in Vista, the Norwegian tables have had the three variants of the aa compression removed, while the Danish (and also the Greenlandic, which uses the same sort as Denmark) have not. Therefore, these formerly unified sorts will now been disunified.

You may be filled with one or more of the following questions:

1. What does this mean for prior versions of Windows and the .NET Framework?
2. What does this mean for Microsoft Access (7.0 - 11.0) and their Norwegian/Danish sorts?
3. What does this mean for DAO 3.5/3.6 and their dbLangNorwDan/dbSortNorwdan enumeration members?
4. What does it mean for the Norwegian/Danish sort in SQL Server 7.0?
5. What does it mean for the DANISH_NORWEGIAN_* collations in SQL Server 2000 and 2005?
6. What does this mean for Windows Vista?
7. What does it mean for future versions of Access, Jet, and SQL Server?
8. What does it mean for WinFS?
9. What does it mean for future versions of the .NET Framework?

The answer for questions 1-5 is simple -- not a bleeding thing. We can't change those prior version results.

The answer for question 6 is also simple -- it is going to be changed. The two sorts with the two different LCIDs (0x0406 versus 0x0414/0x0814) and different names (da-DK versus nb-NO/nn-NO) will return two different results.

The answer for questions 7-9 is also simple (though that may change at some point!) -- I do not have a freaking clue. But you can bet your lunch money that I will be asking people some questions about this issue for WinFS and for the next version of SQL Server and for the upcoming version of Access that ships with Office 12.

This post brought to you by "ø" (U+00f8, a.k.a. LATIN SMALL LETTER O WITH STROKE)

Regular reader Mike Lippert asked the Suggestion Box:

I have talked about digit substitution many times in the past.

I was reminded of it recently when developer Kollen pointed put a pretty lame article:

Kollen is right, this article is poorly named. But beyond that, what is the benefit of putting together a huge function with no other purpose than to not work? How many times will people not read the surrounding text and copy/paste that code into their applications?

Yuck!

Well, at least the answer to Kollen's other question can be found here.... :-)

This post brought to you by "９" (U+ff19, FULLWIDTH DIGIT NINE)
(a Unicode charascter that is dressed to the nines....)

Back at the Unicode Conference, after the "Design Principles for A Regional, Multilingual Keyboard" birds-of-a-feather, I had a chance to talk with Klaas Ruppel, who has been helping with the Finnish government standards.

(Among other things, he gave us some data about how the Cyrillic script versions of Sami work to help with our collation efforts. I'll talk more about this another day....)

One of the interesting things he mentioned was something that Raymond Chen mentioned in passing about Swedish collation:

(In marginally related news, the Swedish Academy recently released its latest official Swedish word list, and it changed its longstanding policy and now lists the words beginning with "W" separately from words beginning with "V". Up until now, "W" and "V" had been considered merely typographical variants of one another and had been treated as identical for alphabetization purposes.)

For the record, neither the government contacts in Sweden nor the MS subsidiary PMs in Sweden have asked Microsoft to follow this particular recommendation from the Swedish Academy, in part due to the general reluctance that the Swedes have to cause their sort to be different from the one in Finland (and the Finns have not agreed to make this change at this time).

it will put an interesting cat among the pigeons for Access 12, Jet 4.0, SQL Server 7.0, SQL Server 2000, and SQL Server 2005, given the fact that they have folded the two locales into a single sort (called Swedish/Finnish in Access and Finnish_Swedish in SQL Server).

As I mentioned in International Features of SQL Server 2000:

Of course what will be most important down the road (if the Swedish language goes along with the recommendations and Finnish language does not) is how they plan to deal with the disunification of the two sorts!

There is no way to know what will happen eventually, though luckily change will not come too terribly quickly, if it does come....

This post brought to you by "w" and "v" (U+0077 and U+0076, a.k.a. LATIN SMALL LETTER W and LATIN SMALL LETTER V)

I honestly can't believe how strange the conversations I have seem to get, sometimes.

I mean, Cathy came by to grab a snickers bar from the candy dish, and she was fussing with the stuffed cows on the bookshelf. She was trying to figure out why they were all facing the books instead of out at the room. I pointed out that this is just something Shawn does while he is telling me that he is stealing something from the candy dish.

So as she was sitting there repositioning the cows to more outward facing positions I asked her not to hurt the cows (the three stuffed cows and one beanie baby cow are all I have left of the Godot project, UniCOWS!).

She seemed shocked and offended that I suggested she would ever hurt them, which forced me to remind her of the "cow on wheels" incident.

We were at Crate and Barrel waiting on a dinner reservation (meeting someone else there, if memory serves), and Cathy had found one of those European toys, a 'cow on wheels', something like this one:

She was pulling it around by a string that was attached, and she accidentally ran it into the wall.

Shocked, I asked her how did she think Julie would feel if she saw Cathy's little exercise in cow abuse?

She paused, and said apologetically "you're right."

And then she proceeded to run it into the wall repeatedly. I was certain that she'd be giving them a credit card to pay for damage to the wall (and the cow!).

Of course at this point I remembered the recent posts from over at Language Log about the cow, and I asked whether that cow on wheels (assuming it was still ambulatory after Cathy's exercise in bovined abuse) clouded the issue of the "cow a motor vehicle" further....

Occasionally we manage to get work done there, too. I promise.

This post brought to you by "⺧" and "𐂌" (U+2ea7 and U+1008c, a.k.a. CJK RADICAL COW and LINEAR B IDEOGRAM B109F COW)

It is easy to take cheap shots at documentation writers for their mistakes.

But it is usually placing the blame in the wrong place, since in many cases it is the actual functionality that was confusing first. So if the writer's attempts to clarify that which is hard to explain fall short then it falls upon the PM/developer/tester of the functionality to step up and help with that....

(In fact this week we are doing a doc. review for Vista to try and help with this!)

Anyway, way back near the beginning of this whole blogging adventure for me, I posted about API Consistency and Developer Comfort. In it I talked about some of those consistencies between small groups of functions within the Win32 API.

It is a comfort thing, as I said. But sometimes it can be a burden.

Let's take return values for example. In most of the NLS API that does any kind of conversion, transformation, or retrieval, the pattern is simple:

• On success when NULL is passed for the destination buffer, return the required size, including the NULL character if the NULL character was in the source;
• On success when a destination buffer was passed, return the size of what was converted/transformed/retrieved, including the NULL character if the NULL character was in the source;
• On failure, return 0 and GetLastError will return the reason; the destination buffer, if specified, is usually not touched.

It is simple, sure. And pretty consistently applied across most of the functions.

Of course one problem with this pattern is that you have to call it at least twice to get the size to use, or else risk allocating too much. Another is that there is a performance hit to getting the exact size. And yet another is that to not touch the destination buffer, the function may have to allocate.

So, when Shawn did the dev work on the NormalizeString function, a different pattern was used:

On success, the function returns the length of the normalized string in the destination buffer.

If the destination buffer is NULL or if cwDstLength is zero, the return value is the estimated buffer length required to do the actual conversion.

If the string in the input buffer is null-terminated or if cwSrcLength is -1, then the string written to the destination buffer will be null-terminated and the returned count of characters will include the terminating null character.

If a problem occurs, the function return will be less than or equal to zero. The application should call GetLastError, which will return one of the following values:

ERROR_SUCCESS	No error; this occurs when the actual size of the output string is zero.
ERROR_INSUFFICIENT_BUFFER	Need a bigger destination buffer. The return value is the negative of a better estimated guess of the required length. Try the conversion again with a buffer of -(Return Value) size.
ERROR_INVALID_PARAMETER	Input pointers were incorrect or normalization form was incorrect.
ERROR_NO_UNICODE_TRANSLATION	Invalid Unicode was found in string. The return value is the negative of the index of the location of the error in the input string.
ERROR_BADDB	The configuration registry database is corrupt.

There is inded a lot that is different in there, and you will probably notice that if pretty aggressively tries to deal with the three problems I mentioned. In fact, there are really only two real problems with it that I can see:

• It is different, so that whole consistency thing is lost;
• It is incredibly complicated, so many developers will [unintentionally] use it incorrectly (and groups like the Shell folks will be highly motivated to write wrappers to try and simplify the interface any time they need to use it).

Now if anyone decides that the documentation for this function is confusing, I think it is pretty obvious that it has more to do with the underlying functionality than the actual SDK topic. :-)

Everything up until now has been an introduction to a completely different example of overloading meanings that is much, much worse!

Let's talk about the LOGFONT structure.

It's most important characteristic (for our purposes) is that it is not an actual font. It is a descripion of characteristics of either an actual font (e.g. if it is returned by the GetObject function when it is passed an HFONT) or what a developer might want from a font (e.g. if it is being passed to the CreateFontIndirect function, or to the EnumFontFamiliesEx function -- though for latter only a few members are looked at).

And its one member that is most completely overloaded to the point of confusion is the lfHeight member:

So basically, there are three different ways that this member can specify the height (depending on whether the 32-bit value in lfHeight is greater than, less than, or equal to zero. None of which map to what most humans (developer or otherwise) would use to specify a font size.

And, to add insult to injury, in attempting to translate between what those humans would want and what the member specifies, it gives a formula that most people do not understand that depends on a functionality that it does not explain (the MM_TEXT mapping mode). You can find out what the MM_TEXT mapping mode is by looking at the GetMapMode and SetMapMode functions. Though they take an HDC and it may be confusing to many people what happens to fonts in different mapping modes since the font and the mapping mode are set in such disconnected contexts....

But let's get back to those three different usages of the lfHeight member. They also have descriptions that start with "The font mapper..." which would indicate they forgot that the LOGFONT is also used to describe a font after the mapping has occurred. Oops?

Another oops along the same line -- there is no indication in the docs about which type of return is expected if you use GetObject with an HFONT. Or if the expectation is that the result would be inconsistent or not. Or how to definitively get the size in such a case.

And then of course, two of them talk about device units (which are also never defined within the function). You can look at the topic Device vs. Design Units to get some clarity around those, though it is of course completely unclear what insight talking about them here provides. How about, since the LOGFONT is in no way tied to a device, avoid mentioning them here entirely?

Ok, never mind all that communal critiquing, Let's take the and take those apart one at a time:

When it is zero, "The font mapper uses a default height value when it searches for a match." What default value? How is it determined? Hmmm... a quick look at the source is very suggestive of a size of 12pt being considered a default, at least since NT 4.0. Hopefully it is never returned when one is querying font information like in a GetObject call. :-)

When it is less than zero, "The font mapper transforms this value into device units and matches its absolute value against the character height of the available fonts." Once again, an undefined phrase "character height" -- what is meant there? Maybe they mean the TEXTMETRIC/NEWTEXTMETRIC's tmHeight member, which "Specifies the height (ascent + descent) of characters." That kind of makes sense, right? Unfortunately, this would be incorrect -- the calculation in this case is based on the the UnitsPerEm mentioned in that Device vs. Design Units topic, "the em square size for the font". And that topic even says how to get it -- by using the ntmSizeEm member of the NEWTEXTMETRIC structure.

When it is greater than zero, "The font mapper transforms this value into device units and matches it against the cell height of the available fonts." And of course the cell height is also not defined. Though this one actually is based on the TEXTMETRIC/NEWTEXTMETRIC's tmHeight member, which "Specifies the height (ascent + descent) of characters" specified in the font.

So given all of the above, at least this lfHeight member can theoretically be figured out, too.

Though it is not particularly intuitive, is it?

On Friday, typography PM Judy came by my office to explain to me that this was the topic that her fellow typography PM Carolyn had mentioned was really confusing. It should be of no suprise to anyone that even people who work in typography find this member to be really confusing -- there is no one outside of a few people over in GDI who would not be confused at this very low level member that has been partially exposed in this not-quite-so-low-level structure.

I mean, how often can we really expect that people will specify it correctly, except when they use the built-in specified formula that maps to point size of the font? And in that case why did they even bother to expose it any other way?

It does help things like old time KB article 74299, which actually explains the things I did above in more practical terms, and is still just as valid beyond NT 4.0, despite what the "Applies To" section claims.

And it helps third parties like Dr. Dobbs come up with useful topics like Font Creation and Rounding Differences when trying to understand why the MulDiv function is needed here (and the problems in the area help people understand why this one usage of MulDiv is easier to find than documentation on the function itself!).

This would be a case where the managed world made it all a little easier, not just in the GDI+ classses around fonts but in helpful topics like this one on interrogating fonts....

So, as bad as the documentation here may be, I think it is fair to blame the actual implementation for any actual confusion here....

This post brought to you by "ཛྷ" (U+0f5c, a.k.a. TIBETAN LETTER DZHA -- a character that is not afraid to use its descender powers!)

Back in January of this year, reader J. Daniel Smith asked:

Did I miss your other blog entry on this topic?
-----
re: Comparing Unicode file names the right way

Tuesday, October 18, 2005 3:51 PM by Michael S. Kaplan
Ah, the reasons I am resistant to *that* particular path are the subject of another blog entry, coming soon! :-)

This post brought to you by "𐃺" (U+100fa, a.k.a. LINEAR B IDEOGRAM VESSEL B305)

(This post is so frighteningly offtopic that it is impossible to quantify the issue!)

Last year some time I was back in Cleveland, and I saw my first episode of Ghost Whisperer. I made the offhand comment during a commercial break that Jennifer Love Hewitt in this show (with her new darker hair) looked a lot like Alyssa Milano (if you went back a hairdo or three -- both Alyssa and Rose among the Charmed coven change their hair styles and colors a lot!).

I wasn't saying they were twins or anything, just that it would be easy to confuse them at first glance....

My father disagreed with me and suggested that I might be a little bit crazy. Or if nothing else that I did not know what actresses looked like :-)

Anyway, looking at the following two pictures, as a baseline example:

It should be easy enough to see why I suggested there might be a similarity, I think.

Which is not to say that I am not crazy; but if I am it is for other stuff, not for this! :-)

Assuming you are on this planet, you have probably heard about the name of Tom and Katie's new baby.

Of course I lost the faith I had in Katie (dating from Teaching Mrs. Tingle) by the time the second season of Dawson's Creek was over, and I don't think I ever had faith in Tom.

Ah well, at least he was supposedly kidding about the placenta eating bit.

Are there any non-weird actors out there?

In any case, over on Language Log, Benjamin Zimmer gives a good summary of the whole Tom Cruise/Katie Holmes baby naming announcement snafu in the post Adventures in Celebrity Onomastics.

So, the question I got in email from someone named Kenny was quite timely:

The .NET Framework has a file named normalize.dll, and Vista has one named normaliz.dll. What is the difference between these two files other than the name?

I say timely because yesterday was an interestingly busy day....

In the morning I was just doing laundry and such. That part was kind of boring.

But in the afternoon, I decided to use my FlexPass to run some errands downtown. I had had good luck with the scooter on the bus previously, and I had the day, so I figured why not?

First I wanted to go to Caffe Bella to give them some posters for the show that Kristin Connell is doing there on May 5th:

I left them in the care of Kim to hang up appropriately, and then I made my way over to the UW Medical Center to get some bloodwork done (not sure what they were testing for exactly but they got three lavendar tops, one red top, and two jungle tops out of me; I was a bit worried losing so much blood right after I donated a pint but I was encouraged not to worry!).

Anyway, I was waiting to head home on the 540 bus but there was a bit of a wait. While I was waiting around in front of UWMC for the bus, my phone rang. It was my old friend Liz!

Liz and I go way back, like about 15 years back, give or take. When we first became friends there was another Elizabeth in our circle who went by Liz also. But since the other Liz was really wild, we called her 'Crazy Liz' or 'Wild Liz'. And then my friend the not-wild one we just called 'Liz.'

But then, just after that fifth book in the Hitchhiker's Trilogy (Mostly Harmless) came out, the text that inspired us all further and became the subject of a late night discussion over kamikaze shots went something like this:

    Trillian picked up the sandwich and looked at it. She sniffed it carefully.
    "Try it," said Arthur, "it's good."
    Trillian took a nibble, then a bite and munched on it thoughtfully.
    "It is good," she said, looking at it.
    "My life's work," said Arthur, trying to sound proud and hoping he did not sound like a complete idiot. He was used to being revered a bit and was having to go through some unexpected mental gear changes.
    "What's the meat in it?" asked Trillian.
    "Ah yes, that's um, that's Perfectly Normal Beast."
    "It's what?"
    "Perfectly Normal Beast. It's a bit like cow, or rather a bull. Kind of like a buffalo in fact. Large, charging sort of animal."
    "So what's odd about it?"
    "Nothing, it's Perfectly Normal."
    "I see."
    "It's just a bit odd where it comes from."
    Trillian frowned, and stopped chewing.
    "Where does it cone from?" she said with her mouth full. She wasn't going to swallow until she knew.
    "Well, it's not just a matter of where it comes from, it's also where it goes to. It's all right, it's perfectly safe to swallow. I've eaten tons of it. It's great. Very succulent. Very tender. Slightly sweet flavor with a long dark finish."
    Trillian still hadn't swallowed.
    "Where," she said, "does it come from, and where does it go to?"
    "They come from a point just slightly to the east of the Hondo Mountains. They're the big ones behind us here, you must have seen them as you came in, and they sweep in their thousands across the great Anhondo Plains and, er, well, that's it really. That's where they come from. That's where they go.
    Trillian frowned. There was something she wasn't quite getting about this.
    "I probably haven't made it quite clear," said Arthur. "When I say they come from a point to the east of the Hondo Mountains, I mean that's where they suddenly appear. Then they sweep across the Anhondo Plains and, well, vanish really. We have about six days to catch as many of them as we can before they disappear. In the spring they do it again, only the other way around, you see."
    Reluctantly Trillian swallowed. It was either that or spit it out, and it did in fact taste quite good.
    "I see," she said, once she had reassured herself that she didn't seem to suffer any ill effects. "And why are they called Perfectly Normal Beasts?"
    "Well, I think because otherwise people might think it was a bit odd. I think Old Thrashburg called thm that. He says that they come from where they come from and they go to where they go to and that it's Bob's will and that's all there is to it."
    "Who--"
    "Just don't even ask."

So after many kamikazes one of us (I think it was Sam) was thinking about the fact that Liz's boyfriend at the time was named Robert and he said

She comes where she comes and she goes where she goes and that's Bob's will and that's all there is to it. She is a Perfectly Normal Liz.

What can I say, the nicknames stuck -- Perfectly Normal Liz when we were being formal, and Nomal Liz or PNL when we weren't. I think her sister disagreed with our assertion that Elizabeth was actually, in fact, normal. But we didn't really hang out with her sister so we ignored that. :-)

Then, to get back on topic, a few years ago, I was adding the normalization files that were already in the Whidbey builds to the Longhorn build, so that they would be there for users to try out in M6 (Shawn was in a class that week, I can't remember which one it was). Some folks on the base team recommended that since the file was being called by kernel32.dll that an 8.3 name would probably be better than using normalize.dll. Obviously NORMALIZ.DLL was the best answer to that concern, and after I was testing the setup change, waiting for a reboot, I thought back to my friend Perfectly Normal Liz and realized what a hilarious tribute it would make, even if it was only realized thirty minutes after the actual change was made (I often find that I am only clever in retrospect).

Liz was (by her own report) on the floor laughing when I finally had a chance to tell her about it, a little while after that PDC build of Longhorn that contained the file NORMALIZ.DLL in the system32 directory....

This post brought to you by "e" (U+0065, a.k.a. LATIN SMALL LETTER E)