June, 2008

Continuing on with the third of all point from A whole new spin on the term 'Vertical markets' (aka in SiaO we trust?), the one that had not much to do with typography but was a bit more all-encompasing....

And much more to do with me personally than with anything else.

Warning -- an overly introspective post that is too technical to be a worthwhile Potpourri read and too navel-gazing to be a worthwhile technical read. It's just therapeutic for me so you can probably skip!

It has to do with the nature of the work in Windows International.

I am once again drawn back to The Hitchhiker's Guide to the Galaxy:

    "It was a tough assignment,"said Deep Thought mildly.
    "Forty-Two!" yelled Loonquawl. "Is that all you've got to show for seven and a half million years' work?"
    "I checked it very thoroughly," said the computer, "and that quite definitely is the answer. I think the problem, to be quite honest with you, is that you've never actually known what the question is."
    "But it was the Great Question! The Ultimate Question of Life, the Universe, and Everything," howled Loonquawl.
    "Yes," said Deep Thought with the air of one who suffers fools gladly, "but what actually is it?"
    A slow, stupefied silence crept over the men as they stared at the computer and then at each other.
    "Well, you know, it's just Everything... everything..." offered Phouchg, weakly.
    "Exactly!" said Deep Thought. "So once you know what the question actually is, you'll know what the answer means."

This will seem, relevant in a moment, I think.... 

I have spent most of career at Microsoft with a focus on internationalization, and felt myself being drawn to Windows International all of that time, because I felt that it really was the center of the universe, speaking from the standpoint of Microsoft and internationalization, at least.

And then eventually I ended up there, l and I realized I was right -- it is the center of that universe. It is where the support comes from, where it all starts. Where, if people are not using our stuff or building on our stuff in one way or another then they are likely to be doing it wrong.

But the problem, and what led me to the changes I mentioned in Track change (a.k.a. A new job that has a few things in common with the old one) is that while I was sitting in the center of this universe, that it is by and large just where things start -- not where things happen.

Where things happen is further out from the center or source of things, in the applications from both Microsoft and the ISVs outside of it.

The Windows International Fundamentals effort in its purest and most original form (the form that the GM explained to me when I voiced my concerns about the increasing distance between where things start and where they happen) was really intended to be an effort to communicate with those places where things are happening and making sure that they are on course and doing things right. Helping them out in making sure things are happening right.

And that work is incredibly rewarding since to be honest (to go back to that Duvall quote I cited) this is an area where I feel like I do my best work when I am moving through their world but keeping in mind all of the time that it is their world. I don't feel quite as far from them when I am assisting them....

But (and you know there had to a but since the original allusion talked about this all having a 'disturbing' aspect!), the group really seems to be much more Windows focused, most of the time. When I work with groups in Microsoft outside of Windows or customers outside of Microsoft (both groups of which I think make up a large percentage of the SiaO demographic), I am not being told it is a bad thing, but it does feel like it is being treated like it is outside my actual job, but tolerated.

Kind of like that India trip now that I think about it,

And that extends to conferences I speak at or meetings I go to -- I am asked "Isn't that a program manager's job?" or whatnot. Maybe it is, but none of them were stepping up and I was doing those things before I was hired, and was pointedly told when I got the job originally that these things were recognized as being important and they wanted those things to keep happening. But now it is treated the same way -- as being outside of my actual job, but tolerated. Just barely, some days.

Now I am not claiming that Windows is perfect here, or that it needs no help in improving, but I do think that Windows is not the only piece of this puzzle. and I signed up for a great deal more. In the end, Windows is a platform, a foundation. And just as with the foundation under the house, it is doing its best when we aren't thinking about it. Which leaves all the times that we actually are....

International behavior in Excel/Word is roughly 42 million times more important than Calculator or WordPad/Notepad.

And SQL Server's (or Access's) use of collation is roughly 56 billion (the British billion) times more important than the way files are ordered being based on the user locale.

And what the keyboards do in Word is the only thing cutomers need in order to decide if the keyboard is broken -- behavior in Notepad is of trivial interest by comparison and only occurs to them when we ask them to try it.

Windows isn't where the important stuff happens most of the time, it is what sits underneath the important things. Which still makes it important, but not the only thing that is important. Because it is not what most customers really see when it comes to international features (even when those features entirely depend on the OS).

Not all of my colleagues share that same view about the importance though (and not everyone likes my approach to the whole situation, for reasons both good and bad, all of which I accept as valid even if they won't share that feedback with me), and it is hard to gauge how much of what I do amounts in the end to the same sort of strategic miscalculation as Loonquawl and Phouchg were guilty of -- working so hard to find the answer that I did not make sure that the question was being framed correctly.

Well, that isn't entirely right -- when I originally was thinking about issues like the ones in Open it all up, get out of the way, and then what happens? and Subsets of subsets of subsets of subsets of subsets, I was working to frame the question how I imagined it. But posts in this blog do not amount to team mandates, at least not in this case -- since it is my team in the sense that I am a member, not in the sense of ownership. And when people in Windows have to allocate headcount and budget and resources they can hardly be faulted for framing the question differently than I would, from where I sit.

I was momentarily stumped.

But then I had a conversation with someone very wise (way wiser than me, in fact!). A VP in an entirely different business unit and division of Microsoft who I knew from way back when.

With their help, I had an epiphany.

Two years from now, if I am asked to identify some of the most important work I did two years prior -- for myself, for my interests, for my passions, for Windows International, for Microsoft, and most importantly for customers -- a lot of it is stuff I can't talk about yet that [technically by the above definition] falls beyond the scope of my job, as some people see it. The distance therefore between the job and what makes the job worthwhile for me is also pretty big.

So I do think I am doing the right thing by and large, and I'll certainly keep on trying to be doing it.

But I have to wonder what it is has been costing me from a career standpoint to have my [sometimes meager, other times substantial] efforts tolerated rather than embraced.

So taking a step back, I let excerpts of the words of the prophet Gavin DeGraw wash over me (obligtory YouTube ref for people who don't have the album here):

...Part of where I'm going is knowing where I'm coming from

I don't want to be anything other than what I've been tryin to be lately
All I have to do is think of me and I've peace of mind
I'm tired of looking 'round rooms wondering what I gotta do
Or who I'm supposed to be
I don't want to be anything other than me....

...I'm surrounded by identity crisis everywhere I turn
am I the only one who's noticed
I can't be the only one who's learned...

...I came from the mountains
the crust of creation
My whole situation made from clay dust stone
and now I'm telling everybody

I don't want to be...

And I have come up with a plan, I think.

I need to frame the whole problem a bit differently -- with my mangement and the people in the group -- going forward, pulling the bulk of these extra-occupational pieces that are accepted as personal idiosynchrocy in until they are part of my job. Because they are important to Microsoft's customers, and whether I want to blame myself for not doing something sooner or my management for letting it happen, in the end I own my own happiness. And my own career.

Therefore in the end, if I want them to intersect, I own that, too.

Plus other random things of dubious value about language and music and a MacBook Pro and more, in a Blog called Sorting it all Out...

I assume I will probably still have a job by next Monday, but just in case I don't I think this blog would make for a hell of a coda.

This post brought to you by ䷃ (U+4dc3, aka HEXAGRAM FOR YOUTHFUL FOLLY)

Prior posts in the series:

• 0 (You have to start somewhere)
• 1 (Starting with a dictionary simple in every way)
• 2 (Three ways to convert!)
• 3 (Where do keystrokes come in?)
• 4 (Specifying when to modify, if not how)
• 5 (All about the language, baby!)
• 6 (it is sufficient to be iconic, aka DaYi, DaYi Aynu!)
• 7 (Configuration 'junk in the trunk', part 1)
• 8 (Configuration 'junk in the trunk', part 2)
• 9 (Will you be content if I tell you how some content can be defined?)
• 10 ("Even Jove nods," Atlas shrugged, "so we'll convert their old files, now.")
• 11 (The knights who say நீ, redux, #1)

Okay, I have left people handing long enough.

Now, in the grand tradition of all of the above blogs, plus these other relevant blogs:

• And we are the knights who say நீ (NII)
• Learn Tamil in 30 Days (or something like that)
• Back to Sri Lanka (conceptually)

The first public build of this file defining my try at a Tamil source file for the Table Driven Text Service is now available!

The steps to install it:

0) You must be running Vista or Server 2008. It would be wonderful if this worked downlevel, but nobody has agreed to test that and so it will not currently work. And no, trying to make it work on your own is not just copying a few files and adding a few registry keys. Sorry!

1) Grab the source file by right clicking on one of the following links here and saving them to your locale machine -- either in text form from here, or as a zip file here. If you get the zipped file then of course unzip it. You will need the text file for future steps.

2) Copy the text file to \Program Files\Windows NT\TableTextService on your machine (if the "Program Files" on your machine is another language, use that directory, do not create a new one!). If you drg it in you will be prompted to elevate:

so accept that and copy the file. When you are done it will look something like the following:

Though if you do not have MUI language packs installed those language directories will be missing and you might have some localized path chunks up in the breadcrumb bar.

2a) If you running on a 64-bit machine then you must also put a copy of this file in \Program Files (x86)\Windows NT\TableTextService on your machine (if the "Program Files (x86)" on your machine is another language, use that directory, do not create a new one!). If you skip this step on a 64-bit machine then you will not be able to use the input method in 32-bit processes.

You will get that same elevation prompt as the other directory if you drag the file in

and when you are done it will look something like this:

Though if you do not have MUI language packs installed those language directories will be missing and you might have some localized path chunks up in the breadcrumb bar.

3) Open up an elevated command prompt. All you have to do is right click on the icon in the Start Menu or whereever to get this menu:

and select the Run as administrator option.

4) You will be put in the system32 directory. You need to navigate to the place you put the file in step 2, ande then run the command to register the input method:

The command to run is

rundll32 TableTextService.dll RegisterProfile TableTextServiceTamil.txt

and it is somewhat case sensitive so do not experiment with the case unless you are being paid by me to test this (since I am paying no one this rules you all out).

You will then be prompted to OKAY this registration you just requested:

Say OK to this.

4a) If you are running on a 64-bit machine, you must also navigate to the place you put the other file in that elevated command prompt:

Run the same command again:

rundll32 TableTextService.dll RegisterProfile TableTextServiceTamil.txt

and then you'll get that dialog again:

Just OK it again.

5) Go to the Control Panel:

6) Launch Regional and Language Options:

7) Choose the Keyboards and Languages tab:

8) Click on the Change keyboards... button to launch the Text Services Framework's Text Services and Input Languages dialog:

9) Click the Add... button to open the Add Input Language dialog and select the exciting new Tamil input method:

10) After you click OK to that, you will be back at the Text Services and Input Languages dialog. Click OK and you are done.

And now you have this Tamil Input Method 0.5 to play with!

More for Sri Lanka coming in future blog(s)....

This blog brought to you by கு (U+0b95 U+0bc1, aka TAMIL LETTER KA + TAMIL VOWEL SIGN U, aka TAMIL LETTER KU)

In general, people will follow the defaults more often than they will override them.

The simple fact of human nature and how humans relate to software is why stuff like I talked about in What it means to be in the default install ends up being so effective in changing the perception of the application in all situations by simply changing the default.

Because if most people accept the defaults, then most people will get the defaults....

Now over in the Suggestion Box, Andy asked:

Hi Michael,

Do you know why Ctrl-Space is mapped to the space character in many keyboard layouts instead of leaving it unmapped like most other Ctrl combinations?

I was working on the keyboard handler of a terminal program and was trying to give priority to the keyboard layout wherever possible. Ctrl-Space is meant to send 0x00, so I was quite surprised to get 0x20 instead, but quickly tracked it down to the keyboard layout. It's easy to override of course, but it still seems curious as it's a shortcut in many applications anyway.

Now there are three different "defaults" at work here:

#1: The command line tool that builds keyboard layout DLLs, whether in

• the non-Unicode form (kbdtool.exe) that is used within Windows or
• the Unicode form (kbdutool.exe) that is used by MSKLC

both have an interesting behavior in regard to the SPACE character (scan code 0x39, virtual key VK_SPACE) -- if the key is not assigned in the base, shift, or control states, a space (U+0020) is assigned automatically.

The upside is that for most keyboards that have been built and shipped in NT-based Windows since March of 1995 (when the tool was first built), the space key has been inserted any time something else wasn't. Which is to say most times.

#2: MSKLC's Load Existing keyboard... functionality takes existing keyboards and loads them up. Since most existing keyboards have U+0020 assigned to all three shift states and since most users never assign any of these three states on the shift key, the default setting persists.

#3: MSKLC's creating a blank keyboard functionality, because although not the most common ways that keyboards are created, it begins the life of a layout free from the "defaults" of #1 and #2, above.

Users have the opportunity to change the layout if they would like to, but most people do not.

MSKLC has a slightly different default of its own, though -- it assigns U+0020 to the space bar in the base and shift states, but not the control state. It breaks free of the defaults of the past.

Its default is to assign nothing there.

That difference could in theory be thought of as a legitimate bug in MSKLC....

However, it doesn't really matter. When you are done and you build your keyboard, MSKLC calls kbdutool.exe to do the build, and the unassigned CONTROL+SHIFT will be given U+0020 due to #1, above -- unless you override the default.

Which almost no one ever does, since almost no one ever changes the defaults....

This blog brought to you by U+0020, aka SPACE)

Kim Gräsman asks via the Contact link:

Subject: GetTimeZoneInformation changed on XP?

Hi Michael,

I haven't been following your blog closely, but I was recommended on a newsgroup to ask you about this. Hopefully you can help me out, or direct me to someone who can.

It seems GetTimeZoneInformation has changed recently, and broken its documented behavior -- as I understand it:

1) If Automatic DST adjustment is turned off, GTZI is documented to return TIME_ZONE_ID_UNKNOWN
2) If Automatic DST adjustment is turned off, GTZI is documented to return a struct containing StandardDate and DaylightDate with the wMonth member set to zero.

For some reason, on all of our XP machines, GTZI does the following instead:

1) If Automatic DST adjustment is turned off -- returns TIME_ZONE_ID_DAYLIGHT (even though we're still in standard time)
2) If Automatic DST adjustment is turned off -- returns a struct with both wMonths set to October (10), which is, incidentally, the month where DST turns off here.

I'm absolutely positive that this behavior has changed recently, though I can't say when, I stumbled upon it the other day.

Now, a question: is this likely due to a code change in GetTimeZoneInformation (i.e. has the code been updated for some XP patch) or can it be caused by the recent updates to the time zone database? I'm trying to get an idea of where to start looking...

I'm in the "(GMT+01:00) Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna" time zone, FWIW.

Many thanks for any ideas,
- Kim Gräsman

Obviously there has been a great deal of interest in time zones lately.

If anything was ever going to change that was going to break some specific behavior while trying to fix the larger problem that

• products all over the Microsoft landscape were broken, and
• it was noticed inside the US

then we are definitely in the "right" time to see such a change....

Combine that information with the Second Tester's Axiom:

Axiom II: If someone changes it, they probably broke it

and suddenly problems seem more likely -- and I'll take this bug report as more authoritative (off the top of my head) than assurances that no such bug exists. :-)

I don't know of a specific change to the XP-level GetTimeZoneInformation that would change the return value this way for the non-DST-timezone case, and I would be curious if Server 2003 repros this, or whether Vista SP1/Server 2008 show the same behavior -- not just in GetTimeZoneInformation but also in the newer GetTimeZoneInformationForYear or GetDynamicTimeZoneInformation functions.

I'll forward this blog and the forum note to some of the time zone movers/shakers to see if anything pops....

A smattering of prior blogs from this Blog about time zones:

• May 2007: Never without my permission, Redux
• February 2007: I'm sorry, but Microsoft does not get time zones
• January 2007: The Phantom [Time] Zone
• December 2006: Sometimes when sorting the index is the last thing you want to use
• December 2006: Don't get into the zone too soon....
• September 2006: You want to change the time zone? Eto Akta Gamat!
• July 2006: Results of the 'Time Zone Scenario Feedback' feared
• May 2006: Did somebody say time zones were intuitive?
• April 2006: Ok, so there may be *some* worries here
• April 2006: Time zones make me cringe
• April 2006: The Sri Lanka time zone shift?
• January 2006: Brother, can you spare a time[zone]?
• December 2004: The Time [Zone] is Right...

This post has no sponsor

Sometimes the question you ask will have different answers depending on who you ask.

Like previously when I described (in the blogs entitled IsCharSomethingOrOther? and Is Kana 'alphabetic' ? Depends on who you ask....), where the answer would vary depending on whether one was asking the NLS API vs. the wrapper functions around NLS contained in Win32.

One could (convincingly) try to argue the differences between these two interfaces, but in the it usually boils down to different intents translating into a different understanding of the question, and thus coming up with slightly different answers.

Another such problem came up the other day, when a developer asked:

I am using PathIsRelative API is my code to ascertain where a path is relative or not. Now if I am giving input as “\Programfiles” or “\” as input the api does not take it as RelativePath. My understanding is that API will consider both of these paths as relative since in command window these inputs take me to root folder or folder relatives to root.

Let me know if I am wrong and also alternative way to find “\ProgramFiles” or “\” being relative or not.

One developer pointed out the problem in understanding, from the point of view of the function:

"\Program files" and "\" are absolute paths. Relative path is different if starts from different current folder. Relative to root is already an absolute path.

While another developer pointed out the overarching problem of the viewpoint:

I am confused about the meaning of "relative" here.

"\" and "\Program Files" are relative—to the current working directory. If cwd is "D:\foo" then "\" means "D:\", but if cwd is "E:\foo", then "\" means "E:\".

So… why is "relative to root … already an absolute path"? Is this simply how "relative" is defined in the world of the win32 API?

And that second developed answered this one in a way that not everyone will probably be able to parse completely, but which certainly failed to see the conflict that comes from the other point of view....

You  are correct. But \Program files has the same meaning when cwd is "d:\foo1" or "d:\foo2". Relative path starts from cwd. \Program files always starts from root (although the root is relative to cwd), so it’s absolute. Maybe you can just take this as routine.

Another developer volunteered a response to this question that feel a lot like that USER32 response:

...a way to determine if paths with a leading "\" are to be considered relative ("\Program Files" was just an example).

In our code we have the same issue and just special case it ourselves:

BOOL PathIsRelativeBlah(LPCWSTR wszPath) {
    ASSERT(wszPath != NULL);
    BOOL bIsRelative = ::PathIsRelativeW(wszPath);
    if (bIsRelative)
        return TRUE;
    if (wszPath[0] == L'\\' && wszPath[1] != L'\\')
        return TRUE;
    return FALSE;
}

Why would one care?

Well, if the current drive is unknown then in some cases the SHLWAPI PathIsRelative answer is dead wrong awful since one cannot make security decisions based on being sure the path is unambiguous with the response that PathIsRelative provides. So in some cases the function is acceptable, in others it is incomplete.

Now when wants to determine whether one is a relative or not in real life, it is much more complicate, and the many tests include:

• ABO blood group typing
• HLA antigen testing
• PCR (polymerase chain reaction) testing
• RFLP (restriction fragment length polymorphism) testing
• DNA testing

What tests make sense depend a lot on the context -- is the goal to find out just if it is a relive? Or is it a check for parental identification?

What one is prepared to accept as the answer will have a lot to do with which question one is really asking, and for what purpose....

Now if one does want to call those "driveless" paths relative then wrapping the SHLWAPI PathIsRelative function with one's own additional logic the same ways USER32.DLL's Is* functions wrap the NLS GetStringTypeW function so they can provide their tweaks.

So that people who want a slightly different bit of logic applied to "is this relative" have the chance to see that done.

And I'll close this with a true story of a conversation that dates back several years with a friend of mine who was about to marry a single mother and the question of whether to adopt her kids came up:

Doug: So you had to go through this too?
Me: almost.
Doug: Why almost?
Me: One of the kids wanted me to adopt, the other two felt like it would be a betrayal, and it was mostly moot since the dad would never have gone for it and he h just started getting garnished for support so ironically in having the court acknowledge he was a deadbeat dad he actually had a say whereas he didn't before.
Doug: And then entirely irrelevant after you didn't actually get married.
Me: Also true. You have to read the fine print.
Doug: Windows is easier. One call to SetParent and your problems are solved.
Me: Actually, there are many times that the SetParent call is actually the start of even bigger problems....
Doug: So Windows isn't really easier, it just looks that way.
Me: Pretty much. The fine print gets you every time.

This blog brought to you by P (U+0050, aka LATIN CAPITAL LETTER P)

Nothing technical, feel free to pass, you know the drill...

Last night was not according to plan.

I kind of got stood up.

Well, not exactly.

We had sort of made plans but we're both pretty busy so this kind of thing happens.

The upshot, I ended up spending a quiet evening at home.

I chatted with Andrea a bit, via IM (she is trying to watch the phone bill a bit better than she has been, so no calls for a while).

With permission, I'll provide the [edited] conversation:

Andrea: I have a question for you
Michael: shoot.
Andrea: I know you have strong feelings about song lyrics
Michael: yes...
Andrea: what if a song has really strong meaning for you with the lyrics you thought were there?
Michael: thats a good point
Andrea: its not like I am obliged to be true to the artist's vision. I don't even know the artist!
Michael: so, what song in particular has you tied in knots here?
Andrea: what makes you say that?
Michael: cmon.
Andrea: so presumptuous. why can't I just be talking philosophically?
Michael: you can. But you aren't...
Andrea: fair enough
Michael: so tell me
Andrea: well, I don't want to offend you
Michael: ship sailed, and sunk. you can just tell me :-)
Andrea: its one from your girlfriend
Michael: who?
Andrea: an aimee mann song
Michael: ???? Aimee isn't my girlfriend.
Andrea: its an expression, let it pass
Michael: okay. so what's the song?
Andrea: video?
Michael: The Forgotten Arm, 7th track.
Andrea: you know that doesn't make you more attractive to women, right?
Michael: I'm aware of that. But I'm not on the prowl now.
Andrea: ship sailed
Michael: and sunk.
Andrea: I didn't say that. Waterlogged, maybe.
Michael: anyway, about Video....
Andrea: what I originally thought the song was ended up being totally wrong. I put the quotes in the chorus somewhere else, and I thought the word after video was PRAYING, not PLAYING. then I looked in the liner notes and saw that I was just all wrong.
Michael: That's okay. You basically might have made the song sweeter. Nothing wrong with that.
Andrea: really?
[I then reminded her of here, where I point out that although there are limits in general I don't mind this]
Michael: Look at the song before and the song after. The romance becomes less and less hopeful with each song. Making the song sweeter delays the inevitable by some fraction of 3.5 minutes. No shame in that.
Andrea: really?
Michael: yes, really. I mean it isn't too much -- it is either one quoting the other, or the other actually saying that part. Not too much difference except some of those words actually being said just make it warmer, I think.
Andrea: And what about the word mistake?
Michael: Well, that one I like thematically, and there is an interesting linguistic aspect, too.
Andrea: I sense a blog in the future.
Michael: Could be...

And in fact, it is. You're reading it, right now.

The line we're talking about, in the context of the first verse and chorus, runs like this:

Tell me why I feel so bad, honey --
TV's flat and nothing is funny.
I get sad and stuck in a cone of silence.
Like a big balloon with nothing for ballast,
labeled like a bottle for Alice --
drink me down or I'll drown in a sea of giants.

And tell me, "Baby, baby, I love you.
It's nonstop memories of you.
It's like a video of you playing.
(It's all loops of seven-hour kisses, cut
with a couple near-misses). Back
to the scene of the actor saying:
tell me, baby, baby – why do I feel so bad?'"

Maybe you can see what I mean if you look at the quote -- the theme is the same whether one is saying "tell me this" or it's a combination of that and the other actually doing some telling. The meaning doesn't change.

But the difference between

It's like a video of you playing.

and

It's like a video of you, praying.

is interesting from a language standpoint.

First there is the language side -- whether we are talking about what the video is doing (playing) or what might be in the video (praying).

The fact that the lines that follow cross sentence boundaries make inserting a pause there plausible, which in turn makes the change in lyric plausible, too.

Though in the end, praying does seem a little out of place, since the rest of it is about classic movie scenes and such. If I play the song and my eyes are closed I can almost hear it like she did, but the word doesn't seem to fit, then.

And of course there is the whole L vs. R distinction that in English we like to act is so different yet the actual sounds are close enough that mistakes such as mishearing which one is used can be quite common -- even when a language does distinguish them, like English.

I don't know if that makes this difference in some way different from the Flap D thing in some European languages that we can't distinguish in English or not. Though I'd be interested in knowing if it is a different kind of issue or not. Are the things that sound the same to some people really similar to the things that sound similar to us? And can training/habit only go so far in making unique distinctions?

Or are Americans just retards at this sort of thing?

As a side note, I was guessing about the length of the song. It turns out that the length of Video is 3:35, not 3:30.

This blog brought to you by れ (U+308c, aka HIRAGANA LETTER RE)

Just kidding, this blog is completely written by me and none of it is a forgery, except in the sense that I sometimes forge it out of my imagination (which is something very different).

But yesterday over on The Unicode List, John Hudson stirred up a hornet's nest of sorts with an early morning contribution:

The upcoming issue of the archaeology journal Minerva will apparently
contain an essay charging that the Phaistos Disk, whose signs were
recently encoded in Plane 1, is a modern forgery. Should be interesting.

This particular encoding was not without controversy, in part because this charge has been made in the past, and in part because as John later pointed out:

As I understand, the government of Greece has refused to allow a
thermoluminescence test of the Phaistos disc. In this context, a
significant article challenging the antiquity of the disc may be a good
thing, since it will increase pressure for a reliable dating test to be
done.

There is plenty of buzz about the article in places like Language Hat and elsewhere, and there are certainly plenty of reasons to be somewhat suspicious about this exciting development, including:

• Michael Everson's comment that put me in mind of a similar line from The Untouchables: "How weird. Would anyone making a forgery do it that way?"
• Andrew West's comment: "It certainly won't be the first time that someone has claimed that the disc is a modern forgery ... nor the last time no doubt. But it is an opinion that does not represent scholarly consensus."
• Andrew's other comment: "Incidentally, my spidey senses start tingling when I read that the author of the forthcoming paper is also the editor-in-chief and founder of the magazine in which it is to be published."
• Asmus Freytag's [I shudder to call it a] contribution: "Who knows, perhaps there's someone out there already working on an article proving that Unicode itself is a moder [sic] forgery :-D"

Perhaps it is just a board game, this Phaistos Disc, or perhaps it is a forgery. With an inability to really find out for sure at present (why is it that any time progress is stalled we can always find a government to plausibly blame?), this is one of those things that won't really go anywhere, for now....

I had someone ask me if it would be a huge problem if it did turn out to be a forgery.

Well, fictional scripts and artificial scripts are on the roadmap and it's not like all scripts aren't invented at some point.

But it would make the description text:

Phaistos Disc Symbols: U+101D0—U+101FF

The Phaistos disc was found during an archeological dig in Phaistos, Crete about a century ago. The disc probably dates from the mid-18th to the mid-14th century BCE. Unlike other ancient scripts, relatively little is known about the Phaistos Disc Symbols. The symbols have not been deciphered and the disc remains the only known example of the writing. Nonetheless, the disc has engendered great interest, and numerous scholars and amateurs spend time discussing the symbols.

kind of an embarrassment for Unicode since it would be missing out on that all important bit of information that it was, if not a fraternity prank, then at least not on the level.

Given the (mildly obnoxious) Urban Dictionary definition of cretan, we are left with an interesting boggle -- either the Phaistos disc was created by a Cretan, or else it was created by a cretan.

My head hurts just typing that one....

This blog brought to you by 𐇕 (U+101d5, aka PHAISTOS DISC SIGN WOMAN)

The question developer Cynthia sent via the Contact link was a fun one:

How can I program the "Fn" key on my laptop keyboard via MSKLC?

The answer is that you really can't.

Because the Fn or Function key has a very special job -- its job is to cheat.

Laptop keyboards are smaller, and they have fewer keys on them than they claim to. They get away with that by having a key that can transform other keys, temporarily changing their identity!

I'll show you what I mean....

First we'll go to my Dell, taking that EXE from Handling [Unicode] input in the console to sniff out the keys like I mentioned earlier today.

I'll type some keys that are on the keyboard and identify them right here so you can see what that EXE tells us. It tells us nothing about the Fn key -- that key does not exist as far as Windows is concerned:

ReadConsoleInput test
Ctrl-D to quit.

#      UC    u/d  VK   SC  State

  0: U+0000 down 0025 004b 0100    <-- Left Arrow  (VK_LEFT)
  1: U+0000  up  0025 004b 0100
  2: U+0000 down 0026 0048 0100    <-- Up Arrow  (VK_UP)
  3: U+0000  up  0026 0048 0100
  4: U+0000 down 0027 004d 0100    <-- Right Arrow  (VK_RIGHT)
  5: U+0000  up  0027 004d 0100
  6: U+0000 down 0028 0050 0100    <-- Down Arrow  (VK_DOWN)
  7: U+0000  up  0028 0050 0100
  8: U+0000 down 0024 0047 0100    <-- Home  (VK_HOME)
  9: U+0000  up  0024 0047 0100
 10: U+0000 down 0021 0049 0100    <-- Page Up  (VK_PRIOR)
 11: U+0000  up  0021 0049 0100
 12: U+0000 down 0023 004f 0100    <-- End  (VK_END)
 13: U+0000  up  0023 004f 0100
 14: U+0000 down 0022 0051 0100    <-- Page Down  (VK_NEXT)
 15: U+0000  up  0022 0051 0100
 16: U+0008 down 0008 000e 0000    <-- Backspace  (VK_BACK)
 17: U+0008  up  0008 000e 0000
 18: U+0000 down 002e 0053 0100    <-- Delete  (VK_DELETE)
 19: U+0000  up  002e 0053 0100

Now of these 10 keys, note that half of them do not exist as dedicated keys on my MacBook Pro.

What's a guy who likes his Mac to do?

No worries, because I can use my Fn key!

Here is the same output on the Mac Book Pro (booted into Vista 64 bit and humming along quite happily):

ReadConsoleInput test
Ctrl-D to quit.

#      UC    u/d  VK   SC  State

  0: U+0000 down 0025 004b 0100    <-- Left Arrow  (VK_LEFT)
  1: U+0000  up  0025 004b 0100
  2: U+0000 down 0026 0048 0100    <-- Up Arrow  (VK_UP)
  3: U+0000  up  0026 0048 0100
  4: U+0000 down 0027 004d 0100    <-- Right Arrow  (VK_RIGHT)
  5: U+0000  up  0027 004d 0100
  6: U+0000 down 0028 0050 0100    <-- Down Arrow  (VK_DOWN)
  7: U+0000  up  0028 0050 0100
  8: U+0000 down 0024 0047 0100    <-- Fn + Left Arrow  (VK_HOME)
  9: U+0000  up  0024 0047 0100
 10: U+0000 down 0021 0049 0100    <-- Fn + Up Arrow  (VK_PRIOR)
 11: U+0000  up  0021 0049 0100
 12: U+0000 down 0023 004f 0100    <-- Fn + Right Arrow  (VK_END)
 13: U+0000  up  0023 004f 0100
 14: U+0000 down 0022 0051 0100    <-- Fn + Down Arrow  (VK_NEXT)
 15: U+0000  up  0022 0051 0100
 16: U+0008 down 0008 000e 0000    <-- Backspace (Delete)  (VK_BACK)
 17: U+0008  up  0008 000e 0000
 18: U+0000 down 002e 0053 0100    <-- Fn + Backspace (Delete)  (VK_DELETE)
 19: U+0000  up  002e 0053 0100

Pretty clever, huh? even though Windows can't see the Fn key, the keyboard can. And the hardware can fake out and make Windows think that one key is another, as needed.

As far as Windows knows, that key identity is indistinguishable from it being that totally different key....

Now if they could just use the Fn key to fix up the whole  | (U+007c, a.k.a. VERTICAL LINE) yet printed upon the face of the key is ¦ (U+00a6, a.k.a. BROKEN BAR) issue!

This post brought to you by ¦ (U+00a6, a.k.a. VERTICAL LINE)

Robert asked via the Contact link:

Dear Mr. Kaplan,

I just found an interesting story on your blog, namely http://blogs.msdn.com/michkap/archive/2006/09/26/771554.aspx .

I own a Japanese Windows Vista laptop, it has the typical Japanese Laptop Layout of 108 keys, that is 2 Alt keys, Hiragana/Katakana/Romaji key, Half-width/Full-width/Kanji key, 2 Control keys and the usual Windows keys plus Fn key.

However, it features a yen sign key and a backslash key. The one next to backspace is indeed the yen sign and another one (backslash and underscore next to right shift key) has an actual backslash imprinted on it.

This is all very good, however, if I ever wanted to type a real yen sign in alphanumeric mode (of the Japanese IME), I would indeed be typing a backslash that looks like a yen sign only in the Japanese locale. Everywhere else it would look like a backslash!

I wanted to change this behavior by setting the yen sign key to use the proper Unicode value for the yen sign. I tried Microsoft Keyboard Layout Creator 1.4, but I cannot have the layout match my keyboard at all; though the keys do work in testing mode, thought they are not assignable!

I tried your clue and save the layout as a .klc file, and indeed and very much so I could probably add the proper support in for having a seperate yen sign key and backslash key, giving those keys their proper values according to unicode.

However, as the special Japanese keys don`t work in MSKLC 1.4, they are not in the file and I could not find them in winuser.h. Secondly, I checked with a precompiled Windows executable for what the keys actual keycodes are, but those change depending on what operation they are ared for, e.g. changing to/from Hiragana to Katakana, changing to/from Kana keyboard mode, changing to/from half-width, full-width etc. are all different transitions codewise.

Currently it seems it is always a US-Layout of sorts (what the Japanese layout is derived from...), but hence it doesn`t feature the keys for input mode changing and I did not seem to find those VK names/values anywhere for Windows.

Is there any way in which I can obtain a .klc with all the codes already like in the standard Japanese keyboard layout?
Or if there is not, are the key codes for the VK keys at least noted somewhere with their respective transitions?
Also, if I actually could find them, would I be able to compile it with MSKLC 1.4, or would it skim those settings out of the final dll?

The post that Robert saw was In case you have a yen to extend your keyboard (or at least want a yen?).

Now MSKLC is of some use when this kind of forensic work is going on since it works to interrogate the keyboard, but it only looks up information for scan code it knows about -- and it is likely that it does not know about this key....

(This is kind of like what we were doing in Tracking down problems with the 102 key on an Acer Tablet with the UK Extended keyboard layout?)

You can basically take code like the bit I put into Handling [Unicode] input in the console and use it to get the scan code value of any of the keys.

From there, you can look at examples like this one or this other one for the MSKLC syntax for the .KLC file, since all you need to start the line with is a scan code and a virtual key (the scab code you get from the above. the VK you just have to pick a unique one).

Now this key is not one you would be able to necessarily edit in MSKLC since if the tool does not know about the scan code, it can't show the key.

But if you have the row correct in the .KLC file, you will be able to build and use the keyboard just fine! :-)

If you try this out (or at least that first part where you get the scan code), let me know what that other key does -- I am very curious whether the keyboard hardware vendor ever worked with Microsoft to define this keystroke so that it could be used....

This post brought to you by | (U+007c, a.k.a. VERTICAL LINE)

So back in the beginning of 2008, Harold Fuchs (in response to my Typing in random Unicode code points blog from almost 2.5 years prior, asked over in the Suggestion Box:

The method you described in May 2005 ("Typing in random Unicode code points") for entering Unicode characters by installing the Chinese language and IME simply doesn't work on my Win XP Pro + SP2 system.

The idea is that you select Chinese and then, with Numlock turned on, type the decimal character code on the numeric keypad while holding down the Alt key. I've tried it in IE7, Outlook Express 6 and Wordpad. No joy. For example, the decimal code 10003 (hex 2713) should produce a tick (check mark in American English). It doesn't. It merely produces a double exclamation mark, the same as if Chinese were not even installed, let alone selected.

Please, what have I done wrong?

It took me a minute to try and follow this, then I realized the problem.

My earlier blog started with some simple introductory test:

People ask all the time how they can type in random Unicode data.

Some people point out the vast array of supported Keyboard Layouts on Windows.

Others point out how you can create your own keyboards with MSKLC.

Still others talk about fancy things you can do with the numeric keypad.

And then still others like to go on about typing a code point value in Word, highlighting it, and then hitting <Alt+X>.

Personally, I like to just install the Unicode IME, first added for Traditional Chinese in Windows 2000 and available in every version of Windows since then.

I then went on to explain how to install and use the Unicode IME.

Of course note that the "every version of Windows since then" claim has since the time and date of that blog stopped being true when they stopped including it, which begin in Vista (as I explained here when I first gave a possible alternative!).

Now my post a couple of months later (Typing in random Unicode code points redux) links to How to enter Unicode characters into Microsoft Windows which gives lots of ways to get the input done, also.

But for XP SP2, the text in the original article is valid.

The problem is that Harold was taking more than one of the five different methods I listed that can all be used to enter Unicode code points, and combining them. So when it didn't work, the entire article that happened to enumerate five completely different methods was considered incorrect. :-(

Taking a closer look:

The idea is that you select Chinese and then, with Numlock turned on,

No, this part is not needed and therefore not really right. :-(

type the decimal character code on the numeric keypad

No, you use hexadecimal code unit, not decimal.

while holding down the Alt key.

No, this will make it not use the Unicode IME at all. This will use the numpad method (hinted at in the introductory part of that blog).

I've tried it in IE7, Outlook Express 6 and Wordpad. No joy.

Kind of expected under the circumstances. :-(

For example, the decimal code 10003 (hex 2713) should produce a tick (check mark in American English).

Actually, unless you add special information to the registry, the numpad method only uses decimal and only takes four decimal digits. So U+2713 (CHECK MARK) would not be produced in this case.

It doesn't. It merely produces a double exclamation mark, the same as if Chinese were not even installed, let alone selected.

I am not sure how U+203c (DOUBLE EXCLAMATION MARK) came out of here, but the Unicode IME wasn't being used (and it never uses decimal values here).

So, the solution is definitely to keep the different methods separate and just use one of them -- for XP my preference is still to use the Unicode IME, which if you think about it is the only method I gave more than hints at in terms of instructions for use. :-)

This blog brought to you by ✓ and ‼ (U+2713 and U+203c, aka CHECK MARK and DOUBLE EXCLAMATION POINT)

I will readily admit I am not much of a texter.

I said as much to someone who was texting me late last year:

Do you text message? It's what
young people do these days.

Nothing makes you feel older than phrases like that, I always say....

Unfortunately, the answer to the question was mostly no. I have text'ed with the folks who have text'ed me from time to time, but I never really got into it, or very good at it.

I guess I'm just not one of the young people....

That message I quote above has not scrolled off my phone yet, and the scrolling happens at 50 text messages. So that might be a hint at how seldom I use this feature of my phone! :-)

Though the one I got from my sister yesterday was pretty funny:

Are you engaged and haven't told
us?

Apparently one of them saw So logical that even Mr. Spock (and my fiancée?) would approve and took my kind of joking claim that

even if you were right (a point that I would probably not agree with you about unless we were in public and were going to get married (as I think it is best to never disagree with one's fiancée in public!)

was really just intended as hyperbole and thought maybe I was hinting at something. :-)

I also got email from two other people on the same subject (did it really seem like a hint? I don't think I am that oblique here!), but just the one text message. I'm really not that good with the texting. Maybe it's the speed of it all? I don't do twitter either (you may have heard of the latter, it is like blogging for people with ADD who haven't been taking their Ritalin). Texting seems more common but only the people younger than I seem to be doing it (which includes my sister!)....

This blog brought to you by ␂ (U+2402, aka SYMBOL FOR START OF TEXT)

Over the last few years, quite a few of my blogs have mentioned the LCMAP_LINGUISTIC_CASING flag for LCMapString:

• December 2004: The [Upper]Case of the Turkish İ (or: Casing, the 2nd)
• December 2004: What does "linguistic casing" mean?
• April 2005: When casing does not need to roundtrip in .NET
• May 2005: The last word on the FINAL SIGMA
• June 2005: LCMapString's *other* job
• August 2005: New in Vista Beta 1: more use of the word 'linguistic'
• October 2005: Comparing Unicode file names the right way
• October 2005: How best to alter case
• March 2007: The T's are crossed, but not all of the I's are dotted...
• May 2008: İn tıtlıng thıs ınclusıon ın re: the ınterests of Turkısh İSVs, am İ just tryıng to buıld İ's and ı's ınto the tıtle of thıs daıly contrıbutıon to SİAO (SıaO), amıgo?

There are indeed others too -- these are just the ones I remembered off the top of my head.

Of these, the most important two, in my opinion, are What does "linguistic casing" mean? (which explains what the flag does, conceptually) and İn tıtlıng thıs ınclusıon ın re: the ınterests of Turkısh İSVs, am İ just tryıng to buıld İ's and ı's ınto the tıtle of thıs daıly contrıbutıon to SİAO (SıaO), amıgo? (which gives the actual one-way mappings that the flag adds to the casing table.

These one way mappings really tend to be kept out of the default casing table, except for the Greek final sigma (for reasons I explain in The last word on the FINAL SIGMA), and there are really good reasons for thus -- because of the destructive way that people use casing.

Not just the destructive things that people within Microsoft use it, e.g. No Regex in the Unicode room! and 'The 44' (*not* 'The 4400'), but even outside of Microsoft.

For some reason people feel it makes sense to do case insensitive comparisons by changing the case and then comparing.

Even though this is slower and even though it is destructive to the original string, people like to do this anyway.

Of course in managed code, the distinction between passing LCMAP_LINGUISTIC_CASING and not passing it does not exist, unless you use the invariant casing support like I mentioned in Comparing Unicode file names the right way.

Which is not to say that all is nirvanas now. There are a few bugs still, and there are some entries missing. For example:

• The new Unicode 5.1 characters that weren't in the standard when Vista/Server 2008 shipped aren't there;
• A few pairs are reversed (discussed in In Case there is a bug....);
• Some one-way mappings were not added to the LCMAP_LINGUISTIC_CASING tables, like U+017f --> U+0053 and U+00b5 --> U+039c (these and several other entries were added back in early 2005 but were taken out in late 2005 due to several bizarre issues that people were hitting).

The upside is that even though people might for example expect ſ (U+017f, aka LATIN SMALL LETTER LONG S) to become S (U+0053, a.k.a. LATIN CAPITAL LETTER S) in an uppercasing operation, they won't on Windows or .NET. Not just in the default table (which no one would really want, even if they think they would), but also not in the LCMAP_LINGUISTIC_CASING tables, where having it might have been nice.

This even pops up in bizarre places, like Word's uppercasing conversion or HTML/CSS text transformations support.

Though of course many of them suffer from the same issue inherent in not passing LCMAP_LINGUISTIC_CASING -- which is that even if the mappings existed, they wouldn't have been seen anyway!

When I think about it all, Microsoft did something really awful to the word "linguistic" with the LCMAP_LINGUISTIC_CASING flag, in doing (and not doing) so many decidedly non-linguistic things....

This blog brought to you by ſ (U+017f, aka LATIN SMALL LETTER LONG S)

Regular readers may recall having seen the blog entitled Somehow I just get a Visual of the Logical Song (as sung by Supertramp) a few months ago.

Not everyone is convinced fully, just yet.

So I thought I'd add some more information to the mix....

Let's say you go to Regional and Language Options. I'm going to do it here in XP but you can use whatever version you like....

First change the Standards and Formats locale (aka default user locale) to Arabic - Saudi Arabia. Do not click Apply.

But do click that Customize... button and switch to the last tab (the Date tab):

Here is that same view we talked about before, where the logical and the visual varied.

So that dd/MM/yyyy seemed to look like yyyy/MM/dd in the right-to-left context.

Well let's go to the Numbers tab.

Change the Digit substitution setting from Context to National. This time you DO click Apply:

And now go back to the Date tab:

do you see it -- the context of the expression (LTR vs. RTL) determines the order even for almost entirely Arabic text.

But the format string, by convention, is always left-to-right.

Now you may argue that this is confusing, or wrong.

But after being this way in just about every version of Windows that has ever shipped an Arabic locale, even if you were right (a point that I would probably not agree with you about unless we were in public and were going to get married (as I think it is best to never disagree with one's fiancée in public!), changing the behavior in Windows would require changing the expectations of a lot of different people.

Not something to do lightly, if you know what I mean. :-)

I think I could even convince the fiancée about this last part, since I could point out that we don't want make all of those people feel like they have been wrong all of these years. Since she is obviously an intelligent and sensitive (if stubborn) woman, she would never want to cause that much confusion....

This blog sponsored by غ (U+063a, aka ARABIC LETTER GHAIN)

Here is thst big list of well-known SIDs, the bulk of which are localizable and a significant number of which are localized.

Note that not all of them are necessarily going to be useful for your program, or even for any program.

But with knowledge comes power, so enjoy the list for its own sake. :-)

All of these are greate for those LookupAccountSid calls that solve problems previously discussed....

This blog brought to you by 🀫 (U+1f02b, aka MAHJONG TILE BACK)

If you don't watch or at least want to watch Lewis Black's Root of All Evil, you can probably skip this blog (and come to think of it, you may want to skip this Blog in that case, as well!)...

Lewis Black is incredibly funny. He really is.

So far his show has put the following head-to-head:

• Oprah vs. The Catholic Church
• Donald Trump vs. Viagra
• Weed vs. Beer
• YouTube vs. Porn
• Dick Cheney vs. Paris Hilton
• High School vs. American Idol
• Kim Jong-il vs. Tequila
• Las Vegas vs. The Human Body

Anyway, he was on the Opie and Andrew show at the beginning of this month, and mentioned that they had put the kibosh on two specific episode ideas:

• Scientology vs. Disney
• Marriage vs. Death

Now I can sort of understand the first one a little (the ruling apparently came from the Comedy Central lawyers), given how aggressive the Scientology lawyers have proven to be over the years (plus the fact that despite some bad things about Disney this particular match-up does feel to me perhaps a bit more one-sided than a lot of the previous ones).

But the Scientologists have survived South Park, and Comedy Central has re-run the episode lampooning them since. So perhaps the legal department at Comedy Central is being a little paranoid here.

Nixing the Marriage vs. Death idea is a little screwier -- does this seem like the kind of show that attracts the viewers who so adamantly favor marriage that it will affect ratings?

I mean, I believe half of the advocates who have been on the show are actually married, and although it is true that none of them have died (that would make for a much shorter show!), it seems like a huge assumption that it would be anything more than comedy and satire and entertainment.

Maybe they just thought it wouldn't be funny.

But there isn't much you put people like Patton Oswalt and the other advocates on that wouldn't be really funny, if you know what I mean.

Just imagine the guest spots with famous people talking about how much death can slow down a career -- with the counterclaim being that Elvis seems much more active in sighting since his death (if anything his touring schedule is more active now!).

Or imagine the scandalous claim that there should be a mistrial given the well-known fact that marriage is death, making all of the proceedings a huge conflict of interest!

I think both episodes would have been pretty funny, and I would have watched them.

Oh well, the new season will start on July 30th, and even what the lawyers don't reject can quite entertaining!

This blog brought to you by ☺ (U+263a, aka WHITE SMILING FACE)