March, 2006

Regular reader Ivan Petrov asked the following in the Suggestion Box:

Hi Michael

I've the following problem:

In MSKLC in the 'Control state' of the keyboard (when the Control key is pressed) I'm trying to make VK_OEM_3 a Dead Key. So, I assign to it  'U+0060 (') GRAVE ACCENT' and then I set it as Dead key. To this point everythig goes ok! Then I go into the Dead key dialog box. And here is the BIG problem. What I mean:
I want to do the following:
When I press "Ctrl" + "`"  followed by one of this vowels in the Bulgarian alphabet:
"а", "е", "и", "о", "у", "ъ", "ю" and "я",
the keyboard layout to produce one of this results:

а̀  (0430 + 0300)
ѐ  0450 or (0435 + 0300)
ѝ  045d or (0438 + 0300)
о̀  (043e + 0300)
у̀  (0443 + 0300)
ъ̀  (044a + 0300)
ю̀  (044e + 0300)
я̀   (044f  + 0300)

And finally the problem:

Let's take for example the first vowel "а":
In the Ded Key dialog box in the Base (code point) field I type "а". Then in the Composite (code point) field I type "U+0430 U+0300" and then MSKLC says that "The value must be either a single character or code point." So, this is the problem!

Can you help, how to deal with this.

I've no problem with the two precomposed letters "ѐ" and "ѝ", but the rest ... ;-(

Thank you in advance.
Regards,
Ivan Petrov.

Hmmm.... maybe Ivan was not reading often enough. :-(

The problem that he is reporting on has no solution. As I pointed out back in December of 2004 when I mentioned that Dead keys are not intuitive, and then a few times since then -- the end result of a dead key transaction must be s single UTF-16 code unit.

This is also explained in the MSKLC help file, as is the explanation that the dead key architecture is one that is around for legacy purposes only, and not generally for the creation of new keyboards.

Even if this ever were changed in a future release of Windows, it could not be used on existing versions due to the backcompat break that this would cause -- and the future version of MSKLC would have to support different keyboard layout DLLs for different versions of Windows. Which as I am sure people can imagine is not a terribly popular plan....

For solutions, there are two obvious ones:

• You can add each sequence to its own key on the keyboard, or
• You can add the combining character and then type letter plus combining character on the keyboard

Either of these plans will allow the characters to be supported....

This post brought to you by "у" (U+0443, a.k.a. CYRILLIC SMALL LETTER U)

Over at Language Log Plaza, Geoffrey K. Pullum was talking about Lexical Drift.

I tend to think of the specific issue he raises there:

Psychologist Alex Delaware and his best friend the gay detective Milo Sturgis are always having long and complex discussions about how much the current evidence favors this or that suspect. And Milo will often say, for example, "So, yaou like the husband now?" — meaning (and I have no idea how it was that I could see this instantly), "So, you now favor the hypothesis that the husband is the murderer that we seek?" The verb like has taken on a new sense where A likes B means "A favors the hypothesis that B is the culprit." See how that works? Maybe the new sense will catch on more widely, maybe it will be limited (or is limited) to police talk, maybe it will never spread much; we don't know, and we can't predict.

is actually more of a technical jargon thing that we accept in such situations, the same way we accept for example "The AIDS test was positive" to mean that the most negative of all possible results was returned.

Such usage (which we seem to accept without too much trouble) does not seem to affect our own usage in other places....

Previous posts in this series: Parts 0, 1, 2, and 3.

We're going to do a bit of preparatory adjustment in this post. Just so we can be ready for what comes later, you see.

If you look at information like the Scan Code and the Virtual Key, they really are independent of shift state. It means the code is kind of wasteful, continually asking for mappings over and over that it already has gotten.

Or perhaps I should say would be wasteful once we started adding new shift states to the mix. At the moment we are only mildly wasteful, where the code gets the scan code for VK_SPACE more than once.

Let's fix it now....

(As before, the older code is gray, the new code is black)

using System;
using System.Text;
using System.Windows.Forms;
using System.Runtime.InteropServices;

namespace KeyboardLayouts {
    class Class1 {

        //  You'll want to insert that enumeration from part #0 here!

        internal const uint KLF_NOTELLSHELL  = 0x00000080;

        [DllImport("user32.dll", CharSet=CharSet.Unicode, EntryPoint="MapVirtualKeyExW", ExactSpelling=true)]
        internal static extern uint MapVirtualKeyEx(
            uint uCode,
            uint uMapType,
            IntPtr dwhkl);

        [DllImport("user32.dll", CharSet=CharSet.Unicode, EntryPoint="LoadKeyboardLayoutW", ExactSpelling=true)]
        internal static extern IntPtr LoadKeyboardLayout(string pwszKLID, uint Flags);

        [DllImport("user32.dll", ExactSpelling=true)]
        internal static extern bool UnloadKeyboardLayout(IntPtr hkl);

        [DllImport("user32.dll", CharSet=CharSet.Unicode, ExactSpelling=true)]
        internal static extern int ToUnicodeEx(
            uint wVirtKey,
            uint wScanCode,
            KeysEx[] lpKeyState,
            StringBuilder pwszBuff,
            int cchBuff,
            uint wFlags,
            IntPtr dwhkl);

        [DllImport("user32.dll", ExactSpelling=true)]
        public static extern int GetKeyboardLayoutList(int nBuff, [Out, MarshalAs(UnmanagedType.LPArray)] IntPtr[] lpList);

        [STAThread]
        static void Main(string[] args) {
            int cKeyboards = GetKeyboardLayoutList(0, null);
            IntPtr[] rghkl = new IntPtr[cKeyboards];
            GetKeyboardLayoutList(cKeyboards, rghkl);
            IntPtr hkl = LoadKeyboardLayout(args[0], KLF_NOTELLSHELL);
            if(hkl == IntPtr.Zero) {
                Console.WriteLine("Sorry, that keyboard does not seem to be valid.");
            }
            else {
                KeysEx[] lpKeyState = new KeysEx[256];
                uint[] rgScOfVk = new uint[256];

                // Scroll through the Scan Code (SC) values and get the Virtual Key (VK)
                // values in it. Then, store the SC in each valid VK so it can act as both a
                // flag that the VK is valid, and it can store the SC value.
                for(uint sc = 0x01; sc <= 0x7f; sc++) {
                    uint vk = MapVirtualKeyEx(sc, 1, hkl);
                    if(vk != 0) {
                        rgScOfVk[vk] = sc;
                    }
                }

                for(uint vk = 0x01; vk < rgScOfVk.Length; vk++) {
                    if(rgScOfVk[vk] != 0) {
                        StringBuilder sb = new StringBuilder(10);
                        int rc = ToUnicodeEx(vk, rgScOfVk[vk], lpKeyState, sb, sb.Capacity, 0, hkl);
                        if(rc > 0) {
                            StringBuilder sbChar = new StringBuilder(5 * rc);
                            for(int ich = 0; ich < rc; ich++) {
                                sbChar.Append(((ushort)sb.ToString()[ich]).ToString("x4"));
                                sbChar.Append(' ');
                            }
                            Console.WriteLine("{0:x2}\t{1:x4}\t{2:x2}\t{3}\t{4}",
                                rgScOfVk[vk],
                                sbChar.ToString(0, sbChar.Length - 1),
                                vk,
                                ((KeysEx)vk).ToString(),
                                ((Keys)vk).ToString());
                        }
                        else if(rc < 0) {
                            Console.WriteLine("{0:x2}\t{1:x4}\t{2:x2}\t{3}\t{4}\t\t\tDEAD!!!",
                                rgScOfVk[vk],
                                ((ushort)sb.ToString()[0]),
                                vk,
                                ((KeysEx)vk).ToString(),
                                ((Keys)vk).ToString());

                            // It's a dead key; let's flush out whats stored in the keyboard state.
                            ToUnicodeEx((uint)KeysEx.VK_SPACE, rgScOfVk[(uint)KeysEx.VK_SPACE], lpKeyState, sb, sb.Capacity, 0, hkl);
                        }
                    }
                }
                foreach(IntPtr i in rghkl) {
                    if(hkl == i) {
                        hkl = IntPtr.Zero;
                        break;
                    }
                }

There will also need to be some thought on displaying all the information -- once we move into new shift states, a bit more economy will be needed. That change will come with the shift states, which are next in the part....

This post brought to you by "4" (U+0034, DIGIT FOUR)
A Unicode character that is in the very small family of those whose VK value is the same as it's code point!

Basically we are looking at two calls to CompareString. The first is:

CompareStringW(0x0409, 0, L"Hello-Bob", -1, L"Hello Bob", -1)

which returns CSTR_GREATER_THAN, and the second is:

which returns CSTR_LESS_THAN.

I promise there are no "spoofing" characters or anything else unexpected in the strings, it is literally

• a comparison of two almost identical strings and
• a comparison of two substrings that literally represent the only differences between those two almost identical strings

The question -- is the difference between the two calls a bug? And if so, then which one is incorrect? And if not, then why?

Answers will be graded for accuracy, or short of that for how convincing the provided expository bullshit is, in an otherwise inaccurate answer....

(All posts will be moderated unless they do not give away the answers!)

The VkKeyScanW and VkKeyScanExW functions have a simple, documented functionality:

...translates a character to the corresponding virtual-key code and shift state...

Not the sort of thing you would need all the time, but it can come in handy.

They both take a WCHAR as a parameter and (from the documentation):

If the function succeeds, the low-order byte of the return value contains the virtual-key code and the high-order byte contains the shift state, which can be a combination of the following flag bits.

If the function finds no key that translates to the passed character code, both the low-order and high-order bytes contain –1.

Bit	Meaning
1	Either SHIFT key is pressed.
2	Either CTRL key is pressed.
4	Either ALT key is pressed.
8	The Hankaku key is pressed
16	Reserved (defined by the keyboard layout driver).
32	Reserved (defined by the keyboard layout driver).

Seems simple enough, right? :-)

Unfortunately, things are never as simple as they seem when it comes to keyboards.

The other day, friend Gregory called me up to ask me a question about the keyboard he created with MSKLC.

On his keyboard, he had added the same letter in two different spots on the keyboard (one on the VK_OEM_102 key, and the other on the ALTGR+VK_OEM1 key).

So he was looking at some code he had that used the VkKeyScan function, and it suddenly occurred to him that he had no idea which VK/SC it would return. After trying it out he found it returned the one he didn't actually want it to (the one on the ALTGR+VK_OEM1 key).

And now, he figured since he knew the person who did the development work on MSKLC that he could ask what the deal was here.

After I explained it to him, I thought it might make a nice blog entry!

I'll start off by saying that VkKeyScan has some limitations, starting with the one Gregory ran into -- when a function's purpose is to return the Virtual Key and the shift state, there is only so much it can do when there are two answers to the question, both equally valid.

And the decision in the case of a keyboard created by MSKLC is not one that can be controlled within the tool -- it is in essence controlled by the order in which the entries in the LAYOUT table are written (that order is deterministically created by MSKLC as matching the order of the keys in the US keyboard, and then putting the VK_OEM_102 key almost at the end).

I pointed out to Gregory that he had to pick the one key that would pretty much always lose the battle here. :-)

Now VkKeyScan has other limitations (documented and undocumented).

Such as the fact that it will not work with ligatures.

Or dead keys.

Or SGCAPS.

Or anything on the numeric keypad.

There is also that it does not distinguish between the left and the right SHIFT/CTRL/ALT keys, which rules out all of those more complex shift states like the RIGHT CTRL key on the Canadian Multilingual Standard keyboard layout. Or any other complex shift states you can set up.

Now that last paragraph talks about features not in MSKLC, but that I'll be covering soon enough in that series I am working on about interrogating keyboard layouts....

But in any case, you get the point -- VkKeyScan and VkKeyScanEx will definitely never be able to be confused with functions that have all the answers. Since they have been around since Windows 95/NT 3.1, I can think of two possible reasons for this problem. Either

• much of this additional functionality grew around them and after them, or
• they were written to handle a specific limited scenario needed by Windows and were only exposed due to someone thinking they might be useful.

The obvious question that comes up at this point is whether the limitation in MSKLC is one that would be considered a bug to fix.

Interesting question, and from a triage standpoint an interesting issue to consider. There are a lot of relevant facts here:

• There is no shortage of unfixable limitations of what would be useful functionality in VkKeyScan and VkKeyScanEx, which makes fixing this issue of limited use (this is sometimes thought of as the "why fix the screen door when the roof has blown off the house" theory of bug triage!);
• The fact that keyboards with duplicate entries for letters is on the whole relatively uncommon;
• Exposing the "solution" here is fairly complicated when one tries to consider what the user interface for such a change would be;
• Addressing the issue would involve some decidely non-trivial changes to the way that keyboard layouts are loaded from and persisted to .KLC files.

But maybe it would make a nice KB article at some point. Or a blog entry up here, some day....

If it helps, the sample I am putting together in that other series of posts will not suffer from any of those limitations from either VkKeyScan or VkKeyScanEx! :-)

Though I am sure those who are following that series will understand why a new version of those functions that addressed the limitations might not be likely. It is pretty difficult to extract that info!

Which is not to say that a generic function that took and end result of one or more UTF-16 code units and returned the exact set (or sets) of keystrokes that would produce it would not be useful....

But conceptually where does one draw the line between requiring two or more keystrokes for dead key entry but not allowing the typing of two or more separate characters for combining forms? Perhaps keyboard authors would be willing to make the distinction, but what about the rest of the world?

This post brought to you by "ĩ" (U+0129, a.k.a. LATIN SMALL LETTER I WITH TILDE)

Marc Brooks asked in the Suggestion Box:

In an ASP.Net 2.0 application, I want to fill a combobox (and my own internal lists) with the list of CultureInfo.

That's easy.

But how can I only include the ones whose localized resources (e.g. the correct version of the .Net runtime) has been installed?

I want my application to automatically "offer" the ones that will have localized Framework UI available.

It's funny, until I looked into Mark's question, I had no idea that there was not a way to enumerate the language resources in managed code (akin to the unmanaged EnumResourceLanguages, with the managed resource model in mind).

But after spending a little time it was clear that there was nothing like this in the System.Resources or related namespaces (that I could find, at least). The resource model clearly seems to rely on more of a "let the user choose and fall back p.r.n." mechanism than an "enumerate and choose" mechanism.

This seems mildly ironic to me given the fact that the CurrentUICulture in client apps relies on the UI language in Windows -- which is explicitly enumerated for the user, who gets to choose a language from that enumeration. :-)

But I am stubborn, so I kept digging and came up with the following:

using System;
using System.IO;
using System.Reflection;
using System.Globalization;

namespace Test
{
    class ResourceEnum {

        [STAThread]
        static void Main() {
            // Grab a type that we know is in mscorlib
            Type type = Type.GetType("System.Object");
            Assembly assembly = Assembly.GetAssembly(type);
            Console.WriteLine(assembly.CodeBase);

            // Enum through all the languages .NET may be localized into
            foreach(CultureInfo ci in CultureInfo.GetCultures(CultureTypes.SpecificCultures | CultureTypes.NeutralCultures)) {
                try {
                    Assembly satellite = assembly.GetSatelliteAssembly(ci);

                    // If we made it this far, we have the resources
                    Console.WriteLine("\t" + ci.Name);
                }
                catch(FileNotFoundException) {
                    // Swallow this exception, it means no such 
                    // resources exist for the given language
                }
            }
        }
    }
}

Since I already had all of the 1.1 and 2.0 .NET language packs installed, the above code gave me a nice list:

E:\test>csc resource.cs
Microsoft (R) Visual C# 2005 Compiler version 8.00.50727.42
for Microsoft (R) Windows (R) 2005 Framework version 2.0.50727
Copyright (C) Microsoft Corporation 2001-2005. All rights reserved.

E:\test>resource.exe
file:///C:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll
        ar
        zh-CHS
        cs
        da
        de
        el
        es
        fi
        fr
        he
        hu
        it
        ja
        ko
        nl
        no
        pl
        sv
        tr
        pt-BR
        pt-PT
        zh-CHT

Now the code is tied to mscorlib.dll, but it could be generalized into any assembly (by picking a different type, or using GetExecutingAssembly to get the application itself).

This post brought to you by "ጂ" (U+1302, a.k.a. ETHIOPIC SYLLABLE JI)

Andrew West has started a very interesting series entitled Unicode Character Names.

Check out Part 1 (the Good the Bad and the Ugly) and Part 2 (a Name is for Life). And if you are like me you'll be waiting and hoping for additional parts in the future! :-)

My favorite piece so far is at the end of Part 2:

Since the merger between Unicode and ISO/IEC 10646 only two characters have ever changed their name, namely U+00C6 and U+00E6, which were originally called LATIN CAPITAL LETTER A E and LATIN SMALL LETTER A E in Unicode 1.0, then changed to LATIN CAPITAL LIGATURE AE and LATIN SMALL LIGATURE AE in Unicode 1.1 after the merger with ISO/IEC 10646, and finally changed to their current names LATIN CAPITAL LETTER AE and LATIN SMALL LETTER AE in Unicode 2.0. The latter change was due to representations by the Danish Standards Association who considered these two characters to be letters rather than ligatures; but this caused so much trouble and acrimony that the respective committees of Unicode and ISO/IEC 10646 resolved never again to make any name changes, regardless of the severity of the mistake or the triviality of the change required (see the Unicode Standard Stability Policy).

Why is it my favorite?

(perhaps I should say favourite given the British slant on names?)

Well, it is an issue that comes up a lot and people simply don't appreciate what a nightmare it would be to deal with a constant flurry of name changing requests. There would be no time to encode any actual characters!

In any case, I am glad that BabelStone is on the lists of blogs I read, because if it were not I would miss gems like this....

(No, this is not a post about anyone breaking up with me and telling me that they need their space)

In Microsoft's implementation of collation, we have several different categories of characters, and rules for dealing with each category.

One of the interesting categories is the SYMBOL category. All of the miscellaneous odd symbols show up here. And they all come before the various letters and numbers.

Of course, as a feature most of the symbols do not really have any linguistic meaning that would foster a set of rules for how to sort them. So as I pointed out in Not all characters are created equal: take SYMBOLS, for example, there must be some order within the symbols "block", and the order is usually arbitrary.

And that gets us back on topic, to U+0020 (a.k.a. SPACE). It is a symbol, too.

"But wait,Michael!" you may be crying now. "The space actually represents an absence of symbols, or numbers, or letters, or anything. So it should not be a symbol!"

Well, this gets us kind of existential, which collation usually tries to avoid. It is based on expected behavior. So let's try some thought experiments to see where expected behavior leads us.

If you were comparing the strings "Microsoft" and "Micro soft", would you expect them to be equal?

Probably not.

But if SPACE were given no weight in collation, then they would always be identical. And what is more, the name Ray Mond would show up in the Exchange global address book after Raye. And all kinds of other weirdnesses.

So, it has to have some weight.

As perhaps a psychic nod to those who are philosophically against treating SPACE as a symbol, it is the very lightest of the true symbols. And from a behavior standpoint everything works, as long as you do not pass that NORM_IGNORESYMBOLS flag to CompareString and LCMapString.

This last paragraph may make some people wonder what I meant when I mentioned "true symbols" -- what are the symbols that are not true to us? Am I actually talking about relationships at this point, even though I said I was not?

I did not change my mind on the subject, I promise. :-)  I am simply talking about a subcategory of symbols that are treated specially which weigh even less than the space -- the punctuation. They are the ones affected by word sort vs. string sort decisions (as I discuss here), and will weigh either less than the regular symbols (in the case of string sort) or less than even the difference between uppercase and lowercase letters (in the case of word sort).

Let's see some of this in action. If we look at the sort keys for several of these situations, what is happening underneath becomes more obvious:

Microsoft

0E 51 0E 32 0E 0A 0E 8A 0E 7C 0E 91 0E 7C 0E 23 0E 99 01 01 12 01 01 00

Micro-soft (word sort)

0E 51 0E 32 0E 0A 0E 8A 0E 7C 0E 91 0E 7C 0E 23 0E 99 01 01 12 01 01 80 1B 06 82 00

Micro-soft (string sort)

0E 51 0E 32 0E 0A 0E 8A 0E 7C 06 82 0E 91 0E 7C 0E 23 0E 99 01 01 12 01 01 00

Micro soft

0E 51 0E 32 0E 0A 0E 8A 0E 7C 07 02 0E 91 0E 7C 0E 23 0E 99 01 01 12 01 01 00

Microsoft / Micro-soft / Micro soft (NORM_IGNORESYMBOLS)

0E 51 0E 32 0E 0A 0E 8A 0E 7C 0E 91 0E 7C 0E 23 0E 99 01 01 12 01 01 00

If you ignore symbols, they are all the same, otherwise the specific issues with the space, the hyphen, and word/string sort come into play.

Perhaps SPACE could have been a part of some bold new category that is not a symbol, but things are as they are -- and as it stands this returns intuitive results in most cases....

This post brought to you by " " (U+0020, a.k.a. SPACE)

Previous posts in this series: Part 0, Part 1, and Part 2.

Ok, we are making some progress here, and we are at the very least no longer stomping on the user's own keyboard list.

But we are ignoring dead keys and ligatures. Which, once again, is quite lame.

The key here is to have a little more respect for the return value of ToUnicodeEx. Right now we do nothing with the resulting string unless the return value is 1. But there are three other possibilities:

• If it's a dead key, the result will be -1;
• If it's a ligature (by which I mean the keyboard definition, a string of 2-4 UTF-16 code points);
• If it fails, the result is 0.

Clearly the only case where we want to do nothing is when 0 is the return value; in all other cases we want to do something. So let's fix that....

(As before, the older code is gray, the new code is black)

Now a few different things happened here. First, any time the return of ToUnicodeEx is greater than zero, all of the code points are dumped out.

Secondly, any time it is less than zero, it is known to be a dead key, which as I point out in this post are always limited to a single UTF-16 code unit. So we grab that one code unit and use it.

Thirdly, in that dead key case a second call is made to clear out the buffer -- otherwise the next call will be contaminated by the dead key value and will return either a different character entirely or two separate characters. Neither of those situation is too terribly desirable, so the buffer is cleared out.

(In an upcoming post I will explain why I chose VK_SPACE as the character for clearing out the buffer.)

It is very important to pay attention to that return value and never look past it when the string is not null terminated (and there is no guarantee that it will be). In fact, let's look at the return values table from the documentation:

Of course I am assuming people never mistype a dead key combination and thus tend to think of that "2 or more" case as being for ligatures -- certainly in the code provided the only case that applies is the ligature one (since it never combines with other characters!).

Ok, we are making progress now -- dead keys and ligatures. But we are still missing some important pieces like:

• the base characters that go with the dead keys and the composite characters they create
• the easy shift states
• the CAPS LOCK key
• the harder shift states
• chained dead keys

Obviously we'll need something a bit smarter in the way of algorithm for some of these; this will be happening too. Remember that the main point of this sample is to show off some of those lessons that can be gleaned from this stuff....

This post brought to you by "3" (U+0033, DIGIT THREE)
A Unicode character that is in the very small family of those whose VK value is the same as it's code point!

Previous posts in this series: Part 0 and Part 1.

The most immediate and nagging problem for me is the fact that the UnloadKeyboardLayout call will unload the keyboard even if you had it loaded already as one of your many "installed" keyboard layouts.

That stinks.

But the problem is that there is really no way to know if a keyboard is already loaded other than using the following algorithm:

• Load the installed keyboard list;
• Load the keyboard you want to use;
• Compare the HKLs in that list the one you are using;
• If it is not in the list you loaded, then it is okay to unload.

To get that list, there are two different approaches -- one managed and one unmanaged. The unmanaged one uses the GetKeyboardLayoutList function, and the managed one uses the static InputLanguage.InstalledInputLanguages property.

Let's take a look at two possible ways to do this (the older code from the post #1 is gray, the new code is black).

#1: The managed solution:

And #2, the unmanaged one:

This post brought to you by "2" (U+0032, DIGIT TWO)
A Unicode character that is in the very small family of those whose VK value is the same as it's code point!