March, 2006

As I mentioned before, I'm not a big fan of blog stats. I really do seem to break a lot of the obvious, common sense 'best practices' to help drive traffic to the blog (for example, any time I hit a topic that gets a lot of traffic I stay away from that topic as if I had just picked up something that I really want to set down and walk away from).

But also as I said in that other post, I was really interested in the whole Web Views vs. Aggregate Views ratio. Since the latter is basically the subscribers while the former is the people who look at individual posts in the browser (either directly or via clickthrough), the ratio just seemed interesting to me.

Here is the info for the last 11 months (thanks to Calvin Hsia for the script that provided the data; I stripped out the actual raw numbers since they are not relevant here):

I kind of like the direction things seem to be going in, since as I tend to syndicate entire posts rather than excerpts, the only good reasons to click through are

I like it mainly because I don't always have the answers, and sometimes the comments do. Lots of good resources get put in there!

Of course these raw numbers don't give much indication of what kinds of posts interest people enough to look at them directly, and just before I started looking at that I realized that it would make me like someone who was having their behavior driven by the clickthrough rate. Which is something I'd like to avoid for the time being.

So suddenly the numbers aren't as interesting to me as they were a short while ago. Fancy that!

Or perhaps I am just over-analyzing it now. Hugh had a great cartoon the other day that may be relevant:

Kind of puts it all in perspective, I think.... :-)

So, in the end I guess I'll even quit looking at these numbers too. Maybe nobody is reading the blog -- hell, maybe even the people who comment don't bother to read it. As I pointed out in this post, sometimes I need to be reading what I write.

On the other hand, maybe lots of people are reading. If so I hope you are enjoying yourself. I'll write up a slightly more relevant post right after I get this one up....

The other day, I got an email from James Todd:

Hello Michael,

My name is James Todd and I’m a new employee SDE in the SQL Server org in building ##. I’ve been reading your blog externally for a while now, and on rare occasion I posted on it as ‘James Todd’. (You may recall a post of yours about a problem with unicows corrupting the floating point control word.)

Anyway, I just wanted to say that reading your blog was always educational for me, and it helped me decide to come to MS. I’m happy to be here, and I just bookmarked your blog here on my new machine. J

Thanks!

James Todd

Now stuff like that happens to people like Gretchen all the time, but it is something new for me so I figured I would post about it....

While I am on the subject of jobs and hiring and such, I thought I would point out two different roles in GIFT test that we are looking for the right person for (look here and here for the listings).

As I have mentioned before, International Test is an art, and there are few fine artists. We have a small cadre of them now, but we are looking to make it a little bigger, so if you are interested you can take a look. :-)

It was not all that long ago that I talked about The Lack of Uniscribe samples.

In the meantime, James Brown hass put together some very interesting docs with samples -- definitely worth a look!

Excerpted from his announcement in the microsoft.public.win32.programmer.international newsgroup:

The thrust behind the tutorials is to document the development of "UspLib", a Win32/Uniscribe text-rendering-engine which is similar in many ways to the ScriptString API, but with the additional capability of being able to draw fully styled+coloured text using user-supplied style-runs. It is still very much work-in-progress but is fully functional and should be suitable for any win32 application which requires basic styled-text support (such as syntax-colouring text-editors)..

Part 11 - Introduction to Uniscribe
www.catch22.net/tuts/editor11.asp

Part 12 - Uniscribe Mysteries (ScriptItemize/Layout)
www.catch22.net/tuts/editor12.asp

www.catch22.net/tuts/editor13.asp

Part 14 - Drawing styled text with Uniscribe (ScriptTextOut)

www.catch22.net/tuts/editor14.asp

Part 15 - Integrating UspLib (documents UspLib)

www.catch22.net/tuts/editor15.asp

You can see Neatpad in action (and download the source) here:
www.catch22.net/tuts/zips/editor15.zip

Any feedback would be appreciated, especially if you notice any glaring errors/omissions.

Enjoy, and be sure to get James any feedback you have about these chapters or the samples.

Thanks, James!

(No, this is not a post about relationship advice!)

The other day, Karthik asked:

Hi Michael,

The issue is: GetWindowTextLength returns 0 for any edit control that has Chinese Pinyin IME or Japanese IME characters in it.... GetWindowText also returns nothing. Hiding the field not make these characters disappear....

...This is an issue with many dialog boxes hosted in property sheets that I looked around at. It malfunctions with in certain ways with other dialogs also.

The issue here is that when you first type text using an IME, it is not yet in the underlying control that has the focus.

Let's say you are in Notepad and with the Japanese IME in full width katakana mode you type the letters kazuka. What will appear is something like the following:

See that dotted line? It is an important one.

If you hit ENTER now, the text will be committed and then it will be in the underlying control.

If you hit the space bar twice, the candidate window will come up, giving you whatever options are available:

(in this case all that is available is the Katakana and Hiragana versions of ノチツナノチ a.k.a. 30ce 30c1 30c4 30ca 30ce 30c1)

Once you either hit that initial ENTER key or choose from the candidate window, you will have text in the underlying control.

But until then, any call to functions like GetWindowText will not return this text and functions like GetWindowTextLength will return 0. Other operations such as trying to hide control text will have other result consistent with the fact that control itself has not yet been given the text.

Because when you are using the IME, just as in life, you often have to commit before you can get what you are looking for....

(Some other day I will talk more about what to do if you want the interim text in the IME)

This post brought to you by "ツ" (U+30c4, a.k.a. KATAKANA LETTER TU)

Alistair asked me via email:

V.quick question (I am contacting you as I can see you are an expert in this area from your excellent blog entries ;-)

Do you happen to know how I can automate changing the location listed in the "Location" field in the "Regional and Language Options" control panel (first tab and the bottom)

I need to change this for imaged installs and / or post install.

I can use sysprep.inf [RegionalSettings] and / or rundll32 (rundll32.exe shell32,Control_RunDLL intl.cpl,,/f:"<file.inf>").
This changes most of the stuff I need to change fine.

However "Location" listed in this dialog does not seem to change irrespective of what I enter.

Do you happen to know what sysprep.inf entry maps to this dialog. Also if it's possible to modify this by using rundll32 ?

Well, there is of course a function call that can do it -- SetUserGeoID. I know this does kind of violate the bit of social engineering we have in place to try to keep developers from calling functions to change user settings, but we inherited this function. :-)

But, as Alistair indicated, that KB article does not detail a way to change the location setting from sysprep.inf or the later run with intl.cpl directly (both of them go through the same engine).

As far as I have been able to gather, this setting is not available in the setup/pre-setup case -- perhaps we are lucky that no one seems to be using the GEO stuff?

(I am admittedly hoping that someone from the team will correct me if I am mistaken, though my hopes are not high at this point)

Note that in Vista we do support this particular setting, along with some other stuff (I'll be posting soon on the new features and syntax)....

This post brought to you by "" (U+0053, a.k.a. LATIN CAPITAL LETTER S)

I embarrassed myself a little bit at one of my presentations, yesterday.

I was doing my Tales of Incorrect String Comparisons talk, and I did the following:

1. On a machine with Windows Server 2003 and an early 2005 CTP build of SQL Server 2005 that supports Windows authentication, create two user accounts on the local machine, named 'AdminЀ' and 'AdminЀЀЀЀ'.
2. Open up SQL Server Management Studio, pick any database and run the following two queries:
   
    create user "MICHKAP3\AdminЀ"
    create user "MICHKAP3\AdminЀЀЀЀ"

As I am sure you can guess, that character at the end is one of those that have no weight....

The result for that second query:

Msg 15023, Level 16, State 1, Line 1
User, group, or role 'MICHKAP3\AdminЀ' already exists in the current database.

Basically, SQL Server is validating the entry using its own collation rules -- whether it is doing some sort of optimization here to avoid the performance hit of a call to do the Windows authentication, or whether it is simply storing the entries in a uniquely indexed column that does not allow its own collation rules to be broken, it is clearly not allowing two different valid Windows accounts to be used here.

(this was a problem I had reported early last year, along with another few bugs that were fixed, and I was told this one had been fixed)

Anyway, I then did the same steps on a different setup with a released version of SQL Server 2005 that I had spent half the night installing onto another VPC machine.

I mentioned as this VPC machine was loading that I had just installed it last night and had not even tested it out....

(note to self -- even if I am tired and even if I am not doing a billg demo, I should still check the demo, especially if I am sure that I know what will happen)

The result?

I was shocked.

I thought I covered okay by acting dramatic about the situation as if this was what I expected, but the folks who knew me (and at least one of the ones who didn't) noticed I was not expecting that result.

Damn.

Ah well, at least it was a good example of some type of comparison operation that was not being performed correctly.

(And it was not really as bad as the one with Chris and the USB scanner, or even the other one with Tod and the Access 95 demo, a story for another day!)

I think the other examples, of problems that had been fixed were for the most part technically cooler, though I suspect that the one they will remember was the SQL Server one.... :-)

This post brought to you by "Ѐ" (U+0400, a.k.a. CYRILLIC CAPITAL LETTER IE WITH GRAVE)

(via the awesome Kate Gregory)

I sometimes wonder what all this blogging crap is worth. I guess I know now....

That is the worth if I use http://blogs.msdn.com/michkap/ but if I tack on the default.aspx that Community Server seems to like so much:

Now I have never really been one for defaults, but I never imagined the penalty would be so steep!

Of course since (a) you have to be a Microsoft employee to use this blog given the server it is on, and (b) only a complete and utter fool would really want to try to assume my online persona, I have to ask myself whether I could really do anythng useful with this knowledge?

Somehow I think that the bank would not be willing to consider it in deciding a credit limit. :-)

The other day, I was talking with a developer about a problem -- how to deal with a font linking choice and displaying some localized text based on it without requiring a reboot.

We talked through a bunch of options, realizing that all of them would require some core changes in GDI, and none of them were really all that feasible.

The truth is that I should have just started reading my blog a little bit more often, so I can remember to follow my own advice. :-)

As I point out in Font substitution and linking #3, font linking is a useful feature if you want to not specify a font and instead rely on the system locale's prefence for font linking. But in a localized dialog, one can simply let the localizer choose the right font and you will see the appropriate text as the localizer wanted it, without a reboot being required!

Because the truth is that font linking is a backup solution -- a strategy for how to get glyphs if you cannot find them. But it is by no means an assurance of getting the best results for a specific language -- so if you have localized resources for a particular language, you can pretty much always do better to not rely on GDI font linking to do the job!

This post brought to you by "一" (U+4e00, a.k.a. the first CJK Unified Idedograph)

Yesterday when I posted about The real problems with keyboard switching, I listed what were the three most common complaints I had heard previously about working with other language keyboards:

• The movement of some letters within a script;
• The movement of punctuation always;
• The lack of phonetic relationship between scripts.

And then I said I'd think about this since it seems like it should be easier to handle some of these things in a more automated fashion....

I ended up with a whole bunch of mail with people asking me what I meant -- what could be done here?

Well, I was wondering whether MSKLC could handle some of this a bit more gracefully?

For example, the first problem is caused in Latin keyboards by movement of the VK_* values, so what if there were an option to load existing keyboard #! but using the VK_* positions of existing keyboard #2?

Similarly, the second problem is caused by a combination of the first problem and a new one with the actual assignments being changed, so what if there were an option to load the letters of existing keyboard #1 but to use the symbols of existing keyboard #2, both for values and position?

The third problem could be a bit harder since the phonetic values, especially according to the letters between two languages, may be non-trivial. But building such phonetic keyboards is a situation where some sensible defaults could be attempted and then a little bit of swapping around could take care of the rest. Perhaps a "swap keys" functionality would be possible? (this could be useful in other situations too).

Now obviously there is much more that could probably happen and there may be better ideas (these are just off the top of my head and some of them may not be practical). But it is a start, right?

This post brought to you by "š" (U+0161, LATIN SMALL LETTER S WITH CARON)

(computerized apologies to Ray Charles for the title of the post!)

Will anyone forget when I asked the question What do you get when you combine a base character with a buttload of diacritics?

I was of course talking about fonts there. This time I am going to take a slightly different approach, and talk about collation.

I will give the string, the code points, and the sort key. We'll start simply, with one letter:

e
U+0065
0e 21 01 01 01 01 00

Now we will go with something a little more complicated (the difference from above marked in RED):

ẽ
U+1ebd
0e 21 01 19 01 01 01 00

or its alter ego in normalization form D:

ẽ
U+0065 U+0308
0e 21 01 19 01 01 01 00

Hmmm... let's look at another diacritic:

ê
U+00ea
0e 21 01 12 01 01 01 00

ê
U+0065 U+0302
0e 21 01 12 01 01 01 00

Ok, and now for the kicker:

ễ
U+1ec5
0e 21 01 29 01 01 01 00

ễ
U+0065 U+0302 U+0303
0e 21 01 29 01 01 01 00

But wait -- where did the 29 come from? I mean the first one had no DW (diacritc weight), and the next two had 19 and 12, respectively.

I had talked in previous posts about sort keys about how the minimal weight is 2, but that this weight would only be seen when it was needed as a placeholder, e.g. in the following string:

eễ
U+0065 U+1ec5
0e 21 0e 21 01 02 29 01 01 01 00

So, if you take that (sometimes invisible) 2 that as there on the 'e' always and combine it with the 17 on the tilde and the 10 on the circumflex, you get 29.

Easy.

Now what happens when you get that buttload of diacritics? Let's add them one at a time:

U+0065
0e 21 01 01 01 01 00

U+0065 U+0300
0e 21 01 0f 01 01 01 00

U+0065 U+0300 U+0301
0e 21 01 1b 01 01 01 00

U+0065 U+0300 U+0301 U+0302
0e 21 01 2b 01 01 01 00

U+0065 U+0300 U+0301 U+0302 U+0303
0e 21 01 42 01 01 01 00

U+0065 U+0300 U+0301 U+0302 U+0303 U+0304
0e 21 01 57 01 01 01 00

U+0065 U+0300 U+0301 U+0302 U+0303 U+0304 U+0305
0e 21 01 95 01 01 01 00

U+0065 U+0300 U+0301 U+0302 U+0303 U+0304 U+0305 U+0306
0e 21 01 a8 01 01 01 00

U+0065 U+0300 U+0301 U+0302 U+0303 U+0304 U+0305 U+0306 U+0307
0e 21 01 b6 01 01 01 00

U+0065 U+0300 U+0301 U+0302 U+0303 U+0304 U+0305 U+0306 U+0307 U+0308
0e 21 01 c7 01 01 01 00

U+0065 U+0300 U+0301 U+0302 U+0303 U+0304 U+0305 U+0306 U+0307 U+0308 U+0309
0e 21 01 06 01 01 01 00

U+0065 U+0300 U+0301 U+0302 U+0303 U+0304 U+0305 U+0306 U+0307 U+0308 U+0309 U+030a
0e 21 01 1e 01 01 01 00

U+0065 U+0300 U+0301 U+0302 U+0303 U+0304 U+0305 U+0306 U+0307 U+0308 U+0309 U+030a U+030b
0e 21 01 39 01 01 01 00

U+0065 U+0300 U+0301 U+0302 U+0303 U+0304 U+0305 U+0306 U+0307 U+0308 U+0309 U+030a U+030b U+030c
0e 21 01 4b 01 01 01 00

Uh oh! Eventually we wrap....

We only have one byte of space to store that diacritic weight (any more than a byte would run into the next character's byte), and when we run out there were really only three choices:

1. We could stop at 0xff and simply not added any more after that;
2. We could stop at 0xff and given an error;
3. We could wrap around.

The problem with #2 is that it pretty sharply limits what one could do in a potentially unpredictable way, and the problem with #1 is that all such strings would be equal. Now with option #3 there is a good chance that there will be a difference between strings being compared, though it will sometimes unfortunately make a string that is clearly greater than another string feeling like it is less than it -- a cure that may be worse than the disease....

Well, I won't argue whether one of the other choices might have been better; we are kind of stuck with it now (there are technically a few cases that wrap that are less theoretical than the case above, lest you try to dismiss the example as being a bit too unrealistic!).

But at least that answers the question about what happens when you try to collate a buttload of diacritics....

This post brought to you by "e" (U+0065, a.k.a. LATIN SMALL LETTER E)

I was at the birds-of-a-feather entitled "Design Principles for A Regional, Multilingual Keyboard" last night.

(More on some of the issues there another day....)

In any case, there was an interesting side conversation that brought into specific relief the real complaints that I have heard from users about the problems that come up when one handles multilingual text entry by switching to other keyboard layouts. The complaints generally fall into three categories:

• When the script is the same (e.g. both the original and new layout use the Latin script as in English and French), subtle differences exist in the location of some of the letters that both of them share (such as the switching of the 'A' and the 'Z' keys between English and French).
• In many cases, all of the punctuation moves around in specific ways that seem to be pretty difficult to keep track of in a reasonable way.
• Although people do not tend to complain as much about the letter positions when the script changes  (e.g. from Latin to Hebrew), in general they would usually prefer the new layout to have some sort of phonetic relationship with the old one to lower the learning curve.

Suddenly, it hit me that there may be other (and better) ways to try to address much of the above three issues without breaking features like language tagging via the input language.

I am going to keep thinking about this one and maybe I will have more to report on this in the future.

(In the meantime, I will remember that one of the cool things about MSKLC is that you can actually address all three of the above issues in many cases!)

I think it will be good to see what some of the results are with customers who use some of the multilingual keyboards like the ones described in the Swedish and Finnish standards. With so many smart people paying attention to the problem, I believe that only good things can come up in the future....

This post brought to you by "š" (U+0161, LATIN SMALL LETTER S WITH CARON)

The other day Al asked me via the Contacting Me link:

Hi Michael,

My question has nothing to do with internationalization, but it has something to do with sorting. I hope this makes it interesting enough!

When I group in a ListView, I cannot sort within the groups. Is that a bug?

The ListView class is a managed wrapper around the Shell ListView common control. When you use instances of the ListViewGroup class to set up groupings, the setting of the Sorting property is ignored.

Although at first glance it may look like the native Shell ListView_InsertGroupSorted macro/LVM_INSERTGROUPSORTED message, might be helpful, this is actually referring to sorting between groups, rather than within them....

This post brought to you by "Ḡ" (U+1e20, a.k.a. LATIN CAPITAL LETTER G WITH MACRON)

The chcp.com utility is a simple little program sitting in the \WINDOWS\SYSTEM32 subdirectory. Running it with /? willl give some helpful information about its purpose:

C:\WINDOWS\system32>chcp /?
Displays or sets the active code page number.

CHCP [nnn]

  nnn   Specifies a code page number.

Type CHCP without a parameter to display the active code page number.

There is also more information in the Windows XP documentation, which does hint at a problem in its small list of "supported" code pages:

None of the ACP values are there, though this is I think a bit of social engineering -- to keep people thinking of it as the OEM code page. The 125x series code pages also work well here.

However, another set that is missing from the list is the ideographic code pages. You cannot use chcp to change to one of the ideographic code pages unless it is also the default system OEM code page.

Thus on a system with an 0x0409 default system code page:

C:\WINDOWS\system32>chcp 932
Invalid code page

C:\WINDOWS\system32>chcp 936
Invalid code page

C:\WINDOWS\system32>chcp 949
Invalid code page

C:\WINDOWS\system32>chcp 950
Invalid code page

This is a known and expected limitation for which there is no workaround....

This post brought to you by "Ā" (U+0100, a.k.a. LATIN CAPITAL LETTER A WITH MACRON)

As I have mentioned before here, I will be speaking at the 29th Internationalization and Unicode Conference in San Fransisco and am heading out tomorrow.

Lots of the cool things that I talk about here and more will be coming up during the talks, and I assume in conversations I have with people while I am there.

If you are looking for either me or for one of the other [potentially much more] interesting Microsoft folks who are either presenting there or hanging out, here are the cool places to be that I know of:

• There will be a BoaF (birds of a feather) gathering entitled Design Principles for A Regional, Multilingual Keyboard, as discussed here, which I and at least one other Microsoft person I know of will be at;
• Microsoft will have a booth at the Exhibition (I have no idea why there is no text next to the MS logo here, but on the other hand who doesn't know what Microsoft is?), and I will be a booth bitch for a lot of the time, and other more useful MS people will be there at different times, too;
• Murray Sargent, Russ Rolfe, Marypat Meuli, Marin Millar, Michel Suignard, and li'l old moi are all giving presentations;
• Google is hosting a Welcome Reception on Monday evening from 6-7pm that will be in Sandpebble A, B & C, Lobby Level of the hotel the conference is at (Hyatt Regency San Francisco Airport), and the invite was opened to all Unicode members and aficionados, even those who have not registered for the conference, and I will be there too (per Bob Jung of Google, you can send mail or ask at the conference registration desk for more info;
• I am sure I will be checking comments in the blog (especially to this post) often enough if you will be there and want to say hello or whatever;

I look forward to seeing people at the conference! :-)

Yes, that is right, the Swahili Language Interface Pack is now available! :-)

Some info about Swahili:

Swahili or Kiswahili? - Swahili is often also referred to as Kiswahili. Swahili literally means "coast people" (and then "coastal language"), the word is derived from Arabic. Kiswahili is the name in the language itself, Ki- being a prefix for nouns of the class to which languages belong.

Swahili is widely spoken in Eastern Africa and parts of Central Africa. It is national language in Tanzania and Kenya and extensively used in Uganda and the eastern part of the Democratic Republic of Congo. It is understood in areas in Rwanda, Burundi, the Comoros Islands, the southern part of Somalia, the northern parts of Mozambique and Zambia, and even the northwestern coast of Madagascar. While it is native tongue for only 4-5 million people, 50 million people know Swahili as second language and use it as a lingua franca, making it the most widely understood language in Africa after Arabic.

Spoken originally by people along the Eastern African coast, Swahili became the major language for coastal trade. Since a lot of traders were Arabs the language began to be heavily influenced by Arabic from the 7th century on, later there were also influences from Persian, Portuguese and finally English (from which, for example, baisikeli or penseli come). The European explorers and then the colonial administrators spread and encouraged the use of Swahili.

There are lots of different dialects of Swahili in the vast area where the language is spoken. Standard Swahili developed from the urban dialect of Zanzibar city, Kiunguja.

So you can add it to that big list that includes so many others (such as Nepali, Konkani, Bengali and Malayalam, and all of the others....

Undeniably cool! :-)

This post brought to you by "!" (U+0021, a.k.a. EXCLAMATION MARK)