November, 2005

Brad Abrams posted about The SLAR (vol2) on System.Globalization.CultureInfo and included quotes from members of the GIFT team (Shawn Steele and Ihab Abdelhalim).

One bit in particular stood out in my mind:

Except for Invariant, don’t expect the culture data provided by this class to remain the same, even between instances running as the same user.

Back in January I was talking about the new compiler error C4819 and how the compiler detected invalid characters.

And anyone who has been reading here knows that the reverse solidus is always the path separator, even when it looks like a yen or a won.

So among the so-called 'ANSI' code pages, ASCII (0x00 - 0x7f) will roundtrip 100% of the time.

How many "invalid" slots are there in the 'ANSI' code pages in the 0x80 - 0xff range, exactly?

Let's take a look at the Windows code pages:

• 874 (Thai) - 31 invalid
• 932 (Japanese Shift-JIS) - 5 invalid, 45 lead bytes, 15 reserved lead bytes = up to 65 invalid
• 936 (Simplified Chinese GBK) - 1 invalid, 126 lead bytes, 0 reserved lead bytes = up to 127 invalid
• 949 (Korean) - 2 invalid, 124 lead bytes, 2 reserved lead bytes = up to 128 invalid
• 950 (Traditional Chinese Big5) - 2 invalid, 87 lead bytes, 39 reserved lead bytes  = up to 128 invalid
• 1250 (Central Europe) - 5 invalid
• 1251 (Cyrillic) - 1 invalid
• 1252 (Latin I) - 5 invalid
• 1253 (Greek) - 17 invalid
• 1254 (Turkish) - 7 invalid
• 1255 (Hebrew) - 23 invalid
• 1256 (Arabic) - 0 invalid
• 1257 (Baltic) - 12 invalid
• 1258 (Vietnam) - 9 invalid

There you have it. Code page 1256 is the only one that is guaranteed to be able to roundtrip every single code point without losing any of the bytes....

This post brought to you by "¿" (U+00bf, INVERTED QUESTION MARK)

• RtlCompareUnicodeString has a simple job -- take two strings and return 0 if they are equal, < 0 if string1 is less than string2, and > 0 if it is greater, in a binary/ordinal sense. Of course this is an ordering that will be unsatisfying to any typical user, and I would suspect most of the atypical ones.
• RtlEqualUnicodeString has an even simpler job -- take two strings and return TRUE if they are equal and FALSE if they are not, in a binary/ordinal sense.

Now asking a candidate to write the ideal implementations for each of these functions and then with both of them on the board going through the expected differences in performance between them could make for a mildly interesting Microsoft interview question. But beyond that, the difference is not all that significant (in that post last week I hinted at this and the fastest underlying implementations when I mentioned that "the difference of two numbers not being zero" and "two numbers not being equal" was really not going to be all that significant).

The key issue here is that these functions are looking through the two strings. As soon as they note any difference of any kind, they can return the result. Period.

The world of the less lexicographic and more linguistic CompareString and mythical EqualString functions is a very different one. Because in this world those linguistic weight tables are used.

This is hardest on CompareString where with those weights there are so many different levels of difference, with some levels trumping other levels. Therefore, any difference between the two strings is an interesting bit of trivia unless no greater difference is found later on. Those lower levels include all of the potentially ignorable distinctions in e.g. case, diacritic, width, or kana. Unless a Unicode weight difference is found (which lets CompareString return more quickly), it will have to enumerate the full strings, storing up those lesser differences in case it needs them.

And this is where the fictional EqualString function would have it easier -- because like its binary/ordinal cousins it would be able to return after any difference is found, of any weight. This is (potentially) hugely faster for so many of the potential strings that CompareString has to test for.

This should answer the first question (why the performance difference is so... well... different).

The second question (if the Rtl* difference is so near to inconsequential yet they get the two functions why the NLS side of the world would not get two functions if the performance difference is so bleeding consequential) is a bit more complicated.

There would be essentially two good reasons to consider adding such a function to a future version of Windows:

• The need to have separate comparison/sorting operations to provide appropriate linguistic support;
• The need to have such an absolute linguistic equality operation done frequently enough that the perfomance difference would be significant in implementations.

The linguistic support argument is a tough one for two reasons:

• Existing implementations will by now be depending on the fact that we combine the operations of comparison and sorting, so we could not remove one functionality without breaking callers.
• In most cases, there really is no difference that would confuse anybody -- in fact to date I do not know of any complaint beyond the ones that I directly inspired by pointing out the differences to native speakers of particular languages either to directly answer the question or in this blog! This suggests that the difference is more theoretical in the real world of what people would be sorting/comparing on computers.

The performace question is harder to discount, although it significant to note that good implementations that are building indexes would be using the LCMapString function with LCMAP_SORTKEY flag to build sort keys. Obviously such sort keys are numbers already and allow both comparisons and sorts to be done even more quickly than the imaginary EqualString function could do its work. And it is hard to argue that "Microsoft must add something fast" if the person asking is actually not using something faster already....

In fact, sort keys bring the linguistic comparison down to the level of the binary/ordinal world where RtlCompareUnicodeString and RtlEqualUnicodeString do their work. And in using sort keys the difference between the two different oprerations of comparison and sorting is on the same order of magnitude as between the binary/ordinal functions!

The other problem is that in the small number of scenarios where sort keys are not a practical solution, a linguistic comparison is not appriopriate (e.g. filenames and other symbolic identifiers). That would mean that converting the unreal EqualString into a real NLS API would actually encourage bad usage and bad development practice!

(I will talk more about this issue with symbolic identifiers another day, and also it will cone up in an upcoming presentation at the Internationalization & Unicode Conference on March 6-8 entitled 'Tales of Incorrect String Comparisons'.)

It may be of interest to some that the new in Vista FindNLSString function probably has more in common with the fairy-tale EqualString than it does with CompareString since it too pays no attention to non-ignored differences other than to keep looking for the substring. However, the fact that FindNLSString is also looking for the location within the substring causes it to be slower to answer a different question; the mythological EqualString function would still be faster.

(I will talk more about this issue with the comparison and contrast of FindNLSString, CompareString, and LCMapString with LCMAP_SORTKEY flag another day!)

But irregardless of whether it had been able to justify its future inclusion, the arguments for and against the invented¹ EqualString are certainly worthy of a blog post!

1 - Special thanks to the Microsoft Word 2003 Thesaurus for its support here in finding appropriate words to use for the unreal EqualString function!

This post brought to you by "ß" (U+00df, a.k.a. LATIN SMALL LETTER SHARP S)
(A letter that was overheard in the character locker room after reviewing this post muttering "Curses! Folied again...")

That is right, the 29th Internationalization and Unicode Conference will be going on soon, and you can be there, too! The conference will be happening on March 6-8, 2006 in Burlingame, CA, USA.

The Unicode Consortium press release is right here.

And the initial agenda for the conference is right here.

There are two presentations I will be doing at the conference:

• Sorting it all Out: An Introduction to Collation
• Tales of Incorrect String Comparisons

Though admittedly I may be lobbying them to reverse the ordering (they have the intro talk second rather than first!). I'll talk further on this as I know something further....

Both talks should be pretty cool, as there is a lot of new information in both of them that can affect security, performance, proper internationalization, work in Whidbey, Vista, prior versions, and more!

I'll be talking more about these talks and related issues as the conference start date draws nearer....

Or, more properly, is the hotel owned by (among others) U2, The Clarence Hotel in Dublin city center accessible?

Most properly is the simple question "is there an accessible entrance to the hotel restaurant?"

This was the question that was posed to the doorman of The Clarence Hotel as Chris Kiernan and I pulled up and we were admiring the flight of stairs and lack of visible ramps or elevators, heading to the dinner party celebrating colleague Adrian's graduation.

The doorman, looked at us, puzzled. I then added, realizing we were in the midst of another Screw language; it's about the dialect, baby! moments, added "for a wheelchair."

The wheelchair thing was spur of the moment, but I realized that I did know precisely how to refer to the scooter in Ireland. I don't think I had even seen one other than mine on any of my visits here!

The doorman was no longer puzzled at what we wanted, but he was clearly unsure about the answer. He offered an "I don't think so. (pause) Let me check." and he was off into the hotel for a few minutes.

Chris and I, while waiting, discussed the fact that the term "accessible entrance" was not very well known here, and had a small laugh about the fact that the term to use was simply not known to Chris, either.

Anyway, a few minutes later the doorman came out and apologetically said that no, there was no accessible entrance. Though we could of course  check the entrance on the other side if we wanted (he had probably been gone long enough to have walked to the other side and back, so our chances seemed slim, but we still had some hope).

As it turns out, calling this person a "doorman" was probably inaccurate, he must have just been a big fan of Jim Morrison with an official title of Doorsman. Because the door on the other side of the building was completely accessible and we made it straight in to the restaurant!

But in any case we had a wonderful dinner and went to a bar after that where I had my first pint of Guinness, an entirely non-memorable experience (though Chris tried a bit and was not pleased either; he suggested it had not aged properly. Whatever that means!).

All in all, very nice evening with some great conversations, and a chance to see Anna again (she had just moved from Redmond to Dublin).

And I am pleased to report that The Clarence Hotel in Dublin city center is quite accessible, even if the same cannot be said about all of the staff at the door....

About a month ago, I had someone ask me about the images that appeared in MSKLC for characters that had no visible representation, wondering whether they were part of a font or not. After I explained that they were, she asked me how this font was able to be used without it being on the machine (the font itself was not all that interesting to her, but the capability to do that with any font was).

Then this last week, Rick Engle was asking on an internal Microsoft distribution list whether a font could be included as a project resource.

So today I thought I'd take these two items and handle them both at once, since they are really the same item.

The answer is that yes you can do this, and it works quite well using:

• AddFontMemResourceEx/RemoveFontMemResourceEx for GDI (unmanaged code)
• The PrivateFontCollection class and its AddMemoryFont method for GDI+ (managed code)

The line between managed and unmanaged code can blur when you use managed code like WinForms, since some of the controls are unmanaged underneath (like the RichTextBox) and require you to use both techniques at the same time to help things work properly. There are several controls that fit into this category, so I generally find it better to be safe than sorry and just always include them both.

And it is especially important to use both techniques in Whidbey (VS 2005) since GDI/Uniscribe is used in many cases rather than GDI+, as I have discussed previously.

Basically, start by adding the font file to your project, right-clicking on the file in the Project Explorer, and choosing 'Embedded Resource.' The name you will load is the font file with a fully qualified name (project namespace included), e.g. WindowsApplication1.MyPrivateFont.ttf.

Here are in namespaces we will need:

using System.Reflection;
using System.Drawing;
using System.Drawing.Text;
using System.IO;
using System.Runtime.InteropServices;

And here is the basic code:

// Adding a private font (Win2000 and later)
[DllImport("gdi32.dll", ExactSpelling=true)]
private static extern IntPtr AddFontMemResourceEx(byte[] pbFont, int cbFont, IntPtr pdv, out uint pcFonts);

// Cleanup of a private font (Win2000 and later)
[DllImport("gdi32.dll", ExactSpelling=true)]
internal static extern bool RemoveFontMemResourceEx(IntPtr fh);

// Some private holders of font information we are loading
static private IntPtr m_fh = IntPtr.Zero;
static private PrivateFontCollection m_pfc = null;

/////////////////////////////////////
//
// The GetSpecialFont procedure takes a size and
// create a font of that size using the hardcoded
// special font name it knows about.
//
/////////////////////////////////////
public Font GetSpecialFont(float size) {

   Font fnt = null;

   if(null == m_pfc) {

      // First load the font as a memory stream
      Stream stmFont = Assembly.GetExecutingAssembly().GetManifestResourceStream(
                              "WindowsApplication1.MyPrivateFont.ttf");

      if(null != stmFont) {

         // 
         // GDI+ wants a pointer to memory, GDI wants the memory.
         // We will make them both happy.
         //

         // First read the font into a buffer
         byte[] rgbyt = new Byte[stmFont.Length];
         stmFont.Read(rgbyt, 0, rgbyt.Length);

         // Then do the unmanaged font (Windows 2000 and later)
         // The reason this works is that GDI+ will create a font object for
         // controls like the RichTextBox and this call will make sure that GDI
         // recognizes the font name, later.
         uint cFonts;
         AddFontMemResourceEx(rgbyt, rgbyt.Length, IntPtr.Zero, ref cFonts);

         // Now do the managed font
         IntPtr pbyt = Marshal.AllocCoTaskMem(rgbyt.Length);
         if(null != pbyt) {
            Marshal.Copy(rgbyt, 0, pbyt, rgbyt.Length);
            m_pfc = new PrivateFontCollection();
            m_pfc.AddMemoryFont(pbyt, rgbyt.Length);
            Marshal.FreeCoTaskMem(pbyt);
         }
      }
   }

   if(m_pfc.Families.Length > 0) {
      // Handy how one of the Font constructors takes a
      // FontFamily object, huh? :-)
      fnt = new Font(m_pfc.Families[0], size);
   }

   return fnt;
}

Now this code is assuming you are only ever adding one font but plan to ask for different sizes of it; if you need to expand it to use multiple fonts then you will obviously want to take that PrivateFontCollection object and that IntPtr for the unmanaged font and convert them to be a collection or something to hold more than one font, perhaps using a Hashtable that index by the name?

Although I was tempted to make this procedure more generic, the requirement for the font to be an embedded resource and the fact that I am technically on vacation made this seem a little less critical for a quick sample. People who are interested can obviously do what they need to here, of course!

For cleanup, you will eventually want to dispose of your PrivateFontCollection and call RemoveFontMemResource, though for the latter the documentation claims that when the process goes away, the system will unload the fonts even if the process did not call RemoveFontMemResource (so you can base whether to call that on how clean you want your code to be!).

Those are the basics and should be enough to get you started....

As I think about the way that private fonts are implemented in both GDI and GDI+, a possible model for a future version of a "private CultureInfo" seems to emerge.

It is obvious that if you are creating a CultureInfo that can be enumerated and used by everyone on the machine that you should have administrative permissions. But perhaps this model could be used to create a more private CultureInfo for use in a particular application. Just as with the private fonts you could not enumerate them or use them systemwide, but there already is a mechanism to get those things done. Is it something you could use in an application yourself.

Seems like an interesting idea, right?

This post brought to you by "‰" (U+2030, a.k.a. PER MILLE SIGN)
(A character that was quite impressed to see that I managed to add something 'international' at the end of the post!)