May, 2010

Okay, a little Spot the Bug for everyone on a Monday morning when one shouldn't need to be working anyway....

The other day a question came up about some code.

It looked something like this:

whcar_t* newParams = new wchar[MAX_PATH + MAX_PATH];
whcar_t* p = newParams;
while (*params) {
    *p++ = *params;
    params = ::CharNext(params); 
}
*p = 0;
return newParams;       

 Can you figure out what this code is doing? Assume it is compiled with UNICODE (since the compile would fail, otherwise!).

Well, I know what it is doing. The key is trying to understand what it was trying to do.

You can probably gather why whatever they were trying to do is doomed to failure, of course.

Also, you can look at previous blogs like The torrents of breaking CharNext/CharPrev if you want a quick refresher, of course.

Ignore the bug mentioned there, and think about what CharNextW does.

So, what does the code do, and what did the original authors want it to do?

Nothing technical in the work sense, you know the drill. Or should....

So one of the things that you can have if you work for Microsoft used to be a FlexPass to let you ride on Metro Transit or Sound Transit.

You know, a perk that is a way to encourage people to do The Right Thing™ in regard to not driving every day, etc.

As someone who seldom drove even when I had a car (before it died a few years ago), this was a tremendous boon.

Such a boon that once my car did die, I simply decided to try pretty more "independent of oil", only occasionally using a ZipCar or a taxi but usually either wheeling there with the iBot or taking the bus.

Then Seattle got rid of the FlexPass and moved to ORCA (One Regional Card for All).

Of course some people didn't like the "track-ability" that could be used with ORCA, and the promises that Microsoft made to never do anything like that were treated by people like me as cynically as one might expect.

But that was no biggee, I am used to my cynicism; that is The Right Thing™ for me.

But the bigger problem was that to get the ORCA I was "signing" an agreement that it would be used principally to get to and from work.

It didn't say exclusively, but the language implied (to me, a non-lawyer) that at least 51% of the time it would be work-related travel and/or travel to/from work.

Now I live across the street from work.

I can get to my office in under ten minutes in the iBot in Standard mode, and under twenty minutes in Balance mode (up on two wheels).

If, at the moment I head out my door, it is not beautiful weather and a 233 or 269 or 242 happen to be 30 seconds away, I'll take that bus four stops and get to work quicker. And the same thing going home. And if the bus is more than 2 minutes away I won't bother.

But most of my bus travel is to Kirkland, and Bellevue, and Belltown, and Capitol Hill, and Pioneer Square, and Ballard, and Fremont and so on.

If I was going to have to pay for some of the buses, I'd rather get the handicapped rate - which is roughly half. I'm just saying, you know?

But the ORCA I got from Microsoft wasn't that -- it was full fare ORCA.

I could put money on it for personal travel, but I'd be paying twice as much, every time I used it.

So I talk to the commute people at Microsoft and explain my dilemma. And I ask how to get the RRFP (Regional Reduced Fare Permit) ORCA, which would save Microsoft some money anyway, and let me save money when I had to pay.

They are shocked.

Which shocks me a bit.

Apparently it has never occurred to any of the people who "sign" that agreement that they might actually follow its guidelines. Or at least no one asked about it.

And they do nothing for RRFP or handicapped or over 65 stuff -- they just pay for the card and that is that.

So this is apparently The Right Thing™.

Of course I saved the email where I was explicitly told to not worry about violating the rules of the agreement. Just in case someone ever checked my usage later and saw how I was going all those places that weren't work (out of 100 trips that cross th bridge, maybe one or two is work-related -- maybe ten if it is just getting to work in the morning when I stayed over on the wrong side of the bridge!).

I have talked to a couple of the bus drivers, and they just suggested I scan the card every other time, which would make the money come out right. But that seems wrong too. That isn't The Right Thing™.

And since there is no way to get a RRFP ORCA with corporate sponsorship unless the company does the work, I am costing them a bit more money than I would have otherwise. I offered to try and reduce that but they declined. Because that wouldn't be The Right Thing™ either.

I can't help but feel a little disconnected from reality on all of this. Like every step of the way I try to do what seems like The Right Thing™ and I am told not to bother....

I can't decide if my moral compass is being magnetized, or if I just don't fully understand what The Right Thing™ is.

In other news (following up from that Arising from one's own ashes. Like [up to 80% of ]a phoenix.... blog from the end of April:

First an entirely gratuitous repeat of the Elvira shot:

The reason for that was that every person who talks to me about this drug officially other than the drug company is mispronouncing it. They want it to rhyme with Elvira!

I have received the Ampyra, and am taking it now. Well, not right now, but about every 12 hours.

Now insurance picked it up 100% -- but I swear I had to talk to a nurse in my neurologist's office, someone from the Ampyra company, someone from Medco, and someone from the specialty pharmacy entrusted to deal with expensive medicines (whose name I can't remember). Each one of them had checked with the insurance company itself and wanted to make sure it wasn't an error that insurance was paying 100% of the $1094.27 charge. Remember they also paid for the iBot within months if the iBot being discontinued for lack of insurance companies willing to pay, which surprised people too (though at least that one was on appeal; this one was practically automatic, within months of FDA approval!).

You know its weird when people who regularly deals with Microsoft insurance doubts it.

In fact, if you are a Microsoft employee you may want to consider running over me at an intersection or something. Our insurance is now being so generous that even the people who routinely deal with our insurance are wigging out over what they cover. This has to be a bad thing that will impact coverage in the long run, right? Sheesh!

Good for me may not actually be good for us!

Anyway, I am just a few days in (sent to me by UPS Next Day Air 5/27, I got it on 5/28) and it is already eligible for refill on 6/1. I'm going to wait to see if it looks like its helping. And maybe ask them to not bother with the next day air next time. Maybe the price can be driven down a bit....

If close to the end of the month it doesn't seem to be helping, I'll stop. No idea why they'd approve it for renewal so fast, are they getting a kickback or something? Sheesh. Again.

Plus the Twitterverse is awash with people talking about the high cost and low effectveness. Check here, you'll see what I mean. But I've also heard some positive reports (sites like this one have some of the typical comments you can see if you aren't looking at the news stories, tweets, and blogs that just parrot each other -- I am continuing to doubt that Twitter is the best way to find out truth in such circumstances!). Sheesh, the third.

Apparently the drug is helpful for some people, though the criteria they used to determine usefulness was not as good as it could have been.

But now for the Ampyra IRONY I promised.

As I actually expected, when you read the Ampyra literature, one of the potential adverse side effects for this medication meant to primarily deal with imbalance is...

...wait for it...

BALANCE PROBLEMS!

This is akin to Zofran causing nausea. Remember that one? Sheesh, the fourth.

So, how am I supposed to know if it is helping and I am just seeing the side effect, or it isn't helping? I'll let you know how the course goes but I'll admit to frank amazement that they have not overflowed their irony stack buffer.

Anyway, for now I am going to keep up the Ampyra and see what happens....

It has been almost a week since the new platform was rotated in, with its new version of Community Server and the new look and feel for MSDN Blogs and TechNet Blogs.

Raymond Chen included a Welcome to The New Old New Thing, 2010 edition for when the new site went live, and although I have done blogs like that in the past, I decided not to this time.

Since most of the problems pointed out get fixed right away anyway.

Anyway, if you find anything that looks wrong, from missing images to incorrect links to unreadable formatting in the thousands of blogs on this Blog, let me know. I'll try to fix it up.

On the whole I can't claim to be especially happy with the change; there is too much stuff that just looks wrong to me and that I really miss, especially things like collapsible panels. I'll get used to it eventually, or else eventually I'll dive in and figure out how to add them back (or Raymond will, he has threatened to do it already at least once!), but fotr the most part I am much more interested in adding stuff to the blog itself (aka CONTENT) rather than in the Blog (aka FRAMEWORK).

Maybe if the Blog were about blogging itself I'd have a different take on all of this.

So, getting back to the content....

I have been oscillating back and forth between being over a month ahead on blogs to being less than a day or two ahead.

The former is a bad thing for me since I keep going back and reshuffling and editing, which wastes time. The latter is kind of fun since I never know what I'll will see here until after its up.

It's weird, but the other day someone was asking if I would do some trainings for them and she kind of started the request off with "As one of the most popular faces in the Globalization world" which I am pretty sure made me blush when I read it; that sort of thing doesn't integrate cleanly with my own self-image.

But if more people are going to look at me as being less an outlier and more of an actual something or other in the space, I suppose I should pay them the compliment of taking it a little more seriously. :-)

One of big "backend changes here in the content department" is that for the first time ever I have a management structure above me that actually looks at Sorting it all Out as an asset to the group. This change has both good points and bad points to it, though the benefit of being able to write during normal hours while at work alone outweighs most of the bad I could list.

I suppose that has added a bit more discipline to the content adding process.

There is a part of me that realizes I could have moved the entire blog somewhere else during that period when I had management that didn't care that much for the blog, leaving nothing but a single-page pointer to the new location. Too late for that now!

I will get around to removing some of the specific "disclaimer" text I was required to add from that time when I was a little less appreciated than I am now. No hurry, just something I will have to get to eventually.

If you are a regular (or a consistent even though irregular!) reader than thanks for your support, I really do appreciate it.

If you count back to the very first blog here in the end of 2004 (very few of you can make that claim, I mostly know who you are though!), it has been a very long, strange trip.

And who knows what it will look like tomorrow? :-)

Okay, it all started with The inappropriate nature of getting the Feh out of Uighur, where I pointed out, due to rather frantic and passionate last-minute feedback about a reported problem with the Uyghur keyboard, a change was made, pretty much at the last minute.

And how, as it ended up, the report was in error: how the original keyboard was correct and the changed keyboard was in fact a break.

Then, not quite two years later, in The inappropriate nature of getting the Feh out of Uighur, Windows 7 edition, I explained how the bug was fixed by renaming that essentially incorrect keyboard to add a "(legacy)" to the name, and adding a new keyboard with the correct letter.

Not perfect to my way of thinking, mind you, since they chose not to add the collation equivalencies I suggested. But they may have actually considered that and gotten feedback to not do it. So that may have been the correct behavior; I don't know.

But all's well as ends well, right?

Well, not quite.

At the beginning of this week, someone with the handle One Uyghur posted a note in the Suggestion Box:

 Dear Michael Kaplan,

I hope, at least, this is the one way to contact you.

I am a Uyghur (Uighur) currently living in the US. We (a group of Uyghur Computing fans living around the world) have found some problems in Uyghur Keyboard Layout in Microsoft Windows 7.

We hope to discuss it with you about it, and if possible, we hope Microsoft can release a hotfix to correct the current problems.

Your notice to this problem and timely response are greatly appreciated.

Thank you.

One Uyghur

 

Hmmm.

Anonymous people. Always makes me worry a bit, since no information was included to help know what we're talking about.

So anyway, I hadn't quite gotten around to formulating the reply (which to be honest would have been little more than a request for the information that the specific 'group of Uyghur computing fans living around the world" could share to describe the problem that they wanted to report.

I took the precaution of digging up the work I did to make sure the whole alphabet was supported, which I didn't use last time. But I'll use it now, just cause. I mean I was hearing a vague report about keyboard problems. And we had seen those previously:

I verified the new keyboard had every letter (the ARABIC LETTER YEH WITH HAMZA ABOVE was put on a separate key, but everything here was typeable).

Other reports I had seen of the need for ﺉ and ﺌin Uyghur (that's ARABIC LETTER YEH WITH HAMZA ABOVE ISOLATED FORM and ARABIC LETTER YEH WITH HAMZA ABOVE MEDIAL FORM, compatibility characters in Unicode) would also not be needed -- they were handled already by having ARABIC LETTER YEH WITH HAMZA ABOVE there. This Hamza on a tooth (as I heard it called a few times), was covered.

The work I had done, I puit aside. I figured I'd write up my question about what exactly is missing at some point.

But the next day, One Uyghur apparently found the report of the fixed Windows 7 keyboard I mentioned in The inappropriate nature of getting the Feh out of Uighur, Windows 7 edition and left a comment to it:

Yes, the bug is fixed. But there are still some problems with this Uyghur keyboard layout.

I would like to know how Microsoft makes the local keyboard layout for some of the language. I mean, while making the Uyghur keyboard layout, which authority or standard does Microsoft follow?

I am a Uyghur and among the 12 Uyghur punctuation marks, 3 are missing on Uyghur keyboard. And moreover, we also need to type TATWEEL (or KASHIDA, U+0640, en.wikipedia.org/.../Kashida), but we can not with the current keyboard layout on Windows 7.

Is there anyway that Microsoft will issue a hotfix to fix the problems with the current Uyghur keyboard layout?

Hmmm.

Well first there is Does MS pull new keyboard layouts out of their @!#$%? that answers that first question.

And the comment does count as more information, too.

Kind of.

First the Kashida. The Tatweel.

I talked about that a bit back in You've got to be kashidding me, but really this is not a letter we tend to add to any Arabic script keyboards so far as I know, any more than we add the SOFT HYPHENto keyboards. The "character" is there for compatibility with old standards and fine Arabic typography really calls more for using Arabic justification as supported by libraries like Uniscribe that will fill in its own Kashidas (possibly with hints on where you want them) rather than injecting these "non-letters" into the text stream.

People may wish to use the TATWEEL and if you do use it, then it will work, but really the most common way to see them is in legacy applications that insert them in order to do their own justification system. Not by adding to the keyboard layout....

I do not expect such a fix to really be making it into the keyboard any time soon, really.

So let's focus on the rest of the report -- the "12 Uyghur punctuation marks, three of which are missing", I mean.

I had seen various reports on the web of a few "special Arabic punctuation marks", namely "؛ ؟ ،" (U+061b U+061f U+060c aka ARABIC SEMICOLON, ARABIC QUESTION MARK, and ARABIC COMMA).

But all of those are on the keyboard. So it isn't those three.

Even if it had been, this is not the thing of hotfixes....

There are other punctuation marks on the keyboard, but I don't know which ones the One Uyghur group are referring to as missing....

Though remembering one of the lessons of the original bug in The inappropriate nature of getting the Feh out of Uighur, where we had an actual letter wrong, and did not fix it until the new version of Windows, it seems very unlikely that any form of punctuation would be the subject of any kind of hotfix or really even service pack fix.

Also remembering the most important lesson: don't fix your product without enough substantive information to determine what fix, if any is needed.

I am not on the team that owns keyboard layouts anymore, let alone the person checking in the changs, but to be honest I doubt any change like that would ever be made without a pointer to a national standard that described a widely used layout or documentation on an actual provided hardware layout that was in wide use. Even in a new version.

Just as the burned child fears the fire. And in that sense, the folks who push for a bug to be added ruined the more easygoing verification tendencies in the face of passionate interest.

So without even getting the full information, my answer to the question: "Is there any way that Microsoft will issue a hotfix to fix the problems with the current Uyghur keyboard layout?" is probably not.

Working that way is not in the best interests of Uyghurs.

Or of Microsoft.

Or really of anyone -- the careful and deliberate actions based on substantive problem reports with supporting evidence is the way a product like Windows, the way a company like Microsoft, has to behave....

Yes, that's right, the Oriya Language Interface Pack for Windows 7 is now available!

You can download it for any 32-bit Windows 7 install that contains English resources right here....

The Oriya LIP is produced as a part of the Local Language Program, sponsored by Public Sector.

And for a little background information on Oriya:

NUMBER OF SPEAKERS: 

31 million speakers in India

NAME IN THE LANGUAGE ITSELF:

ଓଡିଆ  

Oriya is one of the official languages of India. About 90 percent of the speakers live in the state Orissa in the East of India, but there are also sizable communities in the neighboring states of West Bengal and Jharkand and, due to labor migration, also in Gujarat.

FUN FACTS:

• Oriya has a rich literary heritage dating back to the thirteenth century and has had a strong tradition of poetry.
• There are many dialects of Oriya, some of which are considered separate languages by native speakers despite the claims of linguists.
• It would seem that many people now refer to Orissa as Odissa or Odisha; given the spelling in the native language (ଓଡିଶା or O DDA I SHA AA) this does not seem entirely unreasonable. Microsoft has not caught up to that just yet.
• Also, it would seem that many people now refer to Oriya as Odiya or Odia; given the spelling in the native language (ଓଡିଆ or O DDA I AA) this does also not seem entirely unreasonable. Microsoft has not caught up to that just yet, either.
• A few bugs were reported for the download page for the LIP within hours of the announcement, which will hopefully be addressed soon. :-)

CLASSIFICATION:

Oriya is a member of the Eastern Indo-Aryan group of Indo-European languages. It is closely related to Bengali, Bihari and Assamese.

SCRIPT:

Oriya has its own script, an abugida with 35 characters. The Oriya script is also used by some smaller languages in Orissa state.

You can click here for more information on the Oriya language.

Enjoy!

If your Latin is a little rusty, the title has a bit from Cicero in it; literally translated it would be something like "He was stirring up billows in a ladle" so think of it as me very pretentiously referring to "a tempest in a teapot"!

Sometimes Microsoft makes mistakes.

I know, major newsflash there. Stop the presses, for sure....

How quickly they are noticed, and how quickly they are fixed? Both points are subject to interpretation and bias. I tend to not be so biased in most cases (being unafraid to call a bone-headed thing bone-headed). But that might be a story for another day....

For now, I'll talk about one of the mistakes.

One that can be be traced back quite a ways though it is not entirely clear how far back.

You may have had your eyes pass over this bug yourself, once or perhaps many times.

If you know what I mean.

Perhaps if I explained what I was talking about it would be easier for you, huh?

I have a feeling this will gio much faster if I just start talking about what I am talking about.

It started with a customer report of a problem on one of web pages on goglobal (this one, the one with code page 1251 on it):

* Name: Юрий
* URL: http://msdn.microsoft.com/ru-ru/goglobal/cc305144(en-us).aspx
* Feedback Details: Здравствуйте! Мне кажется что символы с кодами 01 и 02 перепутаны местами, и символ SOT должен называться SOH (Start Of Heading)

Oh wait,  some of you might feel like your Russian skills are a tad rusty. I know mine are barely enough for this one myself.

Put simply, Yuri (Юрий) was suggesting that the character codes for 0x01 and 0x02 were reversed, and that SOT should be SOH.

In this table:

Every single code page reference on the goglobal site, and the original GlobalDev site tables they were migrated from, has the same layout for these two slots. Including, ironically enough, the ISO code pages.

This is one of those weird cases though. I mean, nothing is reversed in regard to the numbers (0x01 goes with U+0001 and 0x02 goes with U+0002 -- so the data in the table is right).

But if you look at info on these C0 control codes, you'll see that Yuri is right about the symbols the table uses to describe the slots; the symbols should actually look more like they looked in the original Developing International Software for Windows 95 and Windows NT.

Most of the code pages don't have these tables in them, but look in your copy of the book, check out the beginning of code page 932 right where Appendix G starts.

Go on, look now.

I'll wait.

Or, if your copy of the book isn't handy (maybe it is at the office and you are at home, or vice versa), you can check it out online here:

See what I mean?

Oops.

Now unless I'm mistaken, the tables for the book were provided by the same person who provided the tables for the website (most likely one of two people: either the person who I would totally excuse the lapse in, or the other person who I'd shrug and say "that figures"). But either way, how this particular mistake came about is not entirely clear, either way.

The names themselves are also odd in that there are not the names according to Unicode; instead they are defined in places like ISO/IEC 6429:1992, a standard that I actually have a copy of from a few years back and which does quite clearly match the one in the book and not the ones on the globaldev/goglobal sites.

Now this error is not a recent problem -- according to the Internet Archive it has existed at least since 1999 when GlobalDev went live (as the WayBack Machine link proves pretty conclusively!).

And if you consider every code page table that has this wrong and the fact that they have been online for over decade and no one has ever reported it before, quite possibly INCLUDING YOU, which tends to put the  words of the person who forwarded it on to someone who forwarded it on to someone who forwarded it on to someone who forwarded it on to someone who forwarded it on to someone who asked me to comment about it:

Seems like one of our customers found a huge, huge issue with our GoGlobal center. The issue happens across all locales.

in rather sharp relief.

I mean, is it really that huge?

The text tables below have it right and there is no programmatic process that uses those graphical representations in Windows and the names are correct, it is just the two symbols in the table that are wrong.

Now there are some people out there who, when see that Microsoft is wrong about something, they alert Digg or SlashDot or Mary Foley or whoever and it's a big thing. But this small (admittedly x 35) graphical error that has been around over 10 years that no one ever noticed and if they had would never have made that kind of impact.... does not seem like such a big deal.

Maybe it is worth fixing (though I don't happen to think so, and if it were my call I wouldn't bother). Perhaps it is worth a KB article; I mean, if we can put up ones like my old favorite KB172653 (the one which made me want to look into working for Product Support for a bit!) then certainly we could consider writing one up about this web site graphics issue that has made it through over 10 years and almost ten versions of Windows. :-)

It just occurred to me that (in this blog) I assumed that you, as a reader here, knows both Latin and Russian, has a copy of the 1st version Developing International Software, and has seen those code page table at least once and maybe many times just because I have linked to them many times. I hope at least some of it was true, and if not that you were not too distracted thereby!

I may have blown the stack on ridiculous puns with this blog. If your stack was blown then please accept my apologies (and be happy there were no other scripts in the font to continue the punning!).

So the other day, friend and colleague across the vast divide that is Unicode Charles Riley contacted me over IM (I must admit Facebook has been a boon for some communications, the key is to be in front of the machine with the right tab showng in the browser and knowing who to answer when their window pops up!).

Anyhow, Charles pointed out a smll oops in Word, and in particular in its famous Insert Symbol... dialog.

It is something he found in looking at the dialog with his interest in African languages....

In particularly the Ebrima font in Windows 7, and the support of N'ko (U+07c0 - U+07ff):

and Vai (U+a500 - U+a63f):

Note that Tifinagh (U+2d30 - U+2d7f) works:

and Osmanya (U+10480 - U+104af) kind of works:

The font works just fine in all cases, though.

It looks like a simple set of range errors in the code supporting the dialog. Presumably this would be easy to fix, though I couldn't say for sure without looking at the code....

Though unfortunately this bug (which Charles found in Office 2007) also occurs in the recently released Office 2010, as well.

The above screenshots were all from Office 2007 but I verified the same bug is in thew new version as well.

Now Windows and its Character Map are running a slightly different record, with N'ko (U+07c0 - U+07ff) and Tifinagh (U+2d30 - U+2d7f) working:

while Vai (U+a500 - U+a63f) does not:

And of course Osmanya (U+10480 - U+104af) won't work since Character Map can't handle supplementary characters, as I mentioned in I thought that font had a lot of supplementary characters? (aka The highest number of tangents I could fit into a single blog).

That blog, incidentally, explains how no one knows how to update the names on the ranges in Character Map, which suggests that either they figured it out and didn't remember to add Vai, or that the support for N'ko and Tifinagh has been in Character Map since XP even though the fonts haven't been there.

I must admit I am not too curious to know which, since the fact that Tiffinagh was added in Unicode 4.1, N'Ko was added in Unicode 5.0, and Vai was added in Unicode 5.1 suggest either psychic abilities or the incomplete update hypothesis. :-)

One of the last things Charles asked me was what I thought the odds for a service pack fix were (for either bug or both).

The fixes for either one before the next major version seem unlikely since the functionality was in no way impacted for either -- just the cool feature of seeing the name in the user interface....

Now at long last I can point everyone to the Indonesian Language Interface Pack for Windows 7!

It is available for both 32-bit and 64-bit systems, and you can get it right here.

Yet another LIP brought to you by the Local Language Program,. sponsored by public Sector.

Indonesian is a huge market that is growing very fast (as befits the language known by such a uge percentage of all the people of Indonesia, in the fourth largest country in the world!).

A little background information on Indonesian:

NUMBER OF SPEAKERS: 

30  million native, 140 million second language speakers

NAME IN THE LANGUAGE ITSELF:

Bahasa Indonesia

Indonesian is the official language of Indonesia since the country's independence in 1945. It is spoken by about 30 million native speakers, mostly on the island of Java, while another 140 million use it as a second language in the multilingual archipelago. Most formal education in Indonesia is in Indonesia as are nearly all national media, and the government is promoting the language.
Linguistically spoken, it is a variant of a diasystem, representing a standardized dialect of Malay, which has been a trade language in the region for at least a thousand years. While Malay borrowed many words from English in colonial times, Indonesian was influenced by Dutch. The word for exhaust pipe demonstrates this very strikingly: it is knalpot in Indonesian, but ekzos (from exhaust) in Malay. Today, many words from other languages spoken in Indonesia, most prominently from Javanese, are finding their way into the Indonesian language.

As in Bahasa Melayu (Malay), the Bahasa in Bahasa Indonesia simply means language, so shortening the name to Bahasa does not make any sense.

FUN FACTS:

Indonesian shows the influences of many languages in its vocabulary: A lot of different peoples coming into the area brought new concepts and left their traces:

• From Sanskrit, words like agama (religion), bahasa (language) or pustaka (book) were borrowed;
• Arabic left kitab (book) or halal (lawful);
• portuguese keju (cheese, from queijo), mentega (butter, from manteiga), sepatu (shoe, from sapato), or jendela (window, from janela);
• Dutch more than 10,000 words, among them kantor (office, from kantoor), rekening (account, from rekening), kartu (card, from kaart), apotek (pharmacy, from apotheek) or buku (book, from boek).

As the word book, appearing three times in the list above, shows, Indonesian often draws from different foreign sources and the different loanwords can have slightly different meanings.

CLASSIFICATION:

Indonesian belongs to the Western Malayo-Polynesian languages, along with languages like Javanese, Balinese, which are also spoken in Indonesia, Malagasy, spoken in Madagascar, or Tagalog, spoken in the Philippines. The Malayo-Polynesian languages form a subgroup of the Austronesian language family.

SCRIPT:

Indonesian is written in Latin script. In 1972 Indonesia and Malaysia agreed on a unified spelling for Bahasa Indonesia and Bahasa Melayu. The Indonesian spelling until then had still shown major influences of Dutch, like the tj which was used for the sound in the beginning of the English chin, or the dj.

For more information on Indonesian, click here.

Enjoy!

Web developer Matt's question was to my way of thinking a very reasonable one:

Hi all…

Is there any way to automatically change the UI culture for every background thread that .NET creates?

At application startup we force the UI culture on the main thread since our application is localized independently of the OS.  This works fine for exceptions that we throw from the UI thread – we’ve already forced the culture and so we look up the resource strings in the application language, instead of the OS/.NET language.  However, we’ve added a bunch of multithreading this release and noticed that all of the exceptions on the background threads aren’t getting localized since we haven’t forced the culture on those threads.

It would be a LOT of work to go force the thread culture EVERYTIME we create a thread.  We’re also using the TPL extensively so it’s not even clear when threads are being created since we’re not explicitly creating them ourselves – the TPL is!

With the move to more parallel applications, this seems like something that other applications must have run into, so I’m wondering if there’s any way to hook into the framework to override the default culture for all new threads?

One response:

No. There is currently no way to make this happen. Your option will be store the culture in a static variable and use that culture to load resources on the background thread. 

Right – except that once the TPL comes into play I don’t even KNOW when I’m on a background thread or not, or if the culture has already been forced on said thread.  It also requires driving knowledge of the application’s localization strategy into EVERY single piece of code that runs in the application – I now have to pepper calls to force the culture like spaghetti all over the place into every single piece of code that could conceivably run on a background thread, which just seems like a terrible, terrible mess.

With the advent of much more parallel programming, this sounds like it could be a nice scenario for the framework to help with by providing me with some way to tell the framework how to force the culture for all new threads.

This is a question that is asked fairly often.

And although I have mentioned in blogs like New for Windows 7: The PROCESS to keep MUI from being THREADbare.... how .Net really could and in fact ought to be doing something better here, another question comes to mind, one that is perhaps due to my Win32 grounding and the straightforward nature of the web applications and sites I have been tasked to design.

In Win32, the UI is by its nature meant to be very single threaded even in the most complex multi-threaded applications. And any time something must be done in another thread complex interactions have to be done for the application to behave reliably.

Thus as big of a fan as I am of the process default user interface language, I hardly need for every thread in a threadpool to be updated since most of those worker threads are probably doing nothing where the user interface language is even relevant.

Now I may be missing something in the web scenario, or perhaps there is some sort of nut-job object oriented purity objection to an object like thread having to know more about some external setting to do its work, but is its really that common for so many worker threads to be doing things that need the UI language, and that retrieving the value is such a bad idea technically?

I mean, I will be happy when they solve the problem, but my happiness is based on a scenario that seems much more defensible to me.

In Matt's scenario, are the errant threads all popping up their own UI every time? I think I have seen those kinds of web sites, the ones that pop up random alerts one on top of the next, without waiting for me to respond. My rule is to avoid those kind of sites, but maybe that is just me....

In the world of a complex web application that spans multiple servers and has thousands or tens of thousands of clients, each client's settings that are relevant to work done for the client will have to obtain whatever settings they need -- and language is just another of these settings that a lowly threadpool thread will have to obtain to do its work.

Is there some other scenario I am unaware of?

What am I missing here?

Sometimes you can have text in documentation that is entirely true that if you translate it to another language, it's not.

This is usually the essence of the difference between localization and translation; however, at times the difference in what you can or would say in the two language is so utterly different that the localization process may consist of simply deleting paragraphs, with nothing to replace them with other than

English-specific claims were made in this space as if they applied universally, but we know better, don't we?

A fun example of this came up the other day. Well, fun for me, at least.

First I'll inject two terms into the conversation: intuitive and understandable.

Intuitive is the stuff you just get right away, without any real help. You see it or hear it or read it or whatever, and you understand right away what is going on. To use a Star Wars metaphor, Ben "Obi Wan" Kenobi was intuitive when he said stuff like "I felt a great disturbance in the Force, as if millions of voices suddenly cried out in terror and were suddenly silenced. I fear something terrible has happened." it is all pretty intuitive what he is saying. We intuitively understand the words even though we may have neither knowledge nor understanding of "the Force" or any of the stuff behind it.

Understandable is the stuff you have to do a little more work on to get it. it may be easy to do so and it likely gets better with practice. But it cannot be called "intuitive" because there is some degree of a learning curve. To put in a comparable Star Wars metaphor, Yoda was understandable when he said stuff like "Never his mind on where he was" or "Stopped they must be; on this all depends" and "Unexpected this is. And unfortunate.". As many linguists like Geoffrey R. Pullum have pointed out, the way Yoda-speak is applied is unusual (all the moreso because of how it is not used by the jedimaster). Even children can understand him, but everyone has to go through some mental gyrations to do so. It is thus totally understandable, even though it isn't intuitive.

Okay, we have those now?

Good.

Now we will move on to PowerShell.

When pages like Microsoft PowerShell syntax | Grammar rules for Scripts say things like:

The fact that you almost don't need this page is a testament to the intuitive nature of PowerShell.

and I could pull up another dozen examples of descriptions of PowerShell that go on to talk about the intuitive nature of the word ordering and so on.

The assumption here is that it would be intuitive to take a very expressive syntax like the "English" used by PowerShell and have native speakers of another language who, if they know English at all (not all of them will!), they do not know it as a first language.

Obviously that assumption cannot be true across all users and the myriad of languages they speak.

So let's say you were a localizer tasked with the work of taking text like the aforementioned

The fact that you almost don't need this page is a testament to the intuitive nature of PowerShell.

and putting it into another language, one where it may not always be so intuitive?

Languages that are more symbolic and less expressive as if it were language actually have an advantage in some of these cases than the Visual Basics and PowerShells do; it is often only, for example, by thinking of PowerShell symbolically that it becomes easily understandable for these people.

Now perhaps the localizer will simply assume that given the likely scripting/programming background of the reader that they have gotten "over the hump" of the learning curve and thus it is pretty much intuitive to them anyway. Though compared to the expressive nature of the language to people who know English, I think it will always be a stretch to call it intuitive across all target languages.

Now I won't necessarily call this ignorance on the part of the core team doing the writing (since by and large it is a non-English issue), though given the large number of non-English speakers who use the English version, a bit more knowledge of this fact might add a little humility to the somewhat prideful statement about "the intuitive nature of PowerShell" since to many, it won't be at first.

Because in cases like PowerShell, that one particular "intuitive" feature so often cited is not going to be as intuitive or as expressive to everyone. Which makes it best to modify that description to make the language sound more broadly applicable, even to those people who aren't native speakers of English....

Especially since in this case, the fix would have to be in the English version; whatever changes localizers would make would be to work around issues that hadn't yet been fixed. :-)

So everyone knows that you can only [legally] get Multilingual User Interface (MUI) support on Windows in the Enterprise and Ultimate SKUs.

Some people don't care much for this, technically I don't either. But that can be a topic for another day.

And yes they do it differently in Apple. That can be for some other day, too.

The "legally" issue is one I am not going to explore today, either.

But there is a fun scenario that our OEMs will sometimes notice and not be happy about.

Like needing to ship 30 or more different packages so that the per single user interface language situation that may be needed all over the world for all of the other SKUs.

This is just really really ugly.

So they added support for that scenario where an OEM might want to ship multiple languages in one image, and then after the user chooses one language the setup itself can then delete all of the files associated with the various "roads not taken" represented by other languages.

I once mentioned to one of the program managers how wasteful it seemed to me to build a whole huge image of files only to have a huge chunk of them assured of deletion.

She suggested an analogy to the female reproductive cycle which I will not share here despite its admitted applicability; you can probably gather what the analogy is without further hints....

There is one interesting side effect of this design, though.

The one-way nature of the decision.

Once you choose the language you cannot choose again.

There haven't been a ton of people to tun into the problem of the wrong decision in setup leaving one in the wrong state.

The report caused me to smile a bit....

...the number of calls generated by users who are selecting incorrect language during OOBE is not significant but there have been several users who have called in for support on this regards.

Currently, we do not have detailed information on why users have chosen the wrong language.

Hmmmm.

It is likely fair to expect that confusion in the use interface and the consequences of the choices made in it are the most likely contributing factors, under the "when you hear hoof beats expect horses not zebras" doctrine, right? :-)

This is not something I have talked about very much.

I mean, I mentioned it a little bit in UCS-2 to UTF-16, Part 10: Variation[ Selector] on a theme... a while back.

And if you look far and wide, you may find additional odd mentions from time to time.

Variation Selectors.

Those funky characters that are meant to be invisible but if they happen to be there and are on a magical list, combine with the previous character to change the appearance of it to make it look different in some way.

Now to whatever extent I jumped the shark on Unicode it wasn't UTF-8S (now called CESU-8) or variation selectors, it was emoji.

But it was very nearly variation selectors....

And not really variation selectors per se; it was much more

Ideographic Variation Sequences.

which, with UTS#37 (Unicode Ideographic Variation Database) define a method by which Unicode, the standard that encodes scripts not languages which therefore cannot encode any random variation of a character, defines a means by which anyone who is dealing with CJKV who has some money and wants to encode their specific graphical forms of already encoded ideographs can basically do so, with minimal oversight compared to the process by which new characters and scripts are usually encoded.

If you think about this and decide it sounds like a train wreck waiting to happen, then you know how I feel....

Now the recent Unicode Technical Committee meeting just happened included an interesting point:

• UTC has decided to revise the procedures related to IVS registration
• When two bodies want to register variation sequences for the same CJK ideograph glyphs, UTC would like to have a way that the two bodies can collaborate so that there is only one registered sequence for any given glyph, thereby fostering interoperability.
• This is relevant to current IVS work and to a registration for IVSs that is currently in progress submitted by the Japanese gov’t (their set has significant overlay with IVSs already registered by Adobe)

What was that thing King Canute was saying about laws to sweep back the tides?

I can't remember, exactly.

But I look forward to the first time something is rejected that is part of a set that one side feels is not an overlap, is not the same ideograph. And the fact that this mechanism meant to minimize the need for oversight now requires a whole lot of people to be watching things to avoid the duplication that this whole approach seems architected to allow.

Now when I say I look forward to this, I was being sarcastic. It actually makes me somewhat happy I don't have to be in those meetings anymore, even though usually I tend to miss them a bit....

So, the message from Dharma via the Contact Link was:

Dear Michael,

First, I would like to thank you for your great BLOG!  Well done!  I've spent many hours reading and studying various articles and have found all of them helpful AND entertaining as well.  And it's hard to do both of those things at once!

Although I've worked in the Java world for years, I'm new to Windows programming.  So please forgive my ignorance!

1.  The app that I'm building involves using 2 new keyboard layouts.  So I built them using MSKLC.  Everything was fine except they did not quite work in Microsoft Word 2007 running on XP.  Most of the keyboard worked just fine.  In particular the Devanagari layout didn't quite work in Word although it works fine in Notepad, Wordpad, and Excel 2007!  It turns out that when I use a dead key to obtain some of the cerebral consonants in a glyph they don't combine!  And it only happens when the dead key letter is in second position!

Let me explain more clearly with an example.

Here is what is supposed to happen (see the Devanagari Unicode table):

Let + be short for the words "followed by" in the example below

0938 + 094d + 0925

This sort of thing works just fine in MOST cases.  The 2 letters are combined into a glyph & are properly rendered on-screen

However, if the last code point in the example is created by using a dead key (dead key + someKeystroke), the 2 letters do NOT combine into a new glyph as they do all the other times.  If you then move the cursor backwards on the screen to select the letters of the word, then the glyph fixes itself and properly renders!

So I've found a way around this by simply not using dead keys but it makes for a less convenient keyboard mapping.

What's happening here?  I saw an article where one of your readers said something about Rich Edit causing something like this.  It was a small and somewhat vague remark.  Could this be the case here?  I don't mean for you to spend a lot of time on this.  If you could just point me in the right direction to understand what is happening, I'd be grateful.

Thanks again Michael.  Tell your boss that I said you're doing a great job!

Dharma

It is funny, a colleague of mine was talking about this "feature" recently.

He was talking about RichEdit though he explained how the same feature existed in Word.

The feature?

Well, providing rendering smarts by using information about the keyboard and the input stream.

This may be what is interferring here.

A good way to check is to take the text that looks good in the other apps, copy it, then paste it into Word (or RichEdit).

If it looks right in both then it is the keyboard "smarts", if not then read on....

Now that I think about it, I am goint to a priori blame Indic sequence checking in this case (ref: You're not the one out of sequence, and that's the Word), since that is one that you can verify being the problem by turning off the feature and seeing what happens!

The problem with the sequence checking is its assumptions about the way proper input looks can be entirely unforgiving in such cases if it does not recognize the nature of the input (like this keyboard may be causing for it).

The best fix there is to turn it off, always, as it breaks proper use of language in so many cases with its provincial and narrow view of what constitutes normal input....

Please forgive the Marty McFly allusion in the title!

When I wrote Capitalized tag != capitalized month (not for nothing, but it was the least-read blog of mine in recent memory!), I only had half the question there.

It was a side question the support person asked as they were looking at the actual question.

The actual question was about the fact that between XP and Windows 7, there was a change.

You can see it right here:

The month name used to be capitalized.

And now in Windows 7 it isn't, apparently:

This underscores a particular reality about our locale data: the fact that it is constantly changing.

Changing due to new preferences.

Changing due to new standards.

Sometimes even changing due to bugs.

In this case, it looks like a change due to standards, as one of the developers noticed:

Having the month in lower case is the right thing to do for Spanish.

Your customer can refer to the following official document, published by the RAE (Royal Academy of the Spanish Language):

     http://www.rae.es/rae/gestores/gespub000015.nsf/(voanexos)/arch7E8694F9D6446133C12571640039A189/$FILE/Ortografia.pdf , especially section 3.4.

Also, the AML (Mexican Academy of Language) has the same document: 

     http://www.academia.org.mx/pdfs/ortografia.zip

This kind of thing happens often, and of course if the information is received in reasonable time to take the update then the next version will see that update happening.

Of course if the source needs to be verified that can take more time but the advantage of it being based on a standard is that the source is usually quite quick/easy to verify.

Microsoft gets a non-trivial amount of direct feedback suggesting direct changes being needed in a market without any validation that turn out to be incorrect for the majority of people using the locale. Not for mean-spirited reasons, just for the simple fact that not everyone knows what the "official" way is just because it is the way that seems most natural to them.

But that occasionally leads to bugs, like in this case, for example - a bit of well-intended feedback from one group of native speakers of a language leading to a major bug in the opinion of another group of native speakers....

The big question that comes up is what to do about the down-level versions of Windows, all of which contain the "best known data" from when they were shipped but after that could be "wrong" for any of the three reasons given above or any combination thereof.

The policy, almost without exception, is to change nothing.

Upgrade is the way to get the latest.

In fact, the only time you see exceptions would be very extreme cases, such as new countries moving to use the Euro, and even in that case they are "fixed" by using the built-in "current locale modification" rather than by updating the underlying binary.

Of course within a country it is unlikely to every single person following such standards so readily that a change leads to instant 100% adoption, so there are always going to be people who will consider such changes to be bugs, by default.

Given the potential impact on parsing and formatting, it is quite possible it can look that way in some cases.

All of which is a good reason to not be so completely proactive that the second a standard comes out Windows gets it -- since it can be years or even never before a standard is widely adopted in some cases....

So, where do I start?

Oh yes, I remember.

The .Net Framework's first and in the minds of some still respectable to be using right now UI technology, WinForms.

Windows Forms.

There is a particular feature they have, an AutoSize feature. When used (and used properly) it tremendously lightens the burden on localization since it will resize forms and controls to fit the text they contain.

Note the  "used properly" notation there. This is important as I know of a project or two (or ten, or twenty!) that are part of a tiny little product you have never heard of called Windows that didn't really tend to do it right.

Which is not the end of the world. Not for you, or for me, or for the developers who created the forms, or for the localizers who had to do some resizing when text would truncate.

Everyone continued to function normally.

I guess the localizers had to work a little harder.

So no big deal right?

Right.

Oh wait, I forgot something.

Something about this AutoSize feature.

Now obviously a developer who was trying to use this feature might want to exert a little control over what it does, right?

I mean, letting people do stuff with no safety controls over how much they do leads to derivative market inspired economic collapse or unprecedented oil spills.

We know we don't want our WinForms to be all bankrupt and oily. So some ability to rein in the way that things might want to grow can make a little sense.

Of course the complexity of those ways to control the unprecedented growth is one of the big reasons why the AutoSize feature was so poorly utilized.

Why Windows became the son and heir to more than just a shyness that is criminally vulgar; it also inherited a whole bunch of oily, bankrupt WinForms applications.

The Smiths would be so disappointed.

But that is not the principal issue I wanted to discuss.

I really wanted to talk about the localization process, and then a couple of the properties that the original developers have to help rein in AutoSize if they believe they have a need to, and how all of that fits together.

The process, in a nutshell:

• Developer makes the form "Localizable" by setting the property of that name;
• Visual Studio creates a bunch of language-specific ResX files with all of the form/control size and other data;
• Localizers can resize the dialogs and controls as they put in the translated text and see what it will look like using WinRes;
• Windows ships those localized components with their localized product.

Easy, right?

Well, there was a problem or two.

Or more.

But for the moment I'll just talk about one or two of them.

Now one way that the developer can exert some control over the AutoSize beast is by use of the Control.MaximumSize and Control.MinimumSize properties.

These properties can be used to keep the difference between the original Control.Size and the eventual size from going way beyond what the developer ever really wanted.

That really does make sense (well, the Control.MaximumSize more than the Control.MinimumSize, but they both make at least some sense). And I would never want to imply otherwise.

Everything still looks okay, right?

Well, let me include the spanner in the works.

Those two properties? Control.MaximumSize and Control.MinimumSize?

They are not included in the localizable .ResX.

This leads to three specific problems:

• The dialog that the localizer sees in WinRes may or may not match the actual dialog's layout, since there is another couple of properties affecting the Control.Size that WinRes knows nothing about. Before the localizer even touches the dialog or its controls, they may be looking at an inaccurate representation of the layout.

• Any time the localizer resizes a dialog or a control (a common occurrence due to the difficulty in implementing the AutoSize feature properly and the known fact that many target languages are, on the average, 30-40% bigger in terms of the amount of text), the localizer's resizing may be partially or fully ignored due to the Control.MaximumSize and/or Control.MinimumSize in the main form modifying the sizes at runtime, something that the localizer (using WinRes) never sees. Note that WinRes, also from .Net, does recognize the properties, so someone was thinking this scenario oughtta work!

Now the combination of these first two factors often leads (as you can perhaps imagine) to the kind of bug that goes back and forth several times between the localizer fixing the truncation bug and the tester reporting it with the kind of "the problem is fixed"/"no it isn't" kind of argument that so often can come from looking at two different things and thinking they are the same.  

• Even once the localizer finally gets wise to the underlying problems, the fact that the Control.MaximumSize and Control.MinimumSize properties are not exposed to them means that they cannot fix the bug except by perhaps unreasonably/unreadably truncating the text themselves to avoid the unreasonable/unreadable truncation that the code will do itself later.

Or make it a core bug -- perhaps only found after everything has been frozen and no one is willing to make changes.

Okay.

Fair enough.

Now with all that said, in the world of Visual Studio, this bug is not such a big deal.

I mean, from their point of view it is really quite unreasonable to even set Control.MaximumSize or Control.MinimumSize to a size that could break localization; in fact, one of the more senior people I talked to about this bug couldn't even see why anyone would set either property in a form meant to be localizable.

Of course I don't know of any other reason why one would set it, which means that someone was admitting to me that the property was a bad idea! But that can be a subject for another day... :-)

Now the WinForms people are not being arbitrary about this judgment; the bug, which has existed since the very initial 1.0 version of the .Net Framework and WinForms, has (I am told) never ever been reported by any other customer external or internal to Microsoft.

Just Windows.

Now to Windows it led to hundreds of bugs (and no, this is not an exaggeration; the actual number is even higher though the "even higher" part is due in part to other problems that I'll talk about in some future blog).

Pain in the butt bugs with all those back and forth problems and difficulty pinning the problems down. And so forth.

Windows actually shipped with many of these bugs only partially fixed or worked around.

And some of the worst of the offenders in the dialogs are known by name to people on the localization team, since they know how many bugs they can expect to have to deal with any time one of them is changed.

But reportedly no one else has ever complained about the problem. Ever.

So, okay.

Eventually there will be a fix for this problem, a fix that I guess no one outside of the internal Windows team even cares about, since no one else apparently ran into the problem.

And there will be a KB article describing it (I'll link to it once I find it exists). In case it turns out anyone wants it.

From a social engineering standpoint I wish I had the time, resources, and knowledge to study the factors that can cause a bug to so hugely impact multiple developers spread across teams and groups and divisions and even continents within one large product while never affecting any other project in the entire world.

I can't even try and fathom that one; in fact I get a headache just thinking about it....