Postings are provided as is with no warranties, and confer no rights. Opinions expressed here are my own delusions; my employers at best shake their heads and sigh, at worst repudiate the content with extreme prejudice, whenever it manages to appear on their radar.
This blog is unsuitable for overly sensitive persons with low self-esteem and/or no sense of humour. Proceed at your own risk. Use as directed. Do not spray directly into eyes. Caution: filling may be hot. Do not give to children under 60 years of age. Not labeled for individual sale. Do not read 'natas teews ym' backwards. Objects in mirror are closer than they appear. Chew before swallowing. Do not bend, fold, spindle or mutilate. Do not take orally unless directed by a physician. Remove baby before folding stroller. Not for use on unexplained calf pain.
A nice FLAIR (FLuid Attenuated Inversion Recovery) view from the not-too-distant past. Every abnormality you can see on this scan (and there is more than one!) is asymptomatic at present. Alongside is a picture of me walking the walls at Fremont Studios, a sign of a damaged brain.
This blog reminds me of "Fall over" is not the same thing as "fall back" for [hopefully] somewhat obvious reasons....
It is the most simple and innocent of questions that can tend to add to the most convoluted of answers.
Like yesterday.
The question was:
Hello,
I’m wondering where could I get a list of languages with info regarding the preferred fallback language for each of them.
Thanks for your help.
This is just such a question.
Unfortunately, there are two different answers.
I mean, at least two answers.
Even after you ignore the kinds of questions that issues like Raymond's The many ways of converting a string from one language to another bring up.
So in one case, you have the fallback described by the Win32 GetLocaleInfoEx/LOCALE_SPARENT and managed CultureInfo.Parent chain.
This one will, for example, tell you that uk-UA (Ukrainian - Ukraine) falls back to uk (Ukrainian - Neutral), which falls back to Invariant.
And in the other case, you have the fallback officially determined for language resources.
This one will, for example, tell you that uk-UA (Ukrainian - Ukrainian) falls back to ru-RU (Russian - Russia), which falls back to English.
Um.
Which one is right?
In managed code, the first one is right -- it is how the resource loader works. Though in .Net 4.0 it co-opts some of the native resource fallback list.
In native (Win32) code, it is a complicated system that builds a UI language list built up from principally the second one, though essentially including the first one as well in most cases.
If (given the fact that Windows uses both native and managed code) this sounds vaguely insane to you, then you should vaguely congratulate yourself for being vaguely on track.
Because all of this way to freaking complicated for words, for no good reason.
Especially since the LOCALE_SPARENT/CultureInfo.Parent scheme should be able to return the "combined" list of (for example):
uk-UA --> uk --> ru-RU --> ru --> en-US --> en --> Invariant
So that the rules of both lists can be accommodated, in ways that users can appreciate the results of and developers query and use as they need (from either native or managed code).
For now we just live with the very different scenarios leading to very different results....
Prior blogs in this series:
I suspect some of my readers are either fans or at least regular watchers of The Colbert Report.
Perhaps just my smarter readers.
Or maybe just the ones with basic cable....
Today's blog ends up being about a combination "tip of the hat/wag of the finger" question.
In honor of Stephen Colbert....
The question goes something like this:
SUBJECT: String.Compare for double byte characters in .Net
I have following two string characters whose comparisons in SQL are equal, however I couldn’t figure out any comparisons in .net (culture/ordinal/case insensitive) that would return me equality. Any ideas?
Goal is to not change SQL settings, but to find insensitive compare in .net.
String.Compare( "0336753496aaa@ae2.dion.ne.jp", "0336753496ａａａ@ae2.dion.ne.jp", CultureInfo.InvariantCulture, CompareOptions.IgnoreCase)
OR (Tried all combinations)
String.Compare( "0336753496aaa@ae2.dion.ne.jp", "0336753496ａａａ@ae2.dion.ne.jp", StringComparison.OrdinalIgnoreCase)
Now I'll consolidate the many different tips and wags:
First of all, a wag of the finger since the question referred to "double byte characters" despite every string involved using Unicode, in a language (C#) that uses Unicode.
Perhaps somewhat forgivable since the example was clearly referencing Japan, so perhaps the questioner was thinking about Japanese at the time. And therefore "double byte" was just old school thinking about CJK. Kind of like how they never migrated all those people off the FAREAST domain, even as everything else started referencing east Asia. Even though domain account migrations are so much easier these days after those thousands of migrations in Windows kind of forced ITG to get better at it....
Second of all, a tip of the hat to the genuine attempt to try to do comparisons that fold out distinctions in an attempt to get parity between SQL Server and the .NET Framework.
Third of all, a wag of the finger for ignoring the most important distinction in this case -- the implicit Width Insensitive nature of all _C*_A* collations in SQL Server, which could have been simulated by adding a StringComparison.IgnoreWidth to the first call, had their names not masked the fundamenta nature of the "hidden width" that makes me wonder if someone in SQL Server isn't worried about their weight too much....
Fourth of all, a wag of the finger for taking a question obviously covering E-mail Address Internationalization (EAI) but doing it without even asking the question in a way or to a distribution list that suggested they were thinking about EAI.
With a bonus fifth of all wag of the finger to SQL Server since it is hiding so much of the problem here that people come out of SQL Server wondering how to make other products act like them, rather than coming out asking the real questions....
Okay, seems like a lot more wags than tips on this one. And that's even ignoring the extra wags i decided to leave for another day.
I've decided I can't do "tip of the hat/wag of the finger" very well. I should leave that sort of thing to the professionals. From now on, I will.
I'll talk more about EAI another day, too....
This blog you are reading was originally a very different one, based on people with questions about several forum posts like this one, this one, this one, and this other one. It was gutted and re-written after large thread that took place yesterday endd up seeming a lot more on topic....
The question was clearly written, so I will give credit for that, at least:
Hi,
We have a UI page that takes a password input. To increase the entropy we would like to ensure that the password as characteristics such as
a) Great than x characters (current 8)b) Has at least one upper casec) Has at least one number
We can use the GetStringType* APIs but I am not sure if for languages such as Japanese etc. if the API will help us do what we would like to, for example not sure if all languages have upper cases, numbers etc.
What will you suggest to enforce a high entropy password that will work globally?
<Please redirect as appropriate>
Thanks!
My first thought was to when a similar question came up a little over three years ago (as previously described in You want to know what's weak? Strong password rules, that's what's weak!
But pretty self-aware, if you catch my drift....
It is quite easy to look at such "criteria" as being rather biased toward languages with case in them, and the question about Japanese makes things all the more complicated, truth be told. Asone colleague responded in part:
As you look at passwords in a global perspective, it seems like there are several things that could be considered besides case. For instance, suppose you expect passwords to be comprised of characters that are somehow related—e.g. they can all be generated by the same keyboard layout (seems like a good assumption), then it seems pretty significant that an Amharic input method can generate one order of magnitude more possible values for each information element than can an English input method, and a CJK input method can generate over two orders of magnitude more possible values. In other words, even though those writing systems do not have case, the set of likely keys will be 10^2^n and 10^3^n (for password length n) greater than for passwords generated using an English input method. In other words, while you don’t have case adding to the entropy, you have different factors that might potentially add much more robustness than case can provide.
Of course, there are other factors for something like CJK: since each information element can be interpreted as a word, it seems like there’d be a certain user propensity to create passwords that are mnemonic phrases—and that will reduce the entropy.
Okay, this miht be overboard, though.
Becauseone of the most interesting facts about password fields (once you ignore the fact that no one should ever be asking for passwords except through CredUI these days!) is very relevant to Amharic. And Japanese. and CJK.
Password fields turn off the IME.
I feel I should repeat that a little louder, as it seems vaguely important!
Of course this specific fact that undercuts almost every single example given doesn't attack the basic argument's soundness.
By opening up the repertoire and suggesting more reasonable additional potential ways to add the required level of complexity, one can remove the more provincial aspects of this whole policy that to me represents the most famous of all the dumb interview questions: writing a RegEx expression to validate the password string....
For example -- one can suggest muliple scripts, like with Japanese Katakana and Hiragana.
Or one can suggest both full width and half with letters, like with Korean Jamo.
Note that when they use the English keyboard for other languages (like English to get the Pinyin for Chinese), case is often not applicable there either. So requiring case often makes minimal sense, culturally.
And before you suggest that I have knocked Japanese and Korean and Chinese out of the running, I'll point out one of the most important misunderstood features for Japanese and Korean (and even Chinese, though Chinese does it differently) -- the additional non-IME keyboard that is kind of "installed" and gets used when the IME itself is disabled.
You can't put random Kanji/Hanja/Han in there, but you can certainly get the underlying Kana/Jamo/English that you would be typing.
It gets even more interesting as you consider passwords in Indic languages/scripts and typing in strings of characters that can be a lot closer to meaningless given how some strings simply will never look right if you were to look at then rendered.
Though as luck would have it, ●●●●●●●●●●●●●● isn't complex -- no matter what the underlying language is.
And I'll add one basic clarifying assumption: people will generally type a password using a single keyboard -- so you have to decide the meaning of complex as it applies to a given keyboard.
Or you can keep the current lame rules that tend to assume it's English or English-like.
In the long run, I think we need to mix it up a bit, we need to add some additional entries to the recommended Password Complexity Requirements or even better the more recently updated Passwords must meet complexity requirements:
Passwords must contain characters from three of the following five categories:
Now look, the article makes it clear that they are using GetStringTypeW to make their decisions, which means that a whole lot of Unicode is available here -- and the current restrictions seem tailored to support some Japanese and Korean just as well as "Western" cased scripts. So we are already part way there....
So now let's just take it the rest of the way and open up this list to even more possibilities!
Regular readers may recall my conspiracy theory series and some of its early parts.
Like Anti-Microsoft conspiracy theories are fun #3 (aka Why the hell can't they just update Uniscribe?).
Or Anti-Microsoft conspiracy theories are fun #5 (aka Microsoft is not supporting the terrorists, dammit!).
You may have seen some of the other parts, it depends on your clearance level....
:-)
Anyway, I wanted to point out recent news stories like Yemen passport holder sparks US flight security scare.
Which make it clear that Microsoft's efforts to foil the terrorists continue apace!
Okay, hold on. That's all way too silly.
Never mind.
Maybe we should talk about the notion of anti-Microsoft conspiracy theories for a bit.
Do people really buy these anymore?
I don't mean the "tin foil hats to block the satellites" crowd, I mean the regular people.
If one argues that it was misbehavior and/or the appearance of misbehavior that led to the whole "Microsoft is Evil" thing that drives the less crazy conspiracy theories, it seems like the conceptual torch has largely been passed.
Mostly it seems folks have moved on to the "crazy" stuff Apple and Google and Facebook are doing. You know, as they find their own ethical compasses to determine the right balance between making customers happy (by giving them compelling features) and freaking customers out (by breaking laws and/or trust).
I mean, we all know that each of these companies wield a huge amount of influence, and we know that each has had missteps. I've witnessed enough from the inside of such situaions that I'm willing to give each company some leeway when it comes to ascribing evil motives -- I mean, most of the time even the people with vision don't see the full consequences. We usually just read about it after someone steps in it, on some gaffe or another.
I really don't want to write about anti-Apple or anti-Google or anti-Facebook conspiracies though. There's no point (you can get plenty of that elsewhere).
So probably I'll just stick with the Microsoft stuff.
Though conspirascy theories seem less and less interesting....
It dates back to some time between when Windows XP shipped and Windows Server 2003 shipped.
Suddenly, in a wee bit longer than a blink of an eye, it was determined that Microsoft's essentially promiscuous use of both the Unicode PUA (private use area) and unpaired Unicode surrogates were not only bad and/or wrong -- they were in some situations potential security issues.
This was kind of the point of Keeping out the undesirables? and similar blogs.
Now if you ask me, we went a little overboard on this one, which is the theme of The torrents of U+fffd (aka When security and conformance trump compatibility and reality) and other U+fffd blogs in this Blog.But that's neither here nor there, today.
I just wanted to point out that now, all these years later, the work is done. We have bombed the PUA village and we've strafed the unpaired surrogate survivors.
Even when (as pointed out in Short-sighted text processing #5: PU[A]! That pad THAI is pretty spicy....) we find we have told tell backward compatibility to Get Bent.
The PUA is gone from our hearts and our minds and our memories -- never to darken our doorstep again.
Um, except the place we forgot.
The Microsoft implementation of The Thai Pattachote keyboard layout.
You can see it right here. Or otherwise here it is, hidden in plain sight right atop the Enter key for over a decade:
This is just Microsoft, mind you.
The various other places where you can see the Thai Pattachote keyboard layout, like here and pretty much most other places, if you ignore a few places like here, which seem to be following us to insert a U+f8c7 on that key.
Note that is not one of the PUA characters on the list that once gave us Thai as we wanted it that I mentioned in Short-sighted text processing #5: PU[A]! That pad THAI is pretty spicy.....This is just a random piece of the PUA whose continued presence is mostly due to this error of ours dating back to at least Windows NT 4.0, if not earlier.
We even have it mapped in Code Page 874, albeit underhandedly.
Oops, we have it there in the code page, too!
Now we have an interesting decision to make, one that will test the universal prohibition that has existed about never removing anything from a keyboard layout once it has shipped.
Do we remove U+f8c7 from the Thai Pattachote keyboard layout in future versions?
In this true case of an irresistible force versus an immovable object, it will be interesting to see what we do....
The advantage here is that no matter what, we are being consistent with previously established behavior, but the disadvantages is that no matter what, we are betraying as previously established pattern of behavior....
I am so predictable, I really am.
Like when I saw Google Translate welcomes you to the Indic web on the Google Blog, wasn't the ideathat I would respond by blogging about it kind of a foregone conclusion?
I decide to start by taking பிறப்பொக்கும் எல்லா உயிர்க்கும் (Pirapokkum ella uyirkum), the song I talk about in The song starts something like "All are equal at birth, they should all live as one race...", and translate it.
Of course the site is an Alpha version, as their link says:
You can expect translations for these new alpha languages to be less fluent and include many more untranslated words than some of our more mature languages—like Spanish or Chinese—which have much more of the web content that powers our statistical machine translation approach. Despite these challenges, we release alpha languages when we believe that they help people better access the multilingual web. If you notice incorrect or missing translations for any of our languages, please correct us; we enjoy learning from our mistakes and your feedback helps us graduate new languages from alpha status. If you’re a translator, you’ll also be able to take advantage of our machine translated output when using the Google Translator Toolkit.
Remembering that, the goal is not poetry -- it is understandability....
It starts with something like "All living humans are one in circumstances of birth" and goes from there, right?
And this site would be the way to take communication in Tamil and make it understandable to those who do not know Tamil.
Here is the orginal Tamil and the translation as of a couple of hours ago, for your perusal:
பிறப்பொக்கும் எல்லா உயிர்க்கும் - பிறந்த பின்னர், யாதும் ஊரே யாவரும் கேளீர்
உண்பது நாழி உடுப்பது இரண்டே உறைவிடம் என்பது ஒன்றே என உரைத்து வாழ்ந்தோம் – உழைத்து வாழ்வோம்
தீதும் நன்றும் பிறர்தர வாரா எனும் நன்மொழியே நம் பொன் மொழியாம் போரைப் புறம் தள்ளி பொருளைப் பொதுவாக்கவே அமைதி வழி காட்டும் அன்பு மொழி அய்யன் வள்ளுவரின் வாய்மொழியாம்
செம்மொழியான தமிழ் மொழியாம்…
ஓரறிவு முதல் ஆறறிவு உயிரினம் வரையிலே உணர்ந்திடும் உடலமைப்பை பகுத்துக் கூறும் ஒல்காப் புகழ் தொல்காப்பியமும் ஒப்பற்ற குறள் கூறும் உயர் பண்பாடு ஒலிக்கின்ற சிலம்பும், மேகலையும் சிந்தாமணியுடனே வளையாபதி குண்டலகேசியும்
கம்ப நாட்டாழ்வாரும் கவியரசி அவ்வை நல்லாளும் எம்மதமும் ஏற்றுப் புகழ்கின்ற எத்தனையோ ஆயிரம் கவிதை நெய்வோரும் புத்தாடை அனைத்துக்கும் வித்தாக விளங்கும் மொழி
அகமென்றும் புறமென்றும் வாழ்வை அழகாக வகுத்தளித்து ஆதி அந்தமிலாது இருக்கின்ற இனிய மொழி - ஓதி வளரும் உயிரான உலகமொழி நம் மொழி – நம் மொழி – அதுவே
தமிழ்மொழி – தமிழ்மொழி – தமிழ்மொழியாம் தமிழ் மொழியாம் – எங்கள் தமிழ் மொழியாம் வாழிய வாழியவே… தமிழ் வாழிய வாழியவே… செம்மொழியான தமிழ் மொழியாம்…
Pirapokkum ella uyirkumpirandha pinnaryaddhum ooreee yaavarum kellirrronnbadhu naazhi udupathu irendeuraividam enbadhu ondreuraithu vazhndhom uzhaithu vazhvom..theedhum nandrum pirar thara vaarai yenumnan mozhiye nam pon mozhiyaamporrai puram thallzhi porulai podhuvaakave..amaidhi vazhi kaatum anbu mozhiayyan valluvarin vaaimozhiyaamSEMMOZHIYAANA TAMIZH MOZHIYAAAMorrarivu mudhal aararivu uyirinam varayileunarndhidum udal amaipai pagirthu koorumorrarivu mudhal aararivu uyirinam varayileunarndhidum udal amaipai pagirthu koorumthozhgapugal thozhgapiyamumoppatra kural koorum uyar panpaaduolikindra silamubum meghalayum sindhamaniyudane..vazhayapathi kundalakesiyumm..SEMMOZHIYAANA TAMIZH MOZHIYAAAMkamba naataalvarum kavi arasiyevai nallaalumyemmadhamum yetrum puzhgal endrumyethanayo aayiram kavidhai neivor tharumthadai anaithukkum vithaaga vilangum mozhiSEMMOZHIYAANA TAMIZH MOZHIYAAAMaagam endrum puram endrumvazhvai azhagaaga vaguthalithuaadhi anddam illathu irukindra iniya mozhimodhi vazharum uyiraana uzhaga mozhithamm mozhi namm mozhiadhuve...SEMMOZHIYAANA TAMIZH MOZHIYAAAMtamizh mozhi tamizh mozhi tamizh mozhi ya ghaaa...SEMMOZHIYAANA TAMIZH MOZHIYAAAMThamizh Mozhiyam Engal Thamil MozhiyamVazhiya Vazhiya ve..Thamizh .. Vazhiya Vazhiya veSEMMOZHIYAANA TAMIZH MOZHIYAAAM
Pirappo of all life - After birth, No Information
Now now only eating utuppatu That is the only shelter Uraittu lived - working live
Pirartara titum bad as well Our golden nanmoliye moliyam Away at war Potuvakkave item Quiet way Love language Ayaan's vaymoliyam valluvar
Moliyam cemmoliyana Tamil ...
The first creature orarivu ararivu varaiyile Shape analysis and confirmed unarntitum Tolkappiyamum olka fame Rate this unique element of high culture Olikkinra cilampum, mekalaiyum Projects cintamaniyutane kuntalakeciyum
Thanmalwila nattalvarum Well kaviyaraci Avvai Load praising emmatamum Neyvorum many thousand poetry Refresh all Vitt, as a language
Puramenrum life akamenrum Beautiful vakuttalittu My language is antamilatu Adi - Oti developing codenamed ulakamoli Our language - our language - it
Tamilmoli - tamilmoli - tamilmoliyam Moliyam Tamil - Tamil moliyam us Tamil Site Support Site Support ... ... Moliyam cemmoliyana Tamil ...
Perhaps we have not hit completely understandability just yet.
Well, let's just call it a work in progress. We'll try back in a month or three and see if there are any improvements.
In the meantime, I can still point out that I am completely in love with Shruti Hassan's voice, for those following along with that kind of thing.... :-)
Shortly after Raymond's How do I convert an ANSI string directly to UTF-8?, someone with the handle (I'm assuming it is not his real name) of Otis asked me to weigh in on the issue.
I chose to hold back, it seemed to me that the blog and comments to it were proceeding appropriately.
But Otis would periodiclly ping me on it, thinking there was perhaps more to say.
I did respond to the other question Otis asked me, about whether I was jealous of the fact that more people commented on his blogs than mine -- I'm not. and you only have to look at many of the comments over there to see why.I'd probably stop reading the comments or turn them off i I had to deal with all that!
But the big question:
Is there a way to convert an ANSI string directly to UTF-8 string? I have an ANSI string which was converted from Unicode based of the current code page. I need to convert this string to UTF-8. Currently I am converting the string from ANSI to Unicode (MultiByteToWideChar(CP_ACP)) and then converting the Unicode to UTF-8 (WideCharToMultibyte(CP_UTF8)). Is there a way to do the conversion without the redundant conversion back to Unicode?
Is there a way to convert an ANSI string directly to UTF-8 string? I have an ANSI string which was converted from Unicode based of the current code page. I need to convert this string to UTF-8.
Currently I am converting the string from ANSI to Unicode (MultiByteToWideChar(CP_ACP)) and then converting the Unicode to UTF-8 (WideCharToMultibyte(CP_UTF8)). Is there a way to do the conversion without the redundant conversion back to Unicode?
MultiByteToWideChar(CP_ACP)
WideCharToMultibyte(CP_UTF8)
has already been answered as well as I would have answered it -- there isn't one. You should use Unicode as your pivot encoding between the ACP and UTF-8.
And bemoan the fact that the ACP can't be UTF-8, since that would make this question much easier to answer.
I could post many links to blogs that continually tell the story about how that isn't gonna happen but I'll just be lazy and point to one of them; UTF-8 and GB18030 are both 'NT' code pages, they just aren't 'ANSI' code pages. It points to some of the others.
It still ain't gonna happen.
If you are using the ACP anywhere, then you are lossy -- you should stop doing that. Keep around the old interface if you must but tell people to not use it.
I'll have to talk more about the UELNT project (as I called it in that post) aka the MSL8 project (as I call it here), one of these days.
It is common knowledge to those not guilty of my dear boy type offenses that native, Win32 NLS pre-dates the managed System.Globalization classes by several years.
And it is perhaps not as completely well known but still fairly common knowledge that the principal developer of the former was in fact the initial architect of the latter.
It is also pretty common knowledge that the underlying data of one has always a minimum been an extension of the other, eventually leading to a common data store and format, and a not insignificant amount of code sharing.
But even knowing all that, it is easy to forget some basic compatibility issues that exist between these two fraternal twins.
Like just yesterday when a tester asked:
A component writes a date string using GetDateFormatEx() API with DATE_AUTOLAYOUT:
GetDateFormatEx( LOCALE_NAME_INVARIANT, DATE_AUTOLAYOUT | DATE_SHORTDATE, SysTime, NULL, dateStr, MAX_PATH, NULL))
This produces a string containing 'LEFT-TO-RIGHT MARK' (U+200E) characters:
DateTime.Parse() for this string is failing with “String was not recognized as a valid DateTime” because of this extra chars.
What is the correct way to parse the date string (in C#)??
Now GetDateFormat[Ex] has had this support in some form for quite a while:
Windows 7 and later: Detect the need for right-to-left and left-to-right reading layout using the locale and calendar information, and add marks accordingly. This value cannot be used with DATE_LTRREADING or DATE_RTLREADING. DATE_AUTOLAYOUT is preferred over DATE_LTRREADING and DATE_RTLREADING because it uses the locales and calendars to determine the correct addition of marks.
Add marks for left-to-right reading layout. This value cannot be used with DATE_RTLREADING.
Add marks for right-to-left reading layout. This value cannot be used with DATE_LTRREADING
But the last decade of managed code support in the System.Globalization namespace has been unable to produce any version that will either use this functionality to format date strings.
And that same decade has failed to produce any code designed to parse strings produced via any of these flags.
Note that "support" for parsing would simply be adding the ability to ignore U+200e and U+200f, but supporting the parsing would certainly lead to a demand for support of the formatting.
Unfortunately, it is very common for tests of many different components to be written in managed code -- which means this question comes up a lot more often than one might expect, given the need to use these flags to get strings that will display properly....
The workaround?
You will need to walk the string, stripping out all instances of the following characters -- the first two in the table below are inserted by GetDateFormat[Ex] when passing any of the three flags above, the rest could be inserted by other, more sophisticated processes (or RtL language localizers doing their job):
Now looking at the reason that we could really go more than a decade without managed code supporting something that native code added so long ago, there are a few (competing?) theories:
In the long run, given that there are such issues, it would be nice if some team just forgot about the politics and tried to solve the problems....
From a Microsoft standpoint, the number of groups that write automation that use managed code is significant enough that I think fixing these problems could be justified solely on a "being a good internal Microsoft citizen" standpoint. But maybe that's just me. :-)
It isn't always true, but it often is -- the issue, when it came up, was brought up by a tester.
This makes sense, really.
No matter how much self hosting people like me might do, by the very nature of the job they do, most of the day of the tester is a mix of random ad-hoc, directed ad-hoc, and other testing....
Now do you remember when I talked about pseudo-localization, back in One of my colleagues is the "Pseudo Man" (a rich source of puns in conversation!)?
Well, a tester noticed that in the pseudo build a bunch of strings that were supposed to have "t" in them seemed to have "m" in them, instead -- in the pseudo build.
A shaping problem? A font problem? A pseudo problem? No one was sure.
Well, I did know there was no shaping involved. A font problem seemed unlikely, and a pseudo issue seemed far-fetched.
The mail thread had some perplexed folks on it, let me tell you.
Now the way pseudo works, each letter has a set of various "replacement" characters that look like them. So take, for example:
t U+0074 LATIN SMALL LETTER T
For this letter, there are several potential "alternates":
There was a part of me that was sad that the Tenge was not on the "Capital T alternatives", especially after I wrote It is with a tenge of sorrow that I say this. I mean, I'm over it now. But I was sad for a little while....
Anyway, back to the t that becomes an m, do you know what's going on?
Just think of it as yet another flavor of the issue i talked about in blogs like Small case is not just tinier capitals; italics are not merely slanted letters and You say ĭtalics, I say ītalics. It is much more complicated in Cyrillic...., previously.
Judy pointed out some of the form issues, with art:
This fact interested me for many different reasons.
So I went off and looked at what various fonts did.
I am pretty sure something in here might be weird. Probably not bugs though:
Sometimes your t is gonna look like an m; deal with it!
As you work though the IDN story in your own company, you will likely find an interesting mix in the support story, just as I have.
Perhaps one of those most interesting areas has been in the tiny (one might say "puny" to invoke a groaner of a pun!) detail work.
Like path detection.
There are algorithms in RichEdit that colleague Murray Sargent tells me are quite sophisticated. You get to it in theRichEdit with things like EM_GETAUTOURLDETECT and EM_AUTOURLDETECT and such. He has even mentiomed the AutoURL stuff in his blog before (like here).
Or in places like WordPad you just type a URL or a path, or load a file with one, and you can see the effect of turning the behavior on via these programmatic means -- it wil detect the path and mark it as if it was a clickable URL/path in a browser.
If you move over to Word it's even more sophisticated, with a config option in the UI:
The good old "Replace Internet and network path with hyperlinks" feature!
Very cool.
Until you include IDN in the mix, at least.
Let's take four URLs that could easily be created if you have set up a machine to test out IDN (server names/namespaces changed to protect something or other), and try to put them in Word 2010:
which was unable to properly detect one of the URLs out of the four.
Can you guess why it failed?
Or you could try it the same four URLs in WordPad on Windows 7:
Wow, 0/4. Not too sophieticated!
I'll point Murray to this post, and within a few hours he'll tell me that the latest version of RichEdit (essentially the one on his machine) supports all four URLs.
Of course not everything on Murray's machine gets checked in without a bug report so I'll work on that too. :-)
Let's try pasting those same four URLs here to see what this Blog Editor does wih them:
http://नांदरी.日本国.test.corp.testcompany.com http://idn-iis1.日本国.test.corp.testcompany.com http://idn-iis1.日本国.test.corp.testcompany.com/интернет_страница http://テストサイト.test.corp.testcompany.com
Wow, that's disappointing.
It looks like there are a bunch of URL detection functions that don't do so well with IDN.
I wonder if UNC paths fare any better?
I'm just kidding, I don't wonder. because I tried it.
\\idn-iis1.日本国.test.corp.testcompany.com\сетевой
The other two (WordPad and word 2010) behaved about like the third URL did, for reasons that might be obvious if you think about how tha Autodetect code works (or doesn't, in this case).
Now this kind of stuff is obviously not core feature work, it's a nice little "extra", but really it isn't so nice when it screws up.
As it does with IDn on the absolue latest version of evedry product/app/control I had immediate access to.
Sounds like there are some bugs for people to look at, huh? :-)
In the end, the roadpath to Hell is paved with IDN bugs!
This is gonna be a weird one. Not a tehnical one. But a technically weird one, in some sense....
I'm going to introduce my story I am going to tell with someone else's story. A story from So long, and Thanks For All The Fish.
I won't claim that my story will be as funny, or as fun. But it may put you in the right frame of mind to read the story I will tell you....
"I'll tell you a story," said Arthur.
"Good."
They found a patch of grass which was relatively free of couples actually lying on top of each other and sat and watched the stunning ducks and the low sunlight rippling on the water which ran beneath the stunning ducks.
"A story," said Fenchurch, cuddling his arm to her.
"Which will tell you something of the sort of things that happen to me. It's absolutely true."
"True Story."
"You know sometimes people tell you stories that are supposed to be something that happened to their wife's cousin's best friend, but actually probably got made up somewhere along the line.
"Well, it's like one of those stories, except that it actually happened, and I know it actually happened, because the person it actually happened to was me."
"Like the raffle ticket.", said Fenchurch.
Arthur laughed. "Yes. I had a train to catch," he went on. "I arrived at the station…"
"Did I ever tell you," interrupted Fenchurch, "what happened to my parents in a station?"
"Yes," said Arthur, "you did."
"Just checking."
Arthur glanced at his watch. "I suppose we could think of getting back," he said.
"Tell me the story," said Fenchurch firmly. "You arrived at the station."
"I was about twenty minutes early. I'd got the time of the train wrong. I suppose it is at least equally possible," he added after a moment's reflection, "that British Rail had got the time of the train wrong. Hadn't occurred to me before."
"Get on with it." Fenchurch laughed.
"So I bought a newspaper, to do the crossword, and went to the buffet to get a cup of coffee."
"You do the crossword?"
"Yes."
"Which one?"
"The Guardian usually."
"I think it tries to be too cute. I prefer The Times. Did you solve it?"
"What?"
"The crossword in The Guardian."
"I haven't had a chance to look at it yet," said Arthur, "I'm still trying to buy the coffee."
"All right then. Buy the coffee."
"I'm buying it. I am also," said Arthur, "buying some biscuits."
"What sort?"
"Rich Tea."
"Good choice."
"I like them. Laden with all these new possessions, I go and sit at a table. And don't ask me what the table was like because this was some time ago and I can't remember. It was probably round."
"All right."
"So, let me give you the layout. Me sitting at the table. On my left, the newspaper. On my right, the cup of coffee. In the middle of the table, the packet of biscuits."
"I see it perfectly."
"What you don't see," said Arthur, "because I haven't mentioned him yet, is the guy sitting at the table already. He is sitting there opposite me."
"What's he like?"
"Oh, perfectly ordinary. Briefcase. Business suit. He didn't look," said Arthur, "as if he was about to do anything weird."
"Ah, I know the type. What did he do?"
"He did this. He leaned across the table, picked up the packet of biscuits, tore it open, took one out, and…"
"Ate it."
"He ate it."
Fenchurch looked at him in astonishment. "What on Earth did you do?"
"Well, in the circumstances I did what any red-blooded Englishman would do. I was compelled," said Arthur, "to ignore it."
"What? Why?"
"Well, it's not the sort of thing you're trained for is it? I searched my soul, and discovered that there was nothing anywhere in my upbringing, experience or even primal instincts to tell me how to react to someone who has quite simply, calmly, sitting right there in front of me, stolen one of my biscuits."
"Well, you could…" Fenchurch thought about it. "I must say I'm not sure what I would have done either. So what happened?"
"I stared furiously at the crossword," said Arthur. "Couldn't do a single clue, took a sip of coffee, it was too hot to drink, so there was nothing for it. I braced myself. I took a biscuit, trying very hard not to notice," he added, "that the packet was already mysteriously open…"
"But you're fighting back, taking a tough line."
"After my fashion, yes. I ate the biscuit. I ate it very deliberately and visibly, so that he would have no doubt as to what it was I was doing. When I eat a biscuit," Arthur said, "it stays eaten."
"So what did he do?"
"Took another one. Honestly," insisted Arthur, "this is exactly what happened. He took another biscuit, he ate it. Clear as daylight. Certain as we are sitting on the ground."
Fenchurch stirred uncomfortably.
"And the problem was," said Arthur, "that having not said anything the first time, it was somehow even more difficult to broach the subject the second time around. What do you say? ‘Excuse me… I couldn't help noticing, er…' Doesn't work. No, I ignored it with, if anything, even more vigour than previously."
"My man…"
"Stared at the crossword, again, still couldn't budge a bit of it, so showing some of the spirit that Henry V did on St Crispin's Day…"
"I went into the breach again. I took," said Arthur, "another biscuit. And for an instant our eyes met."
"Like this?"
"Yes, well, no, not quite like that. But they met. Just for an instant. And we both looked away. But I am here to tell you," said Arthur, "that there was a little electricity in the air. There was a little tension building up over the table. At about this time."
"I can imagine."
"We went through the whole packet like this. Him, me, him, me…"
"The whole packet?"
"Well it was only eight biscuits but it seemed like a lifetime of biscuits we were getting through at this point. Gladiators could hardly have had a tougher time."
"Gladiators," said Fenchurch, "would have had to do it in the sun. More physically grueling."
"There is that. So. When the empty packet was lying dead between us the man at last got up, having done his worst, and left. I heaved a sigh of relief, of course. As it happened, my train was announced a moment or two later, so I finished my coffee, stood up, picked up the newspaper, and underneath the newspaper…"
"Yes?"
"Were my biscuits."
"What?" said Fenchurch. "What?"
"True."
"No!" She gasped and tossed herself back on the grass laughing.
She sat up again.
"You completely nitwit," she hooted, "you almost completely and utterly foolish person."
She pushed him backwards, rolled over him, kissed him and rolled off again. He was surprised at how light she was.
Okay. So now I will tell you my story. It won't be as interesting, I promise you.
Though it is indeed a story about the sort of things that happen to me, and it is completely true. And I know this, because the person it actually happened to was me.
I have a mailslot at work.
Now I get hundreds of emails every day (not including spam), but very little physical snail mail is sent to me at work.
In fact, the occasional times I go to clean out the stuff that builds up over time, it is just a little bit of junk mail. I dump it right into the recycle bin that is conveniently located in the same room.
You can likely see why my visits to clean out the mailslot are so rare.
The other day is when it was not going to turn out to be an ordinary trip to the mailslot.
Buried with the announcements about conferences I would never go to, products I would never use, and other such junk mail, there was a wedding invitation.
A Wedding Invitation.
It was for the daughter of a former colleague of mine.
He and I did do some work together, though we were never terribly close.
He recommended I may not be a good fit when I was thinking about joining a team he was on a few years prior. That kind of thing is easy enough for me to look past in work situations, but it doesn't tend to encourage social interactions outside of work, if you know what I mean.
Anyway, the wedding was out of state.
I don't ever recall meeting the daughter, and I know I had never met the son-in-law to be.
I had never been to the former colleague's house, and he had never been to mine, which I guess explains why the invitation was sent to me at work.
He didn't know my address!
On the other hand, it explain nothing -- as it all kind of indicates the reasoning for not sending an invitation (when one doesn't know someone).
Probably would have been worth a call at least, so I'd know it was coming.
I had a not entirely comfortable moment where I did a mental back and forth and forth about what to do -- whether to go, whether to decline, whether to send a gift, etc. -- I have definitely been known to show up in unexpected situations before, so a wedding I was invited to would hardly be too weird. Though the chance I might decide to randomly make the trip might be an argument against the invitation, if you ask me -- I can be pretty unconventional. So my mind was working through all the possibilities, in that moment.
Before I noticed that the wedding had already happened, months earlier.
You may recall I told you I didn't check my mailslot very often!
I did go down the hall and ask another colleague who used to be on that same team, but it turns out they hadn't been invited. We speculated on the topic for a few moments and, and they suggested some theories about the whole matter.
"So you don't think it's weird?" I asked.
"Oh no, it's weird," they responded.
But what to do next?
Miss Manners has probably covered the subject of the strange invitation before, right? Someone has to know what he right response would have been.
Or what it should be now, if anything....
In the end, I couldn't think of a way to respond after not responding in a timely manner.
So I figured I do the next best thing -- I'd write a blog about it!
I'll see what I can do about ending on the same note as the Arthur Dent story. You know, finding a girl to push me backwards, roll over me, kiss me, and roll off again. It'll probably have to wait until the weekend, at least....
So I was asked if I knew what the following comment from the site was about:
The Microsoft Implementation of the Umm al-Qura Calendar
The implementation of the Umm al-Qura calendar in Microsoft’s recent operating system Vista is claimed to be valid between 1318 AH and 1450 AH. However, as the Microsoft Vista algorithm erroneously assumes that the computational rule used between 1395 AH and 1419 AH was also used before 1395 AH the Microsoft algorithm will often give faulty results for dates before 1395 AH.
I am really not sure what this comment refers to.
I mean I have mentioned issues I know about in blogs like Long term planning is not always done, and Using a culture's format, without using that culture to format?.
Not to mention Evil Date Parsing lives! Viva Evil Date Parsing!, explained.
Or Grody to the Max[Date]!.
Even without issues that would cut down our effective dates, looking to the future the 1450 limit would take us to (unless my mental math is off) some time in 2028, which is not that many years away.
With computer programs written in .Net that work with MaxDate so gratuitously (and I have seen many of those), this is not a very wide range.
Given that there are features in Windows that can deal with dates outside of this range (including certificate creation), I think any application running on Windows that doesn't handle the Um al-Quara calendar is pretty irresponsible....
By the way, does anyone know what that "bug" in our implementation is? I'll ask some people around here, but I thought I'd ask my readers first in case anyone knew.
Long time readers may recall Who owns English, exactly? from back in November 2006 where I talked about the philosophical issues underlying who was the "owner" of transliterations and/or translations of native language names into English.
Everything I said there was of course true, and the issues are of course very complicated.
But there is an additional factor to be considered here too, on much wider than the "corner case" of translated/transliterated language names.
The factor can be summed up in a simple question:
Who the @#%&*! owns en-US?
I mean, it's not like we have an English Language Academy we are beholden to, and there is no special ANSI standard we claim to be compliant with.
The original locale was put together by people in the US, sure. And English Windows has never suffered from not enough beta testers.
But all of that is hardly official, is it?
This occurred to me the other day when someone asked me via mail:
You have the value for LOCALE_IPOSITIVEPERCENT for the US locales as 0, shouldn’t this be 1?
Now the LOCALE_IPOSITIVEPERCENT lctype documentation delineates the choices pretty clearly:
Now I have lived in the USA my entire life, though I have traveled abroad a lot (enough to make filling out an SF-86 ti be too troublesome to ever bother trying to get cleared!).
If you showed me a formatted string like
%50
or
% 50
I'd know it was wrong. But looking at
50%
vs.
50 %
I could go either way; I see both of them, all the time.
To be honest I don't like either of them; I think
using a HAIR SPACE (U+2009) or a THIN SPACE (U+200a) in between is better than any of the other choices -- between the original two "not wrong" choices, one is too close and the other is too far
But we don't have either as an option.
And it's not like we have an authority we can ask.
Since nobody @#%&*! owns en-US....
It was a bug reported to me by twitter friend Rifat Nabi (@rifat) with a simple caption fom the tweet of "Microsoft Microsoft Word Bug":
and by twitter friend Tanbin Islam Siyam (@potasiyam) in a picture with a bit more detail for those who didn't see what the problem was in the first picture:
And some text explaining it, too (and pointing to another instance):
Another bug of MS Office.An user of avro keyboard posted this bug in omicronlab forum (http://omicronlab.com/forum/index.php?showtopic=2436&hl).Quotation marks don't change after Dari (দাড়ি - । ), as it should be corrected to right quotation marks. But while typing in English it works.I also tried typing in hindi with on-screen keyboard. Same result as Bangla.
(Windows xp sp3+Office 2010)
The options in Word we are talking about is the one in this form:
and are documented as:
* "Straight quotes" with “smart quotes” Replaces straight quotation marks ( " " ) and straight apostrophes ( ' ) with curved, open- and closed-quotation marks ( ) and curved apostrophes ( ).
which really says so little about what the options are....
It is hiding a huge feature, the one described on the Wikipedia article Non-English usage of quotation marks.
Of course the feature depends a bit on the "language info via keyboard settings" stuff that so many people don't care for (Word does not retroactively change quotes if you tage text with a language after the fact).
Now every developer has had to at one time or another write their own FInQuotes function, or debug a flawed or incomplete one.
And this bug is just a simple limitation in mighty Word's work to detect open quotes versus closed quotes -- like maybe it isn't smart enough to see some scripts as text within a language, even as the text was tagged with a language due to being input with a Bangla keyboard....
I have previously talked about this feature in blogs like Dumb quotes... or maybe they are just smart-ass quotes. And that blog is cool because it provides the table with data that Word uses for this feature.
The version of the table I was given clearly doesn't have much for these languages (admittedly that was from 2007; Word 2010 may have a bigger list), but note that there may still be problems in trying to deal with characters unknown to the code Word uses to detect text in quotes....
To me, this is just another reason that I am often unhappy with some Office features (e.g. like I described in Address formats are hard, let's go shopping!) that are scenario-based and that do delight some customers, but which are ultimately incomplete.
Someone forgot that language support is a scenario-based feature, too. And did the work to finish the feature....
Today's blog will start with a digression.
I got an HTC Arrive from Sprint to replace my troubled Palm Pre (which was itself a replacement for a broken LG). On the whole the Arrive is okay, though there are four problems with it:
This third problem is the one that is relevant to today's blog; everything else is, as I suggested at the very beginning of the blog, a digression.
It all started when I happened to notice an @ reply to me from @DrPizza (you may know him as Peter Bright) from over a month ago:
Then when I looked at those links, I saw there were several other @ replies from Peter from several days prior to that:
Okay, so it looks like I basically blew off Peter Bright. I guess the @IE account did too (I couldn't find a response from them, at least).
Sorry about that Peter!
Anyway he also gave some more explicit instructions in case I wasn't sure how to to test the result easily:
Yep, that's easier than what I would have done (built a small HTML page to repro it).
Okay so if you look at the Zero-width space Wikipedia page:
and follow the steps:
you will see a result slightly stranger than what Peter was suggesting:
What the hell is it doing?
Let's sretch the page back out to see what happens:
Okay, at least is consistent.
I guess it is somehow related to strange behavior with U+200b, aka ZERO WIDTH SPACE.
But it is not as simple as putting a full space in where a ZWSP was sitting -- the "space" is inserted between the character after the ZWSP and the character after that.
Internet Explorer lets you select just this space and if you paste it somewhere else you can verify it is a U+200b.
This doesn't happen in Notepad, or WordPad, or Word. Or in FireFox for that matter.
A pure IE issue, I guess.
And I could only repro it with Consolas (no other font I tried had the problem). And IE.
Your guess is as good as mine as to whose fault it is, though I'll ping my favorite folks from IE and MS Typography to see if they can explain what's going on here.
And from now on, now that I know I'm not being gently told when someone @ replies me without spamming me with additional text messages, I'll check periodically for responses.
You know, so I don't blow off people whose columns I enjoy reading next time....