I'm not a Klingon ( ) : IDN (Internationalized Domain Names)

The FUD of IDN and Homographs

shawnste — Tue, 24 Nov 2009 00:13:00 GMT

I was pointed to this article http://www.microsofttranslator.com/BV.aspx?ref=Internal&a=http%3a%2f%2fwww.bortzmeyer.org%2fidn-et-phishing.html about IDN and homographs, which points out that most of the fear around IDN and phishing is unfounded. Seemed like a good reference (thanks, Mark), so I'm forwarding. (For some reason Mark used a different translation engine though).

Cross-tagged with EAI since the same concerns about homographs and phishing apply to email.

IDNA2008 / IDNAbis on Windows 7, Vista, Net, etc.

shawnste — Tue, 27 Oct 2009 18:04:00 GMT

Some people have asked what they should do to support IDNA2008 on Microsoft platforms.

We provide IdnToAscii() and related functions in the Windows SDK. That's available natively on Vista+, and through idndl.dll on earlier platforms. Idndl is shipped with IE 7, or through "Microsoft Internationalized Domain Names (IDN) Mitigation APIs" at the Microsoft Download Center.

For .Net, the IdnMapping class provides IDNA2003 conversion since .Net V2.

Obviously these APIs currently only support IDNA2003, however the interfaces won't change for IDNA2008 support. I don't know what mechanism will be used to update for IDNA2008 support, but applications should continue to use the exposed APIs for consistent support across the platform. That way if the system gets updated to IDNA2008, the application will be able to take advantage of the updated support.

Oversimplification of EAI/IMA (International eMail Addresses)

shawnste — Wed, 19 Aug 2009 00:58:00 GMT

A couple months ago I blogged about EAI Email Address Internationalization/Internationalized Email Addresses (EAI/IMA) and felt like blogging again.

China's been very interested in non-ASCII email addresses for some time, and is working hard to adopt the EAI standard. I've heard a target of November 2009 for that standard. http://www.china.org.cn/china/sci_tech/2008-09/27/content_16544162.htm briefly addresses EAI.

Oversimplification of EAI

The basic concept of EAI is "just" to use UTF-8 for email. Most software can comply just by allowing Unicode in their email addresses. Using UTF-8 is reasonably straight forward, and most of the details are just around compatibility with existing mail standards. The IETF working group has a page at http://www.ietf.org/dyn/wg/charter/eai-charter.html.

Local Part of the Email Address

The local part of an email address is the user account part. Often times servers allow it to be case-insensitive, however it can also be case-sensitive. Similarly EAI allows the servers to define any mappings of the local part that are appropriate for that organization. Some may choose to do case mapping similar to existing case-insensitive servers. A different mapping, like Turkish behavior for i and I is possible. Another option would be to perform normalization like NFC or NFKC on the name. Width mapping and aliases are possible. Just like now, clients would just use the names given and let the recipient's mail server figure it out.

Domain Part

EAI allows Unicode (UTF-8) for the entire address, so special mapping isn't necessary. Of course if the domain doesn't have a valid registration, eg: isn't valid IDN, then it won't work, but that's not really an email protocol issue. EAI uses UTF-8 instead of "punycode" for domain names. Punycode only happens when "downgrading."

Negotiation

Mostly, "just using UTF-8" is pretty simple, but for backward compatibility, EAI aware servers and clients will need to negotiate their protocols. For SMTP, the UTF8SMTP does this. EAI aware servers can exchange the UTF8SMTP extension and agree to communicate in UTF-8. If the server doesn't provide that flag, then the client's have to use a different mechanism. The other protocols have similar handshaking.

Downgrading

All email clients and servers aren't going to instantly become Unicode aware, so there is a downgrading concept for compatibility. Downgrade is the area with the most churn in the experimental standards, but the basic concept remains the same.

If you have an EAI aware server and you try to talk to an unaware system, you'll need to fallback to the legacy protocols and encoding mechanisms. Effectively this means that EAI accounts will need an ASCII alias so that if an EAI mail fails, it can be resent using the ASCII alias and MIME encodings.

To a legacy recipient, such a mail would appear as any other legacy email, and replies would go to the sender's ASCII alias. The receiving server would need to recognize that the ASCII and Unicode EAI aliases were for the same account and route the mail appropriately.

There was some discussion of providing additional data that allows reconstructing a downgraded mail, but most of those techniques seem to break at least some legacy clients and have additional problems. My feeling is also that if a client knows how to reconstruct a downgraded mail, it also knows EAI anyway, so likely the mail would never be downgraded, so the additional complexity is unnecessary. I think it's likely that the initial standards will only specify minimal downgrading and not the ability to reconstruct a downgraded message.

Status

Of course the IETF RFCs are still experimental and China hasn't published their standards yet, but my oversimplification probably won't change much in the final version.

Unicode, IDN (IDNA), EAI (IMA) and Homograph Security

shawnste — Tue, 07 Jul 2009 22:33:00 GMT

I wrote about IDN & Security before http://blogs.msdn.com/shawnste/archive/2005/03/03/384692.aspx but thought I'd share some of my more updated views about security of URLs/IDN/Unicode/Email addresses.

People haven't really bothered much with DNS or character based security when it was limited to ASCII. I'm not sure if this because people just didn't think about it, or if they thought there wasn't a problem or whatever. What security attacks happen have been regarded more as "oh, that's curious" rather than a real concern. Basically there seems to be a presumption that a script, like the ASCII subset of Latin, are inherintly secure. Therefore it would seem reasonable that if ASCII Latin can be secure, then other scripts, or mixed script environments have homographs, then those scenarios must be insecure and are therefore broken.

Latin and ASCII aren't Secure

The problem with that logic is that it's flawed. Homographs exist in Latin/ASCII, however http://rnicrosoft.com tends to be regarded as "quaint and amusing" rather than a security problem. (There used to be a web page there, dunno what happened). Similarly g00gle or MlCROSOFT or whatnot can all happen in ASCII. Some things can be done to ASCII to limit the risk, such as choosing fonts or making things lowercase, but that's not always possible.

Strings are Typed and Read by Humans

Even if the scripts themselves are perfect, the strings we use with the scripts are not. For example, users have to type them in, and they may or may not use upper or lower case (in cased scripts). I heard one computer expert indicate that users should just figure out how to enter URLs in lower case, in Unicode Normalization Form C. (Instead of addressing the problem we should educate all the users). I wish he were joking.

Depending on the context, there are things you can do to ASCII only strings that can confuse users. For example http://microsoft.secure.com isn't going to necessarily go to a Microsoft site. http://secure.com/microsoft.com is a similar trick.

DNS isn't the only subject of these problems. I get mail all the time in the form company@mail-servicing.com where "company" is a legitimate company and "mail-servicing" is the people they've contracted to send their bulk mail. So it's impossible for me to determine if that's actually a good address for the company. Even worse is when the mail contains a link. "Provide feedback about your recent warrenty support to http://feedback-surveys.com/OEMsupport"

Strings aren't Even Strings

Sometimes what we click on isn't even related to where we end up going. We've all seen phishing attacks that are look like mybank.com but go to an IP address that no one can tell if it's real or not.

Strings aren't Always Specific

In some environments strings often aren't even very specific. I'm pretty certain that if I want a live.com account that I won't get shawn or shawns or even shawnsteele. Instead I'll be shawn7935 or something. There's another Shawn here at work that gets some of my mail from simple typos, let alone malicious intent. There's a pretty good chance that Fred8374 could pass himself off as Fred8347 if he really wanted to.

We've even been trained that strings don't even have to be close. If I buy something on eBay from "JoesBestStuff", it takes some faith for me to pay SallySewing7@live.com (apologies if those are real accounts). I've been quite amused at the varation betwee "seller's name" and the email sometimes.

Even when we expect them to be the same, there are many spellings for some words. "Mohammed" is often transliterated differently to Latin. Unless you deal with one quite often, you're likely to assume most spellings are the same.

Globalization of Strings

Now we've figured out that strings aren't secure, and we'll get tricked even if they were secure. How does that change in a global environment, such as with IDNA or EAI/IMA strings? Not much.

Sticking to Latin, you suddenly gain a bunch of look-alikes (homographs) by allowing non-ASCII values. Strings like mícrosoft, mïcrosoft and mıcrosoft are all “close enough” to be convused, particularly at a quick glance, even more so if the user is conditioned to expect the "real" string. E.g: "Important security update for windows, go download it from Mícrosoft.com" We're already expecting to see microsoft, so the few different pixels are easily missed.

For other scripts the problem can be much more severe. Complex scripts can have simliar appearing strings, and many include numerous characters. Chinese for example has enough characters available that it can be fairly easy in some cases to find a rare character that is similar in appearance to a common character which people have been preconditioned to expect.

"I Solved Homographs"

This leads to a typical problem for developers, particularly "Western" Latin-script based developers. We tend to expect that if we solve script mixing so that we can't mix up Cyrillic and Latin, that we've solved the homograph problem. Instead, we've barely scratched the surface and effectively buried our heads in the sand.

In some cases the "solution" can be worse than the problem. For example, some browsers decide that I don't understand Cyrillic since my user locale is en-US (or Klingon), and then prints out punycode. That's mildly useful to me as a warning, however it does the same thing for Chinese. It's very unlikely that I'm going to confuse Chinese with Latin, but I'll get Punycode in the address bar anyawy. Now I have no chance of finding out what the actual URL is supposed to look like. Punycode is all gibberish, but I could probably decipher a Chinese glyph enough to see if it looked similar to what I expected. With any punicode strings, I don't even need homographs to confuse me, any Chinese would look the same. For that matter I could be expecting Chinese, but it could actually be Japanese or Korean, or Cyrillic for that matter. I'm not trying to say that the browsers' approach is "wrong", just that while this approach may address some problems, it can also cause new ones.

Most of the "solutions" to Homographs that I've seen are similar in my opinion. They may address a specific issue, but don't solve the entire problem globally. I also think some approaches are unnecessarily limiting. Mitigations that reduce the surface area for an attack are useful, however developers should recognize the limitations of those approaches and make sure they aren't spending tons of effort "shutting the window, but leaving the front door wide open." That only provides a false sense of security, which can be far worse than the original problem.

Comprehensive Solutions

So instead of thinking that strings like URLs are inherintly secure somehow if they're ASCII, and focusing on the differences from ASCII, like Cyrillic homographs, we should rather assume that ANY URL might not take us to a place we want to go. Even an ASCII one.

A much better solution to URL security is one that addresses the entire system rather than focusing on Homographs. IE, for example, detects malicious web sites (I don't know exactly how it works, but I gather there's blacklisting and bad behavior detection, kinda like virus checking for web sites). This is far more effective than preventing mixed scripts, and has the advantage of working with ASCII only URLs. It also does a good job against homographs, pretty much making the punicode-in-the-address-bar irrelevent. It also works with many forms of attack, even non-obvious ones.

My opinion is that if you do a "good job" of detecting any phishing/spoofing type web site, even ASCII-only, then the need for Homograph detection is much reduced. And if you can't do that, then the attackers will merely add an extra label or something to get around your homograph detection.

Mitigation by Protocol

For things like IDN, it is interesting to consider how the protocol itself approaches security. Some things are "obvious" as not being interesting for a name. Compatibility characters, control characters, etc. could somewhat readily be excluded. Some things are generally considered technically "obvious" to some users, but may frustrate others. It is generally considered that lower casing the DNS name causes less confusing (can't mix up lower case l with capital I), but I doubt that AAA.com prefers lower casing. Similarly IDNA2003 allows unicode "symbols," which are widely regarded as being useless, particularly since they're hard to type, but I suspect that someone would like I♥NY. So there's a gray area that gets a bit confusing.

Consideration for other protocols is similar. EAI (email) is interesting because it basically defers "correctness" to the registrar (whoever runs the mail server). IDN provides some restriction by protocol and more at the registrar level.

One problem with restricting valid characters at the protocol level is that it works OK in a small set, but once you get to a global audiance the rules get very complicated. Domain names allowed (most) English names when they were restricted to ASCII, but German and French had difficulties. With IDN additional languages are supported, but perhaps the needs of an English registrar and a German one differ. A complete set of rules applicable world-wide for all strings in all languages may not be possible (eg: turkish i), but even if they were, they would be very complex and difficult to implement for every application adopting a protocol.

Mitigation by Registrar

Restriction at the registrar can be more effective, though perhaps less consistent. A registrar could be like a domain name registrar, but for these purposes you could also think of the person that assigns user accounts at a business, or email address registration from your ISP.

Registrars can restrict languages to those used in the country they support. They can bundle or block homographs or alternate spellings (like Traditional and Simplified Chinese spellings of the same word.) In a business they could have certain rules. First name, last initial, or first initial, last name is common for user accounts in many companies, at least until they get too many employees).

IDN has some restrictions by protocol, but allows much tighter restriction at the registrar level. Ironically, a label at a lower level could then have different "rules" than at the higher level. EAI allows the local part to be determined entirely by the provider/registrar rather than the protocol.

Rules at the "registrar" level can still be very complex for a complete set of rules, however cases with conceptual differences can still be adopted as applicable for the registrar's environment, whereas a protocol level rule has to either be too flexible, or disallow one registrar's legitimate scenario. Rules at the registrar level can also be adjusted more readily than at the protocol level.

Mitigation by Application

An application can also decide to be more comprehensive than the protocol. An application may also have more information, such as blacklists or user settings. They can make choices for some users like "they only read English, so don't bother with Cyrillic then," and a different choice for a different user. Applications can also potentially be grayer in their behavior. Instead of "allowing" and "disallowing" strings, they can say "gee, I'm not so sure, you really want to do this?", or flag it and continue. They can also be dynamic, such as when you add a sender to a junk mail filter.

IDN vs EAI/IMA vs Unicode

Pretty much this entire "strings aren't secure" concept applies to any Unicode (or for that matter any other code page) string. That could be an IDN domain name, an EAI mail address, a user account name, etc. Some environments may be more ameniable to certain solutions than others, but the types of attacks that impact a Unicode IDN label could also succeed with the local (user name) part of a Unicode EAI email address. The general concepts are portable.

I used IDN heavily as an example, but the same things happen to EAI addresses, user account names, logon credentials, etc. Anything that uses Unicode, or strings, needs to realize that strings can't be expected to be inherintly "secure."

There's more info on some thinking about Unicode Security in Unicode TR#39 http://www.unicode.org/draft/reports/tr39/tr39.html. TR39 addresses the appropriate use of Unicode characters and homographs, but this is at best a mitigation of the more general security concerns of identifier strings. Phishing and spoofing would still happen even in plain ASCII.

Hope this was helpful, or at least interesting,

Shawn

Email Address Internationalization / Internationalized eMail Addresses (EAI/IMA)

shawnste — Fri, 05 Jun 2009 03:48:00 GMT

With the IDN work for Internationalized Domain Names using characters beyond ASCII, it is only natural to tackle the problem of Internationalized Internet eMail.

Some smart people have been working on an IETF working group to figure out how non-ASCII email would work, and I encourage people to take a look: http://www.ietf.org/html.charters/eai-charter.html. That page has the charter, a list of drafts and RFCs that have already been produced, and links to the IMA working group mailing list.

Assuming you're an ASCII/Latin character user, imagine having to type all your URL's in Chinese, or Cyrillic (or if you know those, imagine typing everything in Klingon, eg:  ) In many cultures, that's what it's like to use the web. Some users may not be literate in Latin letters, or may have to do a lot of hunt-n-pecking. EAI should help address that problem.

How EAI/IMA Works

The basic idea of the EAI working group is to stick email in UTF-8 instead of ASCII. UTF-8 works pretty well in many systems, and many mailers already handle 8 bit encodings, so this is a pretty "simple" solution. Unfortunately email touches a lot of places, so there're a lot of protocols that need updates (eg: STMP, POP, mailto:, etc.) Additionally everyone knows that UTF-8 email can't happen instantly, so there needs to be a system for existing servers to talk to UTF-8 aware ones, which leads to a few more RFCs.

UTF8SMTP allows the servers to make decisions about the "local" part of the email address, which allows for groups to fit their own needs. The backwards compatibility means that users also need ASCII addresses, as they do today. The server would alias from one address to another so mail to @microsoft.com could map to my normal mailbox, and I'd only have one mail. Unfortunately that simple concept means that places that didn't have to worry about aliasing before may now have to consider aliases and fallback addresses. Contact lists may need to have both forms, etc.

Current Status of EAI/IMA

Currently there are several experimental RFCs, and several people have created interoperating systems that work with each other to demonstrate the feasibility of UTF8SMTP. The next step is to move towards a standards track process, which could happen "reasonably quickly". I'm optimistic that the standards will move quickly, but sometimes these things take a while.

So Who's Gonna Use It?

There are a lot of markets where ASCII doesn't work very well for various reasons. Even when people have ASCII aliases, it may seem artificial, and there may be a desire for an email that reflects them or their country. There are many ISPs in countries like Korea, China, & Japan that are very eager to be able to send email in a native script. Some governments like Russia and China are weighing in on the importance of being able to send mail and use the Internet in their script.

What's IMA Mean To Me As a Software Developer? (who cares?)

If you are a developer, then you may run into IMA addresses. Even if your app doesn't explicitly deal with mail, there may be a place for email to sneak into your app. For example, IDN and domain names don't really have much to do with Word or PowerPoint, yet they often show up in documents and presentations. I could imagine an author address in metadata, such as a photographer contact in a photo's metadata. Many apps probably will run into IMA addresses whether they realize it or not.

Anyway, I have been thinking about this space for a while and thought I'd share my observations. It's worth considering what impact IMA will have on your application (while you're at it, how's IDN behave?)

-Shawn

Rambling about RFC 4690 and IDN

shawnste — Wed, 11 Oct 2006 04:52:00 GMT

There's a reasonably new RFC 4690 ( http://www.ietf.org/rfc/rfc4690.txt ) that raises a bunch of questions about IDN names and Unicode regarding such things as confusable characters and other issues. Some of those are also discussed in Unicode TR36 "Unicode Security Considerations" http://www.unicode.org/reports/tr36/.

One thing that confuses me about the discussion regarding IDN's weaknesses is that most of these issues wouldn't be a problem if the registrar's didn't register the names. So from the client application's perspective, it doesn't matter much if a particular IDN name is legal or not, so long as it does the appropriate mapping. If its not a legal IDN name, then web browsers or other apps won't respond to it because the DNS system won't return any records. So from the client perspective I'm not sure what all the fuss is about.

Obviously the registrars need to be able to disallow bad names, and guidelines such as TR36 help. Registrars also have the opportunity to be pickier than the standards. Many only allow certain scripts or combinations relevent for their TLD.

For similar reasons the migration of IDN to Unicode 5 or newer doesn't bother me. Nobody's going to allow unassigned code points in their domain name. If a Unicode 5 code point appears in a future IDN name, and the DNS system resolves it, it would follow that it is a valid code point, even if some client app only understood Unicode 3.2. That doesn't help case mapping, but a pure Unicode 5 name should be easily understandable and resolvable even by downlevel clients.

Anyway, it seems to me that there's more fuss about these issues than there needs to be. Probably if the IETF IDN folks and the Unicode folks worked with each other to resolve these issues then IDN would get updated faster. Right now it seems like each group is reacting on its own.

IDN Test URLs

shawnste — Fri, 15 Sep 2006 00:57:00 GMT

A colleague gave me some URLs to use for testing IDN. I don't vouch for any of these sites, and some are obvious ads and some may be inappropriate (since I can't read them), but maybe they'll be helpful for testing.

Additionally some sites could be phishing, spoofing or have malicious content, so please use this list carefully.

äöüß.com	xn--ss-uia6e4a.com	IDN Tool in German
visegrád.com	xn--visegrd-mwa.com	Hungarian
házipatika.com/	xn--hzipatika-01a.com/	Hungarian
айкидо.com/	xn--80aildf0a.com/	Bulgarian
www.خداوند.com/	www.xn--mgbndb8il.com/	Farsi
www.çukurova.com	www.xn--ukurova-txa.com	Turkish
nixieröhre.nixieclock-tube.com	xn--nixierhre-57a.nixieclock-tube.com	German
caxap.ru	caxap.ru	homograph in dotRu
игроshop.com/	xn--shop-k4d3a7bp.com/	mixed script in dotCom
北京2008旅行.com	xn--2008-4x5f27utu0bmz1d.com
רישוםב99סנט.com	xn--99-xldpsb3a1ai7dm.com
महरोत्रा.com	xn--h2btgb6b7a7fra.com
தேடல்.com	xn--mlcj7bwe6a.com
தேடு.com	xn--mlcj2gwa.com
தமிழ்நாடு.com	xn--mlcjmx4a2deu8h.com
வலைப்பூ.com	xn--xlcawl2e7azb.com
வலைப்பதிவு.com	xn--rlcla4aoe3er1d9b.com
österreich.ac/	xn--sterreich-z7a.ac/
www.münchhausen.at/	www.xn--mnchhausen-9db.at/
kernöl.cc/	xn--kernl-mua.cc/
www.rêve.ch/	www.xn--rve-fma.ch/
www.hypermarchés.ch/	www.xn--hypermarchs-kbb.ch/
www.rüüsbier.ch/	www.xn--rsbier-3yaa.ch/
山东大学.cn	xn--xhq02ykwbp4a.cn
天蓝蓝.cn	xn--rssy03ha.cn
腊味.cn/	xn--btr765h.cn/
无线扩音机.cn/	xn--fquz0fy3ac29c041a.cn/
时代互联.cn/	xn--blq9g996dr9x.cn/
www.bärmasens.de/	www.xn--brmasens-0za.de/
www.querschnittlähmung.de/	www.xn--querschnittlhmung-1qb.de/
www.düsseldorf-airport.de/	www.xn--dsseldorf-airport-22b.de/
www.möhr.de/	www.xn--mhr-sna.de/
www.travemünde.de/	www.xn--travemnde-v9a.de/
golf.ausrüstung.de	golf.xn--ausrstung-t9a.de
www.hügö.de/	www.xn--hg-gkaw.de/
gratis-klingeltöne.2t2.de/	xn--gratis-klingeltne-e0b.2t2.de/
köln.studenten-wohnung.de/	xn--kln-sna.studenten-wohnung.de/
www.polizeipuppenbühne.de/	www.xn--polizeipuppenbhne-g3b.de/
www.vermögensberatung-berlin.de/	www.xn--vermgensberatung-berlin-blc.de/
berufsunfähigkeitsversicherung.easy-worxx.de/	xn--berufsunfhigkeitsversicherung-8pc.easy-worxx.de/
www.printout-für-experten.de/	www.xn--printout-fr-experten-yec.de/
hansjørn.dk/	xn--hansjrn-u1a.dk/
æøå.dk-hostmaster.dk/	xn--5cab8c.dk-hostmaster.dk/
www.møgelbjerg.dk/	www.xn--mgelbjerg-l8a.dk/
www.lyø-ølejr.dk/	www.xn--ly-lejr-r1ab.dk/
www.sää.fi/	www.xn--s-0faa.fi/
γλωσσολογια.gr/	xn--mxadayia1ab7aa8d.gr/
www.biergläser.info/	www.xn--bierglser-02a.info/
www.dachgepäckträger.info/	www.xn--dachgepcktrger-cibe.info/
€.linux.it	xn--lzg.linux.it
味の素.jp/	xn--u9j479hu21a.jp/
セガ.jp	xn--mck3a.jp
倉敷.jp	xn--0vq553c.jp
フジテレビ.jp	xn--yck2a1bf1j.jp
角川書店.jp	xn--5rt1pr1svu1b.jp
東京理科大学.jp	xn--1lq68wkwbj6ugkpigi.jp
産経新聞.jp	xn--efvu75ac8f49c.jp
じゃらん.jp	xn--78j9dsa4b.jp
정통부.kr	xn--or3bn7qhwh.kr
한글.kr	xn--bj0bj06e.kr
이효리.kr	xn--oy2b15wvzl.kr
정보통신부.kr	xn--on3b2jr1lk6fb1n.kr
alėja.lt/	xn--alja-wva.lt/
tūdaliņ.lv/	xn--tdali-d8a8w.lv/
blåbärsbröd.idn.museum/	xn--blbrsbrd-2zai7q.idn.museum/
www.nårsk.no/	www.xn--nrsk-qoa.no/
www.dødeladen.no/	www.xn--ddeladen-54a.no/
www.blæst.no/	www.xn--blst-woa.no/
www.blåbærmuffin.no/	www.xn--blbrmuffin-25an.no/
www.hovedløpet2006.no	www.xn--hovedlpet2006-gnb.no
www.futtitromsø.no/	www.xn--futtitroms-9cb.no/
alliancefrançaise.nu	xn--alliancefranaise-npb.nu
tobiasvärk.nu/	xn--tobiasvrk-12a.nu/
www.färgbolaget.nu/	www.xn--frgbolaget-q5a.nu/
högkyrklig.nu/	xn--hgkyrklig-07a.nu/
żółć.pl	xn--kda4b0koi.pl
www.noże.pl/	www.xn--noe-42a.pl/
www.szkołagłównahandlowa.pl/	www.xn--szkoagwnahandlowa-lyb21mca.pl/
кремль.ru/	xn--e1ajeds9e.ru/
президент.ru/	xn--d1abbgf6aiiy.ru/
www.företagsbörsen.se/	www.xn--fretagsbrsen-4ibh.se/
blåbärsmjölk.webway.se/	xn--blbrsmjlk-x2aj4s.webway.se/
www.uplandskonstförening.se/	www.xn--uplandskonstfrening-26b.se/
ไดอารี่.th/	xn--l3c4a3auq9f7a.th/
www.türkçe.gen.tr	www.xn--trke-2oa7j.gen.tr
jæger.tv/	xn--jger-voa.tv/
中央大学.tw	xn--fiq80yua78t.tw
台網中心.tw	xn--fiq43lrrlz83a.tw
中文.tw	xn--fiq228c.tw
大大.tw	xn--pssa.tw
清華大學.tw	xn--pssu7c921afvu.tw
網域名稱.tw	xn--eqrt2ge74bp6c.tw
東華大學.tw	xn--pssu7cxyrvv2a.tw
zürich.ws/	xn--zrich-kva.ws/

Does your web site die when it sees unexpected languages?

shawnste — Fri, 01 Sep 2006 11:00:00 GMT

Recently I've seen a spate of bugs involving client-server web applications related to the user locale.

Windows Vista has added a ton of locales that weren't there in XP or Server 2003 or Microsoft.Net v2.0. Additionally users can create and use their own custom locales. Lastly, they've always been able to edit their language preference(s) in IE.

It seems that when some web sites get a language/region pair that their OS doesn't support in the HTTP_ACCEPT_LANGUAGE headers they crash. The way to solve this is to make sure that your application has a fallback path for unknown languages (like to en-US or invariant or some other appropriate locale).

To test that your web site can handle unexpected client locales, you can try some of the new languages in Vista, such as Greenlandic (Greenland), or make your own custom locale, like tlh-US or haw-US or fj-FJ. Even editing IE's language preferences may provide test data to your servers.

Currently users rarely change IE's language preference, but we expect that with Vista and in the future, more users will be choosing locales (cultures) that are more appropriate for them. So it will only become more likely that your server will encounter language tags it doesn't recognize.

- Shawn

Does "Everyone" Use IDN?

shawnste — Fri, 25 Aug 2006 21:40:00 GMT

Recently it came to my attention that arbitrary implementation of IDN can break existing networks.

Particularly on Intranets machine names have not always been restricted to ASCII. In many of our markets it is quite common to have non-ASCII machine names.

So in some cases there are existing deployments that use ANSI or UTF-8 machine names directly without relying on Punycode. Internal DNS services often return these names. It is also possible (but not of much use and very much not recommended) to have sub-domains (like xxx in xxx.example.com) that respond to 8-bit encoded domain names on the Internet itself.

How Come IdnToAscii Fails For String XXXX?

shawnste — Thu, 24 Aug 2006 21:11:00 GMT

I'm often asked why particular strings fail the IdnToAscii function. The answer is "because its not a legal IDN name, that's why" :-) But why aren't some strings legal IDN names?

Some rules act on the "label" level. In www.microsoft.com www, microsoft & com are each separate labels.

IDN prohibits some characters, such as space characters, control characters, etc.
IDN is currently at Unicode 3.2, characters added after that point are prohibited. Eventually that'll probably be upgraded to Unicode 5 or later, but currently many issues regarding security, etc are prohibiting it.
Our APIs also disallow the ASCII control and Backspace characters even if the IDN_USE_STD3_ASCII_RULES flag is not set since control characters don't make sense in domain names. 0 in particular is bad in strings.
There are rules regarding mixing bidirectional characters. Basically if there are any right to left (RTL) characters, then there may not be any left to right (LTR) characters in the same label. Those characters that aren't specifically RTL or LTR can be mixed with RTL or LTR characters.
If the label has RTL characters, the first and last characters must be RTL as well. Other characters that aren't explicitly RTL or LTR may be in the string, but not in the first or last positions.
Punycode labels are longer than their source Unicode strings. So some Unicode strings will convert to Punycode labels that exceed the allowable DNS label or name length. Those will therefore be illegal as well.

For more information about IDN, you can look at the RFCs and sources:

http://www.ietf.org/rfc/rfc3490.txt Internationalizing Domain Names in Applications (IDNA)
http://www.ietf.org/rfc/rfc3491.txt Nameprep: A Stringprep Profile for
Internationalized Domain Names (IDN)
http://www.ietf.org/rfc/rfc3454.txt Preparation of Internationalized Strings ("stringprep")
http://www.ietf.org/rfc/rfc3492.txt Punycode: A Bootstring encoding of Unicode
for Internationalized Domain Names in Applications (IDNA)
http://en.wikipedia.org/wiki/Internationalizing_Domain_Names_in_Applications Wikipedia's entry on IDN.

IDN Mitigation API dll needs to be explicitly linked

shawnste — Thu, 05 Jan 2006 20:52:00 GMT

Since we (I) didn't provide a .lib file for the "Microsoft Internationalized Domain Name (IDN) Mitigation APIs 1.0", you'll have to explicitly link to the IDN Mitigation API dll, which means you'll have to make your own header & code something like:

[5 May 2009 - updated with Siva's comment, but didn't actually try to compile it]

// Declare the stuff to use (repeat for each function/dll to import)
typedef int (__stdcall *PFN_DOWNLEVELGETLOCALESCRIPTS)
    (LPCWSTR lpLocaleName, LPWSTR lpLCData, int cchData);

PFN_DOWNLEVELGETLOCALESCRIPTSm_pfnGetLocaleScripts = NULL;
HMODULE m_hDownlevelDll = NULL;

...
// Load the library and get our function pointer
m_hDownlevelDll = LoadLibrary(L"idndl.dll");
if (m_hDownlevelDll != NULL)
{
    FARPROC pfn = GetProcAddress(m_hDownlevelDll, "DownlevelGetLocaleScripts");
    if (pfn != NULL)
    {
        m_pfnGetLocaleScripts = (PFN_DOWNLEVELGETLOCALESCRIPTS)pfn;
    }
    // else error
}
// else error
...

// Call the function using our pointer
int count = (*pfnGetLocaleScripts)( L"en-US", NULL, 0 );

Hope that helps, we'll probably add the .lib in a future update, but that'll probably be a while.

Microsoft Internationalized Domain Names (IDN) Mitigation APIs 1.0 Released

shawnste — Wed, 10 Aug 2005 01:02:00 GMT

Today we posted some APIs that convert to & from IDN Punycode, provide some mitigation functionality for those APIs, and provide Unicode Normalization support.

http://www.microsoft.com/downloads/details.aspx?FamilyId=E289BB2C-D111-4331-8FB2-CC6C5A026C93&displaylang=en

See the documentation included with the download for details, but this could help apps that need IDN or normalization functionality. Managed apps should of course use the .Net v2.0 String.Normalize() and IdnMapping classes :-)

5 Jan 2006 - See http://blogs.msdn.com/shawnste/archive/2006/01/05/509746.aspx for notes on loading the dll.

IDN & Homographs

shawnste — Fri, 04 Mar 2005 00:10:00 GMT

I’ve divided this into a few parts:

About IDN
IDN & Security
Homograph Thoughts
Conclusion

About IDN:

My interest in IDN is that I’m the SDE for the System.Globalization.IdnMapping class in Whidbey. I also think its pretty nifty for the users in countries that use more than the basic Latin letters.

For those of you that don’t know, IDN/IDNA is trying to solve the problem of international (non-ASCII) characters in domain names. IDN is an “Internationalized Domain Name”. RFC 3490 - Internationalizing Domain Names in Applications http://www.faqs.org/rfcs/rfc3490.html has the details. IDN only addresses domain names, it doesn’t attack the email address user name issue or other internationalization issues related to URLs/URIs/IRIs.

Before IDN, domain names were basically restricted to the Latin character set, A-Z, 0-9 and sometimes -. This is useful if your company is Microsoft.com, but not so helpful if you’re company is in Chinese or Cyrillic characters. IDN provides a mechanism for encoding additional Unicode characters using the allowed a-z, 0-9 and - characters. So a name like きくどら.com (Kikuna Driving School) or www.mäkitorppa.com (Mäkitorppa mobile store) is represented like xn--w8je2f2f.com (きくどら) or www.xn--mkitorppa-v2a.com (www.mäkitorppa.com).

So IDN doesn’t require any changes to the DNS layers of the Internet, but it does require conversion from the Unicode to the ASCII “Punycode” form of a name at some point. A Whidbey .Net application uses the System.Globalization.IdnMapping class to convert between the Unicode and “Punycode” forms.

In addition to the punycode conversion, IDN does some normalization using NFKC and additional mappings such as making the strings all the same case. Some Unicode characters are considered ambiguous or dangerous and are disallowed in IDN, others are folded into a more common form to prevent some repetition.

IDN & Security:

IDN disallows some Unicode characters considered dangerous and “folds” others into a more common form in some cases if they are ambiguous.

Even with these restrictions, it was quite obvious that many look alike characters, or homographs, exist in Unicode. Examples exist even in ASCII as you can construct MICROSOFT.com as MlCR0S0FT.com by using the little el and zero characters. The DNS system would think this is a different domain name and send a user to a different server, yet, depending on your font it could be difficult to distinguish from the real domain name.

Unicode has tens of thousands of characters, so when the IDN RFC was created it was the homograph problem becomes even more complicated when Unicode characters are allowed. For example, Місrоѕоft.com can be written almost entirely in Cyrillic letters (this example has only the r, f & t in Latin. Я just doesn’t look quite the same ;-)).

Even worse, some scripts have characters that are difficult to distinguish. Many Chinese characters appear very similar in small fonts. Other scripts have minor diacritics that could be missing or slightly modified such that the user might not notice. Due to the complexity of the problem, the IDN RFCs leave the homograph problem to be resolved later, perhaps by the registrars or a future RFC.

Since IDN doesn’t directly address the homograph problem, users could be susceptible to spoofing, phishing and other social engineering attacks. This is exactly what happened with the recent paypal attack. The IDN name pаypal.com was registered with a Cyrillic a for the first A. xn--pypal-4ve.com is the punycode version of this name.

A user following such a link in some browsers would see what looked like paypal.com in their address bar, but would actually be a different web site. An email or link from another web site could be used to trick a user into providing their paypal information to an attacker. This type of attack is similar to the socially engineered emails that have already been used to try to get users to enter personal information by trying to get them to go to https://safe-com.com/ebay or some such URL instead of a real vendor site.

Some people were amused that Mozilla, Firefox and other browsers were susceptible to the pаypal.com homograph attack, but Internet Explorer is not (because it doesn’t do IDN conversion). Equally interesting is the browser reaction of removing IDN support and then choosing to display the Punycode name instead.

Personally I don’t think this is just an IDN weakness. Rather IDN merely makes an existing problem with trusting links more obvious. http://paypal-safe.com or http://secure.com/paypal would catch many users anyway.

Homograph Thoughts:

My thinking is that basically the IDN pаypal.com attack where the first A is Cyrillic is a social attack. For this attack to succeed, a user must first follow an untrusted link. That link could be a web site (please buy my book at Amazon.com) or an email (click here to update your bank information). Some users are already wary of following unsolicited links from email, but don’t think twice about a web link. In either case, a look-alike name in the address bar of the browser would be reassuring.

Several solutions to the homograph problem have been suggested. I don’t have a magic bullet, but I think that the root problem is a user education/social engineering problem, remember in some cases these attacks can even happen with the non-IDN DNS names. The following are suggestions I’ve seen in various places and my thoughts about them. Other people & coworkers disagree, so these are merely my thoughts.

Several suggestions seem American centric, and I’m disappointed that the developer community doesn’t have a broader global perspective.

· Disabling IDN – This is perhaps the most obvious suggestion, but seems quite short sighted. After all IDN was created to solve a real problem for DNS names that are not ASCII. If your corporate name is きくどら this “solution” doesn’t help at all.

· Displaying Punycode – This is also a quite non-global suggestion. While showing punycode solves the particular pаypal.com problem, its even worse for the きくどら user. xn--w8je2f2f.com is the same as xn—gibberish.com to their users (xn—gibberish.com would be ٮ٨٧٩ٯٲٳ.com, cool it even decodes!) Even in the www.mäkitorppa.com case how is a user supposed to know if its www.xn--mkitorppa-v2a.com or www.xn--mkitorppa-01a.com? For non-US users displaying the punicode doesn’t solve the problem, it makes it worse.

There are many other suggestions though. My concerns with these suggestions are that either a) they are too restrictive, preventing reasonable names, or b) they aren’t sufficient to catch all of the problems, or both.

Registrars Should Prohibit Bad Scripts – The idea is that since certain countries would only expect names in their language(s), only those scripts should be allowed. Some registrars are doing this and it seems reasonable, however since “everyone” uses .com and the other well known top level domains, this solution is pretty incomplete. I also wonder if this might be a bit too aggressive. What about a Chinese grocery in Germany? It seems reasonable that they might want a Chinese character .de domain name. Fortunately for them (but not for .com), they can currently fall back to .com in this case J

Disallowing Mixed Scripts – The thought is that since paypal was spelled with mixed Cyrillic and Latin, then disallowing such mixes should prevent such attacks. Unfortunately Latin is often allowed with other scripts, particularly for businesses with tech names where it is sometimes popular to pick up words in Latin scripts. Variations of this suggestion include disallowing mixed scripts, except that Latin could appear with most scripts, except Cyrillic. I question however whether this is accurate or not. I can easily imagine an import-export company with a multi-script-but-not-Latin name. I’ve also heard that even Cyrillic can rarely be used with Latin, although I don’t know how the user’s supposed to type a mixed Cyrillic/Latin name.

Disallow Non-User Scripts – Check the user locale or keyboard or whatever to see if the URL is a script the user uses. Many users however have 2nd language skills that aren’t necessarily represented by their current locale or keyboard choice.

Prohibiting or Normalizing Homographs (applications) – This is an interesting suggestion, but I have several concerns. Which form is “correct” if a browser encounters a Cyrillic and a Latin version of the same URL? I’d imagine that this happens often. Who is going to find out which ones look alike? Using which fonts? I cannot imagine that all Homographs could be caught. There’s probably a billion combinations to check, and we (the developer community) would only catch the obvious ones. The “bad guys” would undoubtedly catch the one(s) we missed. This also doesn’t help with some subtle character differences, even í, i, ì, and ï look pretty close and its pretty obvious that í and ì can’t be rejected just because they look like i.

Certificates Are The Solution – This has the same problem as trying to get the Registrars to do it, although since the fees are higher they might have more resources to address the problem. Again only 1 CA would have to let a bad name pass through. Additionally despite the little key icon I suspect many users don’t realize if they’re on a secure connection or not.

IDN APIs Should Filter Bad Names – If homograph filters are implemented, the RFC, Registrars or Unicode should drive them. Obviously different software vendors can’t have different filtering or else those names would break, and I have difficulty suggesting that a registered name cannot be accessed by a software program.

Users Should Retype URLs – This seems a bit harsh. Personally I only care if it’s a “safe” URL if I’m about to enter personal data. Many URLs are too long and this would also be bad for those people that get credit for referrals to Amazon and the like.

Display URLs in Lower Case – Some have suggested that lower case letters are less likely to be confused, however there are still cases like rn vs m. (R + N vs M), and í, i, & ì that look very similar. Font choice could help, but probably can’t solve the problem, particularly for some users.

Some suggestions seem more reasonable to me. (Of course, I’m just me, so other people probably have other ideas).

Registrars Should Prohibit Homographs – The UTC is working on guidelines to help registrars with this problem (UTR #36 Security Considerations for the Implementation of Unicode and Related Technology). This seems quite reasonable if consistently implemented, but it still seems likely that some combinations will still slip through, like using rn instead of m.

Whitelisting – This would be some sort of mechanism for trusting sites. I like this because it would address IP addresses in phishing mail, URLs with keywords and other attacks. The idea is that if the user tries to enter a form on a new site then they’d be prompted before the action was allowed. If the site was already whitelisted no prompt would appear. So a user that used paypal.com would have some warning if they followed an e-mail link to http://1.2.3.4/paypal.com or http://secure.com/paypal. The downside is that determining which forms require extra protection could be hard. Additionally kiosks like those in hotels or libraries couldn’t persist the whitelist or else someone could intentionally whitelist an unsafe site to trap future users.

User Education – At some level user’s need to be aware of what behaviors are safe or unsafe. Unfortunately there are a LOT of users, so even a fraction of a percent that remain unaware of these issues would still be at risk. This could also address problems such as users who use the same name & password for every site they register. One bogus contest entry and they could be at risk if the same credentials worked on paypal.

Conclusion:

Various technical problems, including IDN, can be combined with well-designed social attacks to allow a user to trust a web site that shouldn’t really be trusted. Vigilance by the registrars could prevent many homograph attacks, however some will undoubtedly still be possible. Font choices and browser behavior might limit a few more mistakes, but those can be offset by poor eyesight, dyslexia, monitor differences, color choices (user or application), platform (mobile or PC) and other differences. User education can also help catch a percentage of the problem.

It seems to me that all of these taken together will reduce the available surface for attacks, but there will still be a window for the attackers to attempt their exploits. Many of those socially engineered attacks don’t even require IDN Homographs to trap some unwary or uneducated users.