Stripping is an interesting job (aka On the meaning of meaningless, aka All Mn characters are non-spacing, but some are more non-spacing than others)

re: Stripping is an interesting job (aka On the meaning of meaningless, aka All Mn characters are non-spacing, but some are more non-spacing than others)

Dean Harding — Tue, 15 May 2007 01:50:25 GMT

The ultimate solution would be to build up your own table of "non-non-spacing marks." Bonus points if you make it locale-specific (so å would be in the English one but not the Swedish). Of course, this may be a case of overengineering ;)

re: Stripping is an interesting job (aka On the meaning of meaningless, aka All Mn characters are non-spacing, but some are more non-spacing than others)

Peter Karlsson — Fri, 08 Jun 2007 10:21:49 GMT

Of course, in the Swedish case, a “proper” RemoveDiacritics("wüéåäöæøà") should produce “vyéåäöäöa”. But I guess implementing that could be a bit more difficult…

año del ano, a.k.a. This sentence has several non-skarklish Spanish flutzpahs....

Sorting It All Out — Fri, 08 Jun 2007 16:42:44 GMT

The primary title of this post is meant to be a warning. One that you probably won't get unless you know

re: Stripping is an interesting job (aka On the meaning of meaningless, aka All Mn characters are non-spacing, but some are more non-spacing than others)

Michael S. Kaplan — Fri, 08 Jun 2007 17:37:19 GMT

Hey Peter -- very good point. I'll have to talk about this soon....

Normalize Wide Shut

Sorting It All Out — Fri, 17 Aug 2007 10:46:38 GMT

(Apologies to Stanley Kubrick, of course!) It was almost the very first blog post I ever wrote, back

I am not a nudist, but I do support stripping when it is appropriate, part 1

Sorting It All Out — Tue, 04 Sep 2007 10:31:06 GMT

The title is correct: I am not a nudist. I think people who are nudists are just fine and I enjoyed the

re: Stripping is an interesting job (aka On the meaning of meaningless, aka All Mn characters are non-spacing, but some are more non-spacing than others)

Steven Sudit — Mon, 19 May 2008 18:34:59 GMT

For another approach to normalization, take a look at: http://www.codeproject.com/KB/cs/UnicodeNormalization.aspx

I'm not sure if it's faster or slower than the method mentioned in this blog, but it is smart enough to remove diacritics only when they're attached to Latin characters, which leaves such things as Han radicals alone.

re: Stripping is an interesting job (aka On the meaning of meaningless, aka All Mn characters are non-spacing, but some are more non-spacing than others)

Jan Kučera — Wed, 29 Oct 2008 03:38:27 GMT

Any way for the .NET Compact Framework people out there? (as the normalization does not seem to be available for them...)

Wonder why something is not in the Compact Framework? The answer is in the question!

Sorting it all Out — Wed, 29 Oct 2008 10:04:29 GMT

Regular reader Jan Kučera, in response to Stripping is an interesting job (aka On the meaning of meaningless,

re: Stripping is an interesting job (aka On the meaning of meaningless, aka All Mn characters are non-spacing, but some are more non-spacing than others)

mkukik — Fri, 31 Oct 2008 12:38:05 GMT

In case of stripping diacritics from Latin characters:

Combining above method with converting Unicode to Western codepage, strips even more diarcritics... Like Polish Łł

However it is still not 100%

string unicodeStringOrig = "SE:ÅåÄäÖö; PL:ĄąĆćĘęŁłŃńÓóŚśŹźŻż; SK:ľščťžýáíéúäôňďĽŠČŤŽÝÁÍÉÚÄÔŇĎ; HU:ëőüűŐÜŰ; ES:Ññ¿; CA:àèòçï";

string unicodeString = RemoveDiacritics(unicodeStringOrig);

Encoding nonunicode = Encoding.GetEncoding(850);

Encoding unicode = Encoding.Unicode;

byte[] unicodeBytes = unicode.GetBytes(unicodeString);

byte[] nonunicodeBytes = Encoding.Convert(unicode, nonunicode, unicodeBytes);

char[] nonunicodeChars = new char[nonunicode.GetCharCount(nonunicodeBytes, 0, nonunicodeBytes.Length)];

nonunicode.GetChars(nonunicodeBytes, 0, nonunicodeBytes.Length, nonunicodeChars, 0);

string nonunicodeString = new string(nonunicodeChars);

re: Stripping is an interesting job (aka On the meaning of meaningless, aka All Mn characters are non-spacing, but some are more non-spacing than others)

Amino — Thu, 17 Dec 2009 09:54:52 GMT

Is it possible to work arround it, meaning addingDiacritics?