When is a backslash not a backslash?

re: When is a backslash not a backslash?

Chris Lundie — Sat, 17 Sep 2005 13:07:32 GMT

Interesting and confusing! I fired up Word, changed the font to MS Gothic, typed a backslash and indeed it displays a Yen symbol. Changing back to Times, it looks like a backslash again.

re: When is a backslash not a backslash?

Michael S. Kaplan — Sat, 17 Sep 2005 16:59:47 GMT

Hi Chris -- well, if nothing else, it helps give confidence in the idea that the actual code point value does not change, even if the representative glyph does....

re: When is a backslash not a backslash?

Ben Bryant — Sat, 17 Sep 2005 18:50:47 GMT

So when you convert a string like "Yen 100" (where Yen is the symbol) from Shift-JIS to Unicode, the Yen becomes a backslash, and the fact that it was a Yen is unknown. But most users are not the wiser because their font shows the backslash as a Yen. This is bad! I suppose most Japanese Unicode databases must have some major backslash Yen confusion where you might have to implement a policy across unrepaired data like if you know the string is a file path treat U+005c as a backslash, otherwise as a Yen U+20a9. And Unicode string functions that search for the Yen probably are sometimes implemented due to user demand to cheat and look for the backslash too. Crazy stuff.
Btw, the Korean ks_c_5601-1987 has an encoding for the Yen, but Japanese Shift-JIS doesn't have one for the Won.

re: When is a backslash not a backslash?

Michael S. Kaplan — Sat, 17 Sep 2005 19:06:22 GMT

Hi Ben -- Although I admit the situation is not ideal, I think it is pretty obvious that the situation would be a *lot* worse if there were no path separators since that would pretty much destroy a lot more.

And it was not really up to anyone but JIS to decide what should be done in the Japanese encoding.

Though if you are using Unicode then you can use the actual Yen character and call it a day! :-)

re: When is a backslash not a backslash?

Ben Bryant — Sat, 17 Sep 2005 19:39:29 GMT

Thanks for the quick reply. But you almost answer as if I was blaming you or anyone. No, I agree the right choice was made in the JIS to Unicode conversion; I am just chiming in on the fact that it is a problematic situation. I do take comfort in the fact that the code point 5c is not changed. Anyway, encoding systems are full of these ideosyncracies, but the strange thing about this one is that it involves fonts. So many layers of complexity: a character set can be represented in different encodings and displayed differently by different fonts, what fun!

re: When is a backslash not a backslash?

Michael S. Kaplan — Sat, 17 Sep 2005 19:57:49 GMT

No worries, Ben -- I was not taking it as blame or anything. :-)

re: When is a backslash not a backslash?

Claus Brod — Wed, 21 Sep 2005 17:54:50 GMT

Thanks for the interesting post. I've come across another slightly surprising conversion - the tilde character (0x7e) is mapped 1:1 when converting from CP932 to, say, UTF16 (MultiByteToWideChar). However, when I use libiconv to convert a tilde character, and tell it that the source encoding is "SJIS", it will map the tilde to U+203E. I found a few slightly mysterious references to this behavior, but nothing that would really explain the reasoning behind this mapping...

Claus

re: When is a backslash not a backslash?

Steve Loughran — Fri, 14 Oct 2005 17:29:05 GMT

This is fascinating. Of course, the whole fact that DOS-derived platforms use \ as a dir separator is iself a bit of a mess -I've always assumed it was because DOS 1.x used / as an argument prefix, so when directories came along in 2.0, they had to use a different char. This is just another unintended consequence of the first, well, error.

re: When is a backslash not a backslash?

Rune — Fri, 14 Oct 2005 17:40:38 GMT

IIRC, the backslash maps to "Ø" using a norwegian codepage with 7-bit ASCII. Later (8-bit codepage 865) the Ø moved to the same spot normally inhabited by the Yen sign (american cp 437).

So a backslash is probably a lot of things around the world, historically speaking.

--
Rune

Paths

Craig Ringer — Fri, 14 Oct 2005 20:14:32 GMT

Quite frankly, these days I begin to think that having common characters as path and argument separators is anachronistic and pretty nasty. Any possible choice from the standard character set will be largely arbitrary and will impair "legitimate" use of those characters. It's necessary due to all the legacy code out there and it'd be an unpleasant thing to try to change to dedicated symbols, but that doesn't make it any nicer.

Nonetheless, I still find it annoying that \ (or / on UNIX, or : on Mac OS < 10) are "special" to the system and can't be used in filenames. Similarly the need to quote "multi word arguments" seems silly these days. Dedicated delimeters just for those two purposes, and nothing else, would seem a much nicer way to do it if we got the option to do it all again.

Alas, it's unlikely to ever happen. We'll still want to use 7-bit ASCII serial terminals, still want to use ancient systems that don't understand utf-8 or UCS-2, and so on. Anyway, I swear every time I have to use \these\silly\paths , just as I'm sure many folks here find /these/paths/very/annoying ; getting used to someːotherːpathːseparator (obviously not actually ː , I just use that as an example) would be pretty irritating. Especially having to use some sort of compose sequence or shortcut to type it on "legacy" keyboards...

re: When is a backslash not a backslash?

andypennell — Fri, 14 Oct 2005 21:02:52 GMT

Why was \ the dir separator in DOS? A friend of a friend once bought a big wooden desk from a sale on Microsoft campus in the early 90s. The desk had not been cleared out: inside it was a pice of paper containing discussion notes on which separator should be used. IBM was mentioned, but I cannot recall the rest of the details.

re: When is a backslash not a backslash?

mikeb — Fri, 14 Oct 2005 21:47:03 GMT

As far as the '\' character being used as the path separator in DOS (starting with DOS 2.0, since 1.x did not support sub directories), I would have to agree with Steve Loughran that the fact that DOS commands used '/' as a command line option 'switch character' is probably the number 1 reason.

Remember that internally and at the API level DOS supports using either '/' or '\' as a path separator (I'm not sure if this agnosticism goes all the way back to DOS 2.0) - it's applications that don't like '/'.

Also remember that early versions of DOS supported setting the 'switch character' to something other than '/'. Unfortunately each DOS application is responsible for parsing it's command line, and virtually no 3rd party applications supported the switch character setting (Microsoft applications may have been guilty of this, too).

At some point MS removed the set switch char API - a *very* rare thing for Microsoft to do (just ask Raymond Chen). I mean, the old CP/M compatibility DOS calls are still supported even in the WinXP VDM.

re: When is a backslash not a backslash?

Dewi Morgan — Sat, 15 Oct 2005 02:06:23 GMT

Mikeb points out that the problem with path parsing is that it depends on the applications to support it.

It seems strange that people aren't offered a choice of visual style nowadays, though, since 4c is the character that applications expect.

Just need a little checkbox in windows versions affected by the 0x4c issue:

[ ] Display path separator as '\'

So that wherever 4c was displayed, a '\' would be shown instead of a currency symbol.

re: When is a backslash not a backslash?

Michael S. Kaplan — Sat, 15 Oct 2005 02:37:21 GMT

I think you meant 0x5c/U+005c, right? :-)

You do have a choice -- this only happens with CJK fonts....

re: When is a backslash not a backslash?

foxyshadis — Sat, 15 Oct 2005 06:07:53 GMT

I suppose you could use a tool (eg, fontographer or fontforge) to replace 0x5c in your favorite fonts (tahoma, ms sans serif, maybe times, arial, and verdana) with your favorite path separator. Oh, and paint over your keyboard's \ key. That would be an amusing extension of 'skinning' the OS. =p

re: When is a backslash not a backslash?

koji — Sat, 15 Oct 2005 15:19:11 GMT

IIRC, if you go back a little further, ASCII defines 0x5c as one of the "localizable" code point, and that is why several countries have several different glyphs here.

DOS 2 made a mistake by choosing such a localizable code point as the path separator. Well, I don't think anyone can blame on it though.

Whether we should fix this glyph or not is as good open question as whether we should fix the path separator to "/", which is not a localizable code point in ASCII.

And, although both are good questions, I don't think anyone could fix either.

On the fuzzier definition of a 'Unicode application' on Win9x....

Sorting It All Out — Fri, 17 Mar 2006 16:12:31 GMT

Riffing on Raymond here, and his post On the fuzzy definition of a "Unicode application"....
The points...

Two chickens in every pot, and an ASCII in every code page

Sorting It All Out — Fri, 26 May 2006 22:04:10 GMT

Francisco Moraes asked in the Suggestion Box:

Are there any code pages (exception EBCDIC) where the...

A yen for Yen may be left unsatiated

Sorting It All Out — Wed, 28 Mar 2007 10:15:35 GMT

(No, this post is not about anything kinky) If you are a regular reader then you know that I have been

Trying to get people to use Unicode? Lock and load, baby!

Sorting It All Out — Thu, 18 Oct 2007 22:47:07 GMT

One of the more fascinating conversations I had with a customer at the recent Internationalization and

re: When is a backslash not a backslash?

not given — Mon, 12 Nov 2007 02:02:04 GMT

Was a work around ever found for this issue? Is there a way to keep Japanese language support in Windows and have the backslash ( \ ) display correctly in address fields instead of the yen symbol?

It works correctly when typing in fourms, but not in the address fields? Why is this?

re: When is a backslash not a backslash?

Michael S. Kaplan — Mon, 12 Nov 2007 02:09:52 GMT

It is all about the font selected and sometimes the technology doing the rendering -- and for every person who considers one behavior to be a bug, there is another who thinks the other is a bug.

Which essentially makes it unfixable, at least for everyone....

re: When is a backslash not a backslash?

Mike — Sun, 06 Jan 2008 02:26:11 GMT

I got the same thing, after playing Clannad the primary fonts seem to be MS Gothic lol, yet it still types as \ here.

http://en.wikipedia.org/wiki/backslash

TrackBack — Thu, 14 Feb 2008 18:42:25 GMT

re: When is a backslash not a backslash?

spitzak — Fri, 23 Jan 2009 06:30:29 GMT

The reason 0x5c still prints a Yen is not because of Windows paths, but because of Windows *text* files. There must be vast numbers of Japanese Windows text files where 0x5c is used *as* a Yen symbol. You can't make them all suddenly display backslash. And there is not likely any intelligent way to figure out whether a Yen or backslash is intended and translate the documents.

re: When is a backslash not a backslash?

Anonymous — Wed, 22 Apr 2009 23:04:29 GMT

I recently set up Japanese language support on my Vista machine - it was a surprise to see cmd.exe rendering paths wrong. My fix? Use xterm from the cygwin toolkit as your terminal. It looks better than the Windows terminal as well.

Also, >>spitzak :

It would be an easy enough fix, actually. Just add another backwards compatibility option "Render \ as ¥ (Use for pre-[year] text documents)" to the menu available when alt-clicking a file. That way, people who needed the old style would be able to use it and the general population would be able to see actual '\' characters where they belong.

re: When is a backslash not a backslash?

Russ — Wed, 10 Jun 2009 01:17:53 GMT

Since '\'s are contained in text files where the codepage isn't defined one way or the other, the only sane thing to do going forward is to define it as a '\' and not a Won or Yen. The currencies differ by a factor of about 10. Since we now regularly send files around this thing called the "Internet", it should not be guessed that it is a Won on Yen. Stop the insanity from going any further? Why wasn't this changed when unicode support was added to Windows?

If the '\' is contained in a document where the codepage is defined, then sure, make it the appropriate Yen or Won symbol and if converted to unicode, save it as the appropriate unicode symbol.

re: When is a backslash not a backslash?

Pat — Thu, 18 Jun 2009 18:14:39 GMT

The last post is totally relevant. This really should be fixed. For people in IT who need to make screenshots of file paths and realated documentation this is a real pain.

re: When is a backslash not a backslash?

Michael S. Kaplan — Thu, 18 Jun 2009 20:33:12 GMT

People in IT who need to make screenshots need to be more respectful of decade long differences in their target markets. And if they want to support customers in other countries they need not insist that the other countries change to make the IT life easier....

Also, people in IT making documentation for Japan and Korea should have the documentation in Japanese and Korean anyway -- if not then their yen/won/solidus issues are *not* why their docs are not appreciated!