Smart quotes: The hidden scourge of text meant for computer consumption

re: Smart quotes: The hidden scourge of text meant for computer consumption

DWalker — Wed, 04 Mar 2009 01:57:57 GMT

@ulric and others: I was pasting a short piece of SQL code into an e-mail message using Outlook 2007, today, and since my message format is HTML by default, and uses Word as the editor, I had a heck of a time getting the quotes in the SQL to stay as straight quotes.

I could probably change the default for Word when it's used as the e-mail editor and not when I'm creating a regular Word document, but I don't happen to know how to change one setting without the other.

I occasionally need to underline something or make it RED and BOLD in an e-mail or risk having it overlooked; otherwise, I would use plain text e-mails.

re: Smart quotes: The hidden scourge of text meant for computer consumption

SuperKoko — Tue, 03 Mar 2009 10:41:49 GMT

> There's no reason that you couldn't do the same to escape curly quotes.

Actually, it's easier to do it for curly quotes than for normal double quotes, because existing programs are unaware of curly quotes, and so, accept them as normal command line input.

Usually, double quotes can be used to include space characters in a command line parameter.

some_command "hello world"

However, I'm aware of at least TWO 32 bits hexadecimal editors not supporting this feature, making it IMPOSSIBLE to open files with space characters in their path, through the command line.

On the topic of smart quotes:

They're bad on the Web, because of shared responsibilities involving buggy Web browsers, Web servers and Web scripts.

Typically:

A normal person copy text in Word and paste it in a text form on a Web page, including smart quotes.

The page is encoded in ISO-8859-1, and so, the Web browser figures that it must send a POST request with charset=ISO-8859-1.

Since the buggy browser thinks that ISO-8859-1 = Windows-1252, it sends code points used for control characters in ISO-8859-1 (between 0x80 and 0xA0).

The buggy Web server doesn't validate input and store this buggy data in its data base, believing it's valid ISO-8859-1.

Then, if the POST data appears on a Web page (e.g. a blog comment), it serves the page as ISO-8859-1 or UTF-8. At best, with ISO-8859-1, it serves Windows-1252 and claims it to be ISO-8859-1, which is ok with buggy Web browsers believing that ISO-8859-1 = Windows-1252. For UTF-8, it might mess up the thing, either deleting the characters, or not translating them (sending sort of Windows-UTF-8), or, if we're lucky, properly translating them to UTF-8.

Things are so messy that, HTML5 actually REQUIRES Web browsers to assume that ISO-8859-1 = Windows-1252.

<http://www.whatwg.org/specs/web-apps/current-work/#character-encodings-0>

Whether the WHATWG can, cannot, should or shouldn't require HTML clients to violate HTTP and ISO-8859 standards is another topic.

re: Smart quotes: The hidden scourge of text meant for computer consumption

Unicode General Punctuation — Mon, 02 Mar 2009 23:54:40 GMT

Nevermind, maybe. I think I figured it out.

By "extended ASCII range", he's referring to the upper half of a codepage like 1252, and stating that it is "completely incompatible with UTF-8" because you have to go through the effort of actually /mapping/ the codepoints.

re: Smart quotes: The hidden scourge of text meant for computer consumption

Unicode General Punctuation — Mon, 02 Mar 2009 20:04:28 GMT

Not going to discuss the pros and cons of smart-quotes, but I am confused by Dean's comment that they use "an extended ASCII range that makes it completely incompatible with UTF-8."

Unless I'm mistaken, we're talking about characters in the U+2000 range, but UTF-8 can easily encode most of U+0000 to U+10FFFF, even restricted as it is by RFC 3629.

re: Smart quotes: The hidden scourge of text meant for computer consumption

Yuhong Bao — Sun, 01 Mar 2009 23:56:59 GMT

Daniel Earwicker: In fact Raymond devoted an entire article to exactly this:

http://blogs.msdn.com/oldnewthing/archive/2008/08/19/8877486.aspx

re: Smart quotes: The hidden scourge of text meant for computer consumption

Bulletmagnet — Sat, 28 Feb 2009 12:48:24 GMT

There is a solution: use the demoronizer!

re: Smart quotes: The hidden scourge of text meant for computer consumption

Stephen Jones — Fri, 27 Feb 2009 20:41:21 GMT

I once got send code for a SQL select. Very simple because I'm not a programmer.

It was only finally when I sent the .mdb with the data stripped out that we realized why it worked for him but not for me. Yup, he had Word as his email editor.

I've lost count of the times I've read about people complaining some forum software doesn't show quotes. I tell them to cut and paste into the text box from Notebook not Word, but they still say it's the newspaper/forum software that is wrong.

(Another common complaint is how the discussion forum software has a lousy American spell check. The forum software doesn't have a spell check at all of course, it's the default spell check that comes with Firefox (which incidentally has stopped working for the last two or three updates).

re: Smart quotes: The hidden scourge of text meant for computer consumption

Doug Kavendek — Fri, 27 Feb 2009 18:40:43 GMT

At least it's easy enough to keep yourself from making that mistake. A while back I made a really simple program that just reloads whatever is in the clipboard as plaintext, and it corrects for all those curved quotes, em-dashes, etc., as well as stripping out all the formatting nonsense. Bind it to a global key, like Ctrl-Shift-V and just do that before pasting. I use it all the time now; way easier than looking for paste-special menu items or remembering how each application deals with rich text.

If anyone's interested:

http://www.kavendek.com/stuff/SimplifyClipboard.cpp

http://www.kavendek.com/stuff/SimplifyClipboard.exe

re: Smart quotes: The hidden scourge of text meant for computer consumption

Technage — Fri, 27 Feb 2009 03:50:14 GMT

It's like seeing the insides of a television and being amazed at the little intricacies especially the ones that shock you when you touch them.

re: Smart quotes: The hidden scourge of text meant for computer consumption

Daniel Earwicker — Thu, 26 Feb 2009 18:01:23 GMT

Somewhat similar irritation - people who refuse to email me a plain bitmap. They insist on pasting it into a Word document and sending that instead. This is of course extremely helpful because otherwise they wouldn't be able to introduce me to their collection of macro viruses.