Some files come up strange in Notepad

re: Some files come up strange in Notepad

Nate S — Wed, 24 Mar 2004 15:28:00 GMT

Notepad does a good job of detecting these variations, in my experience.

However, it's confusing when Microsoft documentation refers to UCS-2 (or is it UTF-16 now?) as "Unicode". I've seen a lot of people who think that Unicode means "two bytes per character", which isn't even true of UTF-16. UTF-8 and UTF-7 are no less Unicode than UCS-2/UTF-16.

re: Some files come up strange in Notepad

David Cumps — Wed, 24 Mar 2004 16:22:00 GMT

Hey, thanks for the nice explenation!

re: Some files come up strange in Notepad

Mike Dunn — Wed, 24 Mar 2004 16:25:00 GMT

As I understand it, UCS-2 != UTF-16
UCS-2 can only encode U+0000 to U+FFFF (2 bytes per wide char, no more)
UTF-16 can encode all Unicode code points.

re: Some files come up strange in Notepad

Bertg — Wed, 24 Mar 2004 16:26:00 GMT

that makes a lot of sence...
but...
If notepad is unsure shouldn't it ask to the user if the text is displayed correctely and if not try an other encoding?
(sorry about the spelling)

Notepad file encoding

Deep Thoughts... — Wed, 24 Mar 2004 20:16:00 GMT

re: Some files come up strange in Notepad

Simon Cooke [exMSFT] — Wed, 24 Mar 2004 18:54:00 GMT

Nate: Just wondering... did UCS-2 / UTF-16 even exist back when Unicode was in v1.0? IIRC, when Unicode first started out (and Microsoft implemented it from v1), there was only UCS-2. Which explains why they call that "Unicode" - because when they started implementing, that WAS Unicode.

re: Some files come up strange in Notepad

RichB — Wed, 24 Mar 2004 20:06:00 GMT

Talking about Notepad, why does it reformat itself when I save, yet forget to repaint itself? It reformats itself using it's line wrapping algorithm which is different prior to a save.

Notepad has been like this since NT4 at least.

re: Some files come up strange in Notepad

Nate S — Wed, 24 Mar 2004 20:29:00 GMT

I think you're right, Simon. At the time the Unicode people were thinking it would be a 2-byte encoding. Still, that was a long time ago. Even newer systems like C# are still using UCS-2 and calling it Unicode.

re: Some files come up strange in Notepad

Joe — Wed, 24 Mar 2004 22:04:00 GMT

Unicode is nothing but a big table. Therefore nothing is really "Unicode" except the Unicode table. In the computer world we deal with encodings of value in the Unicode table, and in that way UCS-2 is a Unicode encoding as much as UCS-4 is a Unicode encoding. That is, UCS-4 isn't "Unicode" in the same way the UCS-2 isn't "Unicode." But UCS-2 is a perfectly valid Unicode encoding. Microsoft has chosen UCS-2 as its internal Unicode encoding, they could have chosen UTF-8 and still called it "Unicode".

This is a very handy site dealing with UTF-8, but also addresses a lot of stuff surrounding Unicode: http://www.cl.cam.ac.uk/~mgk25/unicode.html

re: Some files come up strange in Notepad

asdf — Wed, 24 Mar 2004 22:12:00 GMT

Search for "EM_GETHANDLE wrap" (without the quotes) on google groups for an explanation of how notepad (and other apps) implement wordwrapping with the edit control. Basically it flashes because it destroys and recreates the edit control.

re: Some files come up strange in Notepad

Nate S — Wed, 24 Mar 2004 22:31:00 GMT

Exactly Joe. In Raymond's post above, you can see some confusion. He lists the available encodings as ANSI, Unicode, UTF-8, UTF-7, and so on.

But "Unicode" is not an encoding. UTF-8, UTF-7 etc. are encodings. The encoding which he refers to as Unicode is properly called UCS-2.

This is a small but nagging problem in Microsoft documentation.

re: Some files come up strange in Notepad

Norman Diamond — Wed, 24 Mar 2004 23:45:00 GMT

Actually the encodings that do not have special prefixes and which are still supported by Notepad are the traditional ANSI encoding (i.e., Shift-JIS, ANSI code page 932) and the Unicode (little-endian) encoding with no BOM.

But in the example posted by David Cumps, Notepad did not choose between those two encodings. Notepad chose a wildly different encoding. Notepad used its usual Japanese font for display, in which it chose a total of eight items: five double-byte full-width Kanji characters; and three single-byte non-displayable characters for which it displayed three half-width black rectangles. But the encoding is not a Japanese encoding, so Notebook's choice of characters was nonsense.

Here's another example, related to a recently discussed IE security bug. I think you'll need a tool other than Notepad to create the file, a single byte with value 0x01, i.e. a Ctrl-A. Open the file in Notepad and it displays a British pound sign. If you have a Japanese font, it displays a full-width British pound sign, as if it were a double-byte character. But move the text cursor to the right and it only moves half-way through the character, because there's only a single byte. Add some single-byte half-width characters after that, and you can see Notepad get really confused about which characters are which. Add some full-width characters and you can watch Notepad move the text cursor through midpoints of characters instead of between characters.

re: Some files come up strange in Notepad

Peter Evans — Thu, 25 Mar 2004 00:30:00 GMT

Yeah Joe and Nate,

The confusing thing with Unicode is that ever one confuses its various encodings as equivalent to being Unicode.

When in reality the Unicode(ISO/IED 10646) really defines an order set of code values assigned to characters names and properties and rules for mapping those code values and properties to glyphs.

The collective code values and properties can be used with other unicode rules to manipulate the character entities for various writing systems and natural languages.

Even more confusion comes from the need to translate from various UTF-X format encodings or UCS-2 code values into the full spectrum UCS-4 code values.

In the end, all the encoding stuff just results in a precise lookup for finding the code value and character properties from the byte encodings whether they be UTF-7, UTF-8,UTF-16, UTF-32.

I think the confusion is more of UNICODE naming / branding problem than a Microsoft documenation issue. Unicode really is the brand for compliance to the standard and not the encoding of the standard.

Most coders do not care about UNICODE all they need to know is how do I decode/encode it and how do I detect it and what APIS work with it.
Afer all isn't that what the purpose of the Uniscribe APIs were. Not sure what there equivalent is in .NET yet.

re: Some files come up strange in Notepad

Simon Cooke [exMSFT] — Thu, 25 Mar 2004 04:38:00 GMT

> Most coders do not care about UNICODE all
> they need to know is how do I decode/encode
> it and how do I detect it and what APIS work
> with it.
> Afer all isn't that what the purpose of the
> Uniscribe APIs were. Not sure what there
> equivalent is in .NET yet.

No equivalent in .NET, alas. And Uniscribe is one scary, badly documented, weak-on-samples API.

It's fine if you're doing single-style text, but the moment you go for something with more to it, or start worrying about resolution independent layout, and you start biting your fingernails...

re: Some files come up strange in Notepad

/dev/null — Thu, 25 Mar 2004 04:51:00 GMT

That was insightful !

But how come other editors open the file correctly (Wordpad, emacs)? I mean, how do they come to know what encoding is it.

And if they can figure out, then why not Notepad?

re: Some files come up strange in Notepad

Jordan Russell — Thu, 25 Mar 2004 15:52:00 GMT

Notepad would get it right too if it didn't employ IsTextUnicode's documented-to-be-unreliable "statistical analysis" (IS_TEXT_UNICODE_STATISTICS).

re: Notepad bug? Encoding issue?

David Cumps — Thu, 25 Mar 2004 21:52:00 GMT

re: Some files come up strange in Notepad

Raymond Chen — Sat, 27 Mar 2004 08:23:00 GMT

Bertg: On why Notepad doesn't prompt if it's not sure: Because it's almost never 100% sure. The file "Hi" sure looks like 8-bit ASCII but maybe it's the single Unicode character U+6948. The UTF-8 file "Hi" might actually be an 8-bit file that happens to begin with EF BB BF ("ï»¿").

If Notepad prompted if there was ambiguity, it would be prompting an awful lot.

You can try to override the autodetector from the File.Open dialog if you find that it detected incorrectly.

How to Determine Text File Encoding

CodeSnipers.com — Tue, 13 Sep 2005 13:49:21 GMT

With the explosion of international text resources brought by the Internet, the standards for determining file encodings have become more important. This is my attempt at making the text file encoding issues digestible by leaving out some of the unimporta

The Binary Cult Blog

The Binary Cult Blog — Sat, 17 Jun 2006 06:34:32 GMT

PingBack from http://blog.binarycult.com/?p=

My Wierd Wired World » Blog Archive » Buggy Notepad

My Wierd Wired World » Blog Archive » Buggy Notepad — Tue, 20 Jun 2006 11:36:04 GMT

PingBack from http://ychittaranjan.wordpress.com/2006/06/20/buggy-notepad/

El M??dem » Blog Archive » Referencia a Bush en Windows - el motivo

El M??dem » Blog Archive » Referencia a Bush en Windows - el motivo — Mon, 19 Feb 2007 09:22:06 GMT

PingBack from http://www.elmodem.com/archivo/2007/02/19/referencia-a-bush-en-windows-el-motivo/

The Notepad file encoding problem, redux

The Old New Thing — Tue, 17 Apr 2007 20:58:59 GMT

Let's take another look.

' + title + ' - ' + basename(imgurl) + '(' + w + 'x' + h +')

' + title + ' - ' + basename(imgurl) + '(' + w + 'x' + h +') — Fri, 15 Jun 2007 12:35:20 GMT

PingBack from http://kent.ppolis.com/?p=130

BOM BOM BOM « ???????????????

BOM BOM BOM « ??????????????? — Wed, 19 Sep 2007 11:21:51 GMT

PingBack from http://atedev.wordpress.com/2007/09/19/bom-bom-bom/

Mails that I really, really don't want to receive - 2

MSDN Blog Postings » Mails that I really, really don't want to receive - 2 — Tue, 25 Sep 2007 18:19:12 GMT

PingBack from http://blogs.msdn.com/pranavwagh/archive/2007/09/25/mails-that-i-really-really-don-t-want-to-receive-2.aspx

Microsoft crazy facts - WCCFtech.com

Microsoft crazy facts - WCCFtech.com — Mon, 28 Apr 2008 11:07:52 GMT

PingBack from http://wccftech.com/forum/microsoft-crazy-facts-18198.html#post212534

Conspira??ia teoriei « Fly on the Windscreen

Conspira??ia teoriei « Fly on the Windscreen — Wed, 05 Nov 2008 10:59:03 GMT

PingBack from http://dorinlazar.wordpress.com/2008/11/05/conspiratia-teoriei/

StUF – receiving data from a provider where UTF-8 is in fact ISO-8859 « The Wiert Corner

Sun, 10 May 2009 01:38:37 GMT

PingBack from http://wiert.wordpress.com/2009/05/08/stuf-receiving-data-from-a-provider-where-utf-8-is-in-fact-iso-8859/