Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Mike Dunn — Thu, 20 Jan 2005 18:32:00 GMT

Strange things are afoot at the U+004B U+20DD ;)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Centaur — Thu, 20 Jan 2005 21:07:00 GMT

The XML specification explicitly permits a UTF-16 BOM at the beginning of the file or stream. Otherwise, it must start with the XML declaration (<?xml version=…>), no whitespace or other characters allowed. At least that’s how I’d interpret sections 4.3.3 and 2.1.

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Dean Harding — Thu, 20 Jan 2005 22:45:00 GMT

Heh, I used to work for Unisys. I always felt bad correcting people when they thought I said "Unicef", cause suddenly I'm not such the good samaritan that they thought I was...

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Michael Kaplan — Fri, 21 Jan 2005 01:04:00 GMT

Mike Dunn -- something not Kosher? :-)

Centaur -- the XML spec allows the BOM; it even describes it. So anyone who does not allow it does so at their peril....

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Michael Kaplan — Fri, 21 Jan 2005 01:43:00 GMT

The Unicode FAQ talks about this issue a bit, also.

http://www.unicode.org/faq/utf_bom.html#BOM

With the number of bytes wasted in web/email communication over a character that takes up only 2-4 bytes in storage and no visible space, it is no wonder that people find Unicode to be complicated!

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Michael Grier [MSFT] — Fri, 21 Jan 2005 04:17:00 GMT

Back in visual studio, we had a few people who were really focussed on getting the editors to be really good Unicode citizens. My (possibly revisionist) history is that we actually introduced use of the utf-8 BOM over there around the time of win98 (vs 6). NT caught up when visual studio users were creating "text files" (whatever the heck /that/ means... :-) that other people couldn't open in notepad.

Re: so much attention:

My 1st dev mgr at Microsoft always noted that it was the little picayune issues that drew the most heated debates because everyone felt they understood /all/ the issues.

to quote Kosh: the avelance has started, it is too late for the pebbles to vote.

UTF-8 has a BOM and people just need to learn to love it. (The tricky question is when to preserve/not preserve a BOM found in a byte stream...) I think you're right; just because something is 8-bit clean doesn't make it a good utf-8 citizen. It has to be very careful not to split an encoding (just like a good UTF-16 citizen has to know not to split high/low surrogates...)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Michael Kaplan — Fri, 21 Jan 2005 06:29:00 GMT

Interesting! I had not heard this before... but I guess the timing is right. I never remember trying UTF-8 in VS6, did it really work?

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Mo — Fri, 21 Jan 2005 07:42:00 GMT

I think the confusion reigns because people expect saving a file as UTF-8 to mean "Save it as UTF-8 if it contains non-ASCII characters, and ASCII otherwise", so they expect the BOM to be only present if characters with values greater than 127 are contained within the file.

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Serge Wautier — Fri, 21 Jan 2005 08:38:00 GMT

What is supposed to be the caret behaviour when encountering such a character ?

I pasted the sponsor message into Notepad and I noticed that even though you don't see the BOM, you can definitely 'feel' it when moving the caret : You need to press the arrow key twice between the 2 ".

Does it mean that it's not completely true to say that apps may safely ignore it, especially at the beginning of a doc: If the app provides edition of the contents, users will have a weird experience and bug reports will flood in !

Also, how does text rendering work ? The BOM is not in the font I use in Notepad.

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

TLKH — Fri, 21 Jan 2005 09:39:00 GMT

As far as I remember from the time when I implemented unicode line breaking algorithm for my editor, U+200d allows breaking before/after it.
The real zero-width-non-breaking-space character (except for BOM) is U+2060, not mentioned in this article.

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Michael Kaplan — Fri, 21 Jan 2005 14:38:00 GMT

TLKH is right -- U+2060 (WORD JOINER) is the preferred character that took on the job formerly occupied by Job #1 of the ZWNBSP. I will put a correction in on the page).

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Michael Kaplan — Fri, 21 Jan 2005 14:45:00 GMT

Serge -- hard to say what the caret behavior should be here -- after all it *is* a space, even though it is zero width. The fact that it is deprecated makes it even less likely that implementations will do much more than ignore it....

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Robert — Sat, 22 Jan 2005 07:51:00 GMT

"For the record, it has occurred to me in the past that it would not be a bad idea to add an option to save files without the BOM."

It would be convienient if UTF-8 could be selected as the "ANSI" codepage in the control panel's advanced regional and language options. Then Notepad and many applications designed for ANSI would automatically support UTF-8 (without BOM). I would prefer this because nowerdays I rarely create text files with legacy ANSI encoding.

For those few applications that make specific assumptions about the ANSI codepage (hard-coded strings with character codes >= 128 etc.), AppLocale provides a good solution:

http://www.microsoft.com/globaldev/tools/apploc.mspx

(A UTF-8 "ANSI" codepage may cause problems if the API implementation depends on assumptions like "ANSI character <= double-byte").

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Michael Kaplan — Sat, 22 Jan 2005 08:40:00 GMT

Unfortunately, this is not possible -- there are too many bugs in Windows and in apps for components that will not work with UTF-8 here....

UTF-8, BOM, Micrisoft

zmx's Weblog/鍾明勳的部落格 — Sun, 23 Jan 2005 16:08:00 GMT

在 Michael Kaplan 那看到 Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!) 解釋為甚麼 Windows 2000 以後的 Notepad 存 UTF-8 的檔案會加上 BOM(Byte Order Mark, U+FEFF), 主要是因為 UTF-8 和 ASCII 是相容的, 為了避免使用者自己忘記用甚麼存, 造成 UTF-8 檔案用 ASCII...

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Michael Grier [MSFT] — Sun, 23 Jan 2005 23:22:00 GMT

Re: vs6 and utf-8:

It did in the new shell ("vegas" as I recall). Only Visual Interdev and Visual J++ used the new shell.

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Michael Kaplan — Sun, 23 Jan 2005 23:42:00 GMT

Ah yes, that part is true, and I have actually used that before to look at some of the collation source files back before the VS.NET shell was solid enough for daily use.

I never knew it drove the Notepad feature, though -- thats cool. Fascinating how one piece of the company drives another, sometimes....

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Chris Walker — Wed, 26 Jan 2005 16:26:00 GMT

I'm been maintaining Notepad for Windows NT/2000/XP/Server 2003 for more than 10 years, so I know some of the history of it.

First off, any additional complexity to the interface has always had heavy pushback from management and user. Second, it has had to respond to various changes in commonly used character sets over the years.

Notepad has to guess the character encoding if it does not know what it is. It uses the IsTextUnicode() API to help it, but in the end, it is still a guess. It may be worth a blog entry to discuss just this API and its (mis)usage.

Notepad only edits Unicode, so all other formats are converted to Unicode when the file is opened and converted back if possible to its original format when saved. If the saved format is some form of Unicode, it will also output the BOM at the beginning of the file. If the file has a BOM, then there is no need to call the unreliable IsTextUnicode() API the next time it is opened.

Notepad remembers what the format of the file was when it was read in and uses this format as the default to save. If the edited file can not be saved in the same format without data loss, the user is warning when saving. Otherwise, no UI is thrown. The net result is ASCII files stay ASCII and Unicode files stay Unicode

Notepad will never send the Edit Control the BOM. It will skip the BOM if it exists.

History:

NT 3.1 shipped with an ASCII only Notepad. In the fall of 1993, several applications were converted to use Unicode. At this time, Notepad started using the BOM. I can't tell you how this decision was made; my memory isn't that good. We also converted other applications like Cardfile and Paintbrush. These first shipped on NT 3.5.

With the advent of the popularity of the Internet, other character formats needed to be supported. This was how support of the Big Endian Unicode came from. I'm sorry for the name that was used, but that was the best we could do at the time. The BOM helped a lot for this. I believe the first time Big Endian Unicode support was shipped was NT 5.0, er, Windows 2000.

It would be easy enough to have Notepad not output a BOM. It would *just* be a UI change in the SaveFile dialog.

The performance of Notepad on large files is not completely related to the fact that it reads the whole file into memory. Remember that in the ASCII file case, it has to convert the whole file to Unicode before it can start. As it turns out, this is very fast compared to the CPU bound work that the Multiline Edit Control does to build up some internal data structures. Of course, if reading the file into memory requires the OS to page to the pagefile, you are going to be hurting. Even after you get a big file swallowed, your experience editing this file will not be pretty. Just try adding for deleting a character at a time. The Edit Control exposes the memory to the application and is required that it be ready to save at any time. So you can imagine that adding one character will shuffle all the characters above it down one character. Fine for small files, but a killer algorithm for a large file.

One could add complexity to Notepad and solve some of these problems. One could had a "BOM" checkbox to the save dialog complete with an explanation as to what a BOM is and why the end user should care. One could add options to save ASCII files in various code pages complete with code page documentation. One could add a preview on the Open Dialog and allow the user to pick the proper character encoding. One could scan files looking for encoding text and if found use that as the default.

Notepad is not just a wrapper for the Edit control. In addition to the file encoding problems, it also has reasonable printing and a text search capability which has some interesting International issues. It hosts just about every Common Dialog (Find, ReplaceText, Print, PageSetup, Open, Save). As far as I can figure, it is only missing ChooseColor. I know this because bugs in Common Dialogs are often reported to me first.

An interesting blog entry would discuss the different End of Line sequence standards. Windows uses carriage-return/linefeed pairs as the legal EOL, while Un*x implementations tend to use newline. You can see this in Notepad when you load a file that just uses the newline as the EOL since the Edit Control uses the Windows standard and bare newline characters are not considered EOLs.

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Michael Kaplan — Wed, 26 Jan 2005 16:39:00 GMT

Awesome info, Chris! And I agree that some of the entries you mention would make good future posts

Especially IsTextUnicode (which is kind of a pain for us since we do not own it but everyone assumes we do), the EOL issue in from *nix-created files (which gets some complaints but much less often than the BOM issue).

Sorry if any of the backtalk about "just an edit control" caused grief -- it is a shorthand we sometimes when talking about rendering issues to distinguish it from WordPad (which is "just" a wrapper around RichEdit, in the same sense).

Obviously both are more than "just" a wrapper so I will choose those words with more care. As a team that finds most of its bug reported through applications that use us even if the bug is not ours, I can understand bugs being reported that actually are not ours....

But thankis for the info, it makes the post itself markedly better to have the full story from one who knows rather than after-the-fact guesses and suppositions.

I don't suppose you know who we should contact now if we wanted to push the BOM issue? (feel free to send it to me offline if you're prefer). :-)

Getting exactly ONE Unicode code point out of UTF-8

Sorting It All Out — Sat, 21 May 2005 10:52:19 GMT

Now this is a question that I would make into an interview question, if only there were some way to do...

unicodeFFFE... is Microsoft off its rocker?

Sorting It All Out — Sun, 11 Sep 2005 10:23:49 GMT

This is an issue that has been around for a long time.
Back in February (geez, I really have been blogging...

Working hard to detect code pages

Anonymous — Sun, 11 Sep 2005 11:19:09 GMT

Yesterday, Buck Hodges was talking about how TFS Version Control determines a file's encoding: ...

Return of the Mark

Sorting It All Out — Wed, 08 Feb 2006 11:05:56 GMT

Well, the RIGHT-TO-LEFT MARK (and its cousin the LEFT-TO-RIGHT MARK), that is!
(apologies to those of...

7-bit UTF-8?

Sorting It All Out — Sat, 27 May 2006 04:02:04 GMT

A while back, regular reader 'Maurits' noted in the Suggestion Box:

Just submitted my first PSS support...

Writing a Unicode file via perl ...

183 — Wed, 07 Jun 2006 20:54:38 GMT

This is small piece of code and lengthy desc of how it works, for createing / writing a Unicode file via a perl script.

You can just byte me

Sorting It All Out — Sat, 28 Oct 2006 23:35:58 GMT

Evan asked in one the many programming aliases: Hi: Anyone knows why there are 3 extra characters added

No byte order marks when using encodings in StreamWriters?

Sorting It All Out — Mon, 20 Nov 2006 13:41:26 GMT

Sometimes the convenient shortcuts blind us to the functionality we actually need, forcing us to work

Not a kernel guy » Забавное чтиво про Unicode BOM.

Not a kernel guy » ???????????????? ?????????? ?????? Unicode BOM. — Fri, 24 Nov 2006 08:00:42 GMT

PingBack from http://blog.not-a-kernel-guy.com/2006/11/23/104

RichTextBox breaking ranks?

Sorting It All Out — Sat, 13 Jan 2007 11:06:28 GMT

The other day someone noticed something about the WinForms RichTextBox . What he noticed was that the

El M??dem » Blog Archive » Referencia a Bush en Windows - el motivo

El M??dem » Blog Archive » Referencia a Bush en Windows - el motivo — Mon, 19 Feb 2007 09:22:11 GMT

PingBack from http://www.elmodem.com/archivo/2007/02/19/referencia-a-bush-en-windows-el-motivo/

If a bunch of specific Unicode characters can no longer live in the same apartment together, can they really claim that they needed their space?

Sorting It All Out — Thu, 17 May 2007 12:46:02 GMT

So dipaksmistry asks (in an off-topic manner in response to this post): Hi, I have written a small programme.

BOM BOM BOM « 就是愛程式

BOM BOM BOM « ??????????????? — Wed, 19 Sep 2007 11:21:57 GMT

PingBack from http://atedev.wordpress.com/2007/09/19/bom-bom-bom/

Byte Order Mark - BOM

?????????????? - ?????????? ?????????????? » Byte Order Mark - BOM — Tue, 01 Jan 2008 13:14:01 GMT

PingBack from http://www.almashroo.com/articles/byte-order-mark-bom/

Why are UTF-8 encoded Unix shell scripts ever written or edited in Notepad?

Sorting it all Out — Tue, 11 Mar 2008 16:23:28 GMT

Everybody hates Microsoft. Well, not everybody. But hating Microsoft seems awfully popular.... It seems

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Yuhong Bao — Thu, 12 Mar 2009 06:34:50 GMT

"NT 3.1 shipped with an ASCII only Notepad. In the fall of 1993, several applications were converted to use Unicode. At this time, Notepad started using the BOM. I can't tell you how this decision was made; my memory isn't that good. We also converted other applications like Cardfile and Paintbrush. These first shipped on NT 3.5."

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Yuhong Bao — Fri, 13 Mar 2009 05:53:34 GMT

"NT 3.1 shipped with an ASCII only Notepad."

Do you mean ANSI-only?

When keeping things on a level Plane[ 1] doesn't work anymore

Sorting it all Out — Tue, 09 Jun 2009 18:42:37 GMT

It has been over three years (in Every character has a story #4: U+feff (alternate title: UTF-8 is the

Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

UTF-8, BOM, Micrisoft

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

Getting exactly ONE Unicode code point out of UTF-8

unicodeFFFE... is Microsoft off its rocker?

Working hard to detect code pages

Return of the Mark

7-bit UTF-8?

Writing a Unicode file via perl ...

You can just byte me

No byte order marks when using encodings in StreamWriters?

Not a kernel guy » Забавное чтиво про Unicode BOM.

RichTextBox breaking ranks?

El M??dem » Blog Archive » Referencia a Bush en Windows - el motivo

If a bunch of specific Unicode characters can no longer live in the same apartment together, can they really claim that they needed their space?

BOM BOM BOM « 就是愛程式

Byte Order Mark - BOM

Why are UTF-8 encoded Unix shell scripts *ever* written or edited in Notepad?

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

re: Every character has a story #4: U+feff (alternate title: UTF-8 is the BOM, dude!)

When keeping things on a level Plane[ 1] doesn't work anymore

Why are UTF-8 encoded Unix shell scripts ever written or edited in Notepad?