Search Form
Syndication
Administrivia

100% roundtrip ASCII? 100% roundtrip ANSI?

Sorting it all Out
Michael Kaplan's random stuff of dubious value
Be sure to read the disclaimer here first!
MSDN Blogs > Sorting it all Out > 100% roundtrip ASCII? 100% roundtrip ANSI?

100% roundtrip ASCII? 100% roundtrip ANSI?

23 Nov 2005 4:01 AM
  • Comments 15

Back in January I was talking about the new compiler error C4819 and how the compiler detected invalid characters.

And anyone who has been reading here knows that the reverse solidus is always the path separator, even when it looks like a yen or a won.

So among the so-called 'ANSI' code pages, ASCII (0x00 - 0x7f) will roundtrip 100% of the time.

How many "invalid" slots are there in the 'ANSI' code pages in the 0x80 - 0xff range, exactly?

Let's take a look at the Windows code pages:

There you have it. Code page 1256 is the only one that is guaranteed to be able to roundtrip every single code point without losing any of the bytes....

 

This post brought to you by "¿" (U+00bf, INVERTED QUESTION MARK)

Comment on the blather
Leave a Comment
  • Please add 3 and 7 and type the answer here:
  • Post
Blog - Comment List
Comments
  • Ben Bryant
    23 Nov 2005 10:54 AM
    100% roundtrip ANSI is a novel concept -- I am not sure of the purpose but anyway I noticed you glossed over the fact that the double byte sets are hard to quantify the same as the single byte ones by saying "up to" rather than trying to get into how the lead bytes depend on the trail bytes, and a byte like 0x30 that looks like ASCII can fail the round-trip if it is a trail byte. (geeks!)
  • 23 Nov 2005 11:08 AM
    The 100% roundtrp question comes up in ways to encode bytes without having to worry about what someone may or may not consider invalid (UTF-8 has rules as does UTF-16, and they get stricter all the time). Nice that the Arabic code page stands before us as a safe haven from all that meddling. :-)

    I suppose I could have written a program to go through each of the trail bytes for those lead bytes and gotten the % of invalids per lead byte and maybe averaged them, but it seemedlik a strange exercise that may not yield much additional info.

    In the end, I guess I was mostly being lazy. Vacation does that....
  • 23 Nov 2005 11:27 AM
    You say that 1252 has 1 invalid character. I count 5.
  • 23 Nov 2005 11:40 AM
    Whoops, you are correct -- sorry, small transcription error!
  • Mihai
    23 Nov 2005 12:26 PM
    I am not sure I understand.
    Roundtrip to what?
  • Nick Lamb
    24 Nov 2005 5:27 AM
    So, to use UTF-8 with the current/next Microsoft compiler you need to tell the OS that your locale uses Arabic codepage 1256 ?

    And that's a "cool feature" ?
  • 24 Nov 2005 9:25 AM
    Nick, huh? That is not what I said.

    What I am saying is that there are times when a person may not be sure of the code page. If you are not, then assuming it is UTF-8 and converting is guaranteed to cause problems -- because illegal sequences cannot be emitted. But cp1256 is a code page you can szafely roundtrip any byte sequence through without losing any bytes because all bytes are legal there.

    Obviously if you know the code page you do not need this. So the times that this is needed will hopefully be rare. But most people roundtrip data through code pages 1252 when they have to do this sort of thing, which is incredibly dangerous since you can actually lose information!
  • Nick Lamb
    24 Nov 2005 1:18 PM
    Maybe I'm jumping ahead here...

    1. C4819 is generated for input which contains "invalid characters"

    2. Windows isn't really UTF-8 capable because, well basically because Windows wasn't very well designed twenty years ago.

    3. So to avoid C4819 you need a locale where all your 8-bit data, which Windows can't conceive of as Unicode, is "valid" even if meaningless.

    4. In this post we find out that the locale needed is cp1256, Arabic.
  • Sorting It All Out
    14 Feb 2006 11:01 AM
    Some people may recall when I talked about how It does not always pay to be compatible. In that post...
  • Sorting It All Out
    17 Mar 2006 9:12 AM
    Riffing on Raymond here, and his post On the fuzzy definition of a "Unicode application"....
    The points...
  • Sorting It All Out
    26 May 2006 3:04 PM
    Francisco Moraes asked in the Suggestion Box:

    Are there any code pages (exception EBCDIC) where the...
  • Sorting it all Out
    3 Jul 2008 3:09 AM

    Julien asks via the Contact link: Dear Mr Kaplan, I would like to display Japanese Characters in the

  • Sorting it all Out
    25 Nov 2008 11:26 AM

    So it all started in a conversation with some of the folks from the SQL Server team when I was at PASS.

  • 6 Sep 2011 8:04 AM

    The short internal links are dead...

  • 6 Sep 2011 10:26 AM

    Yes, six years later, they moved everything. You have to go the new site to find them now....

Page 1 of 1 (15 items)
MSDN Blogs > Sorting it all Out > 100% roundtrip ASCII? 100% roundtrip ANSI?