Getting the characters in a code page

re: Getting the characters in a code page

Mike Dunn — Sun, 08 Jan 2006 10:52:29 GMT

As a variation on #1, how about passing each of the possible lead+trail byte combos to _ismbclegal?

re: Getting the characters in a code page

Mihai — Sun, 08 Jan 2006 12:08:51 GMT

The only problem (with all of the solutions) is that not all code pages can be covered like this. The main exception is GB-18030, which has characters outside BMP.

Sure, one might extend the range beyond BMP, but the performance goes down and the consumed memory goes up.
In this case a variant of #1 might give better results (whith the warning that the complications and care will be even bigger :-)

re: Getting the characters in a code page

Michael S. Kaplan — Sun, 08 Jan 2006 17:53:33 GMT

For both GB18030 and UTF-8, *all* of Unicode is covered, so if it is a characetr, its oin the code page. And that's that. Easy!

#1 is never easier, though -- it is always more work....

re: Getting the characters in a code page

Michael S. Kaplan — Sun, 08 Jan 2006 17:54:39 GMT

Mike, that is still more work by the time you are done, certainly more complicated to implement....

re: Getting the characters in a code page

Mihai — Sun, 08 Jan 2006 23:24:43 GMT

"For both GB18030 and UTF-8, *all* of Unicode is covered"
True. But the proposed solutions #2 and #3 take BMP only.

"#1 is never easier, though -- it is always more work...."
True again :-)
But I was talking about better results in performance and consumed memory.
Sometimes you have to work more for these two.

re: Getting the characters in a code page

Michael S. Kaplan — Sun, 08 Jan 2006 23:52:43 GMT

Ah, you missed my point -- you do not need to do anything for UTF-8 and GB18030 -- no conversion needed at all!

It is all in there.

re: Getting the characters in a code page

Mihai — Mon, 09 Jan 2006 11:11:27 GMT

"Ah, you missed my point"
True :-) But now I get it.

But do you mean GB18030 contains everything that is in Unicode? I thought it is a (big) subset. So it is a bit like 932, only much bigger, but still not covering all of Unicode. Or is it?

re: Getting the characters in a code page

Michael S. Kaplan — Mon, 09 Jan 2006 14:56:02 GMT

GB18030 is completely tied to Unicode as it is defined, and thus everything in Unicode is in GB18030.

re: Getting the characters in a code page

Rob — Mon, 09 Jan 2006 23:14:57 GMT

Michael,

Thanks for the explanation. If I use recommendation #3 how would I distinguish what characters belong to what code page? Would I use the Hex range for determine the code page?

Thanks
Rob

re: Getting the characters in a code page

Michael S. Kaplan — Mon, 09 Jan 2006 23:40:13 GMT

Hi Rob,

For #3 -- check the return value of the WideCharToMultiByte call -- if the conversion succeeds, it is a valid part of the code page you used to convert, and the byte(s) are in the multibyte param. If it fails, then skip to the next character....

supplementary?

Maurits — Tue, 10 Jan 2006 00:29:25 GMT

This works... until a code page comes out with a supplementary code point in it...

re: Getting the characters in a code page

Michael S. Kaplan — Tue, 10 Jan 2006 00:33:54 GMT

There are no new code pages coming in Windows, sorry! :-)

re: Getting the characters in a code page

Rob — Tue, 10 Jan 2006 20:32:27 GMT

Michael,

Thanks for the info. I'll try #3 and store all the valid code point into a vector of CString and then print them into a file and open them up in IE. Viewing in IE will allow me to see all the valid characters.

Rob

re: Getting the characters in a code page

CK — Thu, 19 Jan 2006 22:56:15 GMT

Hi Michael,

Any code sample(s) for recommendation #3? Thanks in advance if you have any.

Thanks
CK