Van's House : VC

Output Text in Unicode

xiangfan — Tue, 04 Aug 2009 16:35:35 GMT

According to C standard, it only supports output text in MBCS (Multi-Byte Character String):

The wide character output functions convert wide characters to multibyte characters and write them to the stream as if they were written by successive calls to the fputwc function. Each conversion occurs as if by a call to the wcrtomb function, with the conversion state described by the stream’s own mbstate_t object. The byte output functions write characters to the stream as if by successive calls to the fputc function.

However, MBCS doesn’t support mixing characters in different code page. For example, you can’t use both Chinese and Japanese.

VC provides extension to allow you to output text in Unicode:

#include <cstdio>
#include <locale.h>

#include <io.h>
#include <fcntl.h>

void OutputMBCS(FILE *f)
{
    // ".936" is the code page for Simplified Chinese
    // However, you can't use ".1200" (code page for Unicode) to output text in Unicode
    setlocale(LC_CTYPE, ".936");
    // My name in Chinese: "范翔"
    fwprintf(f, L"%s", L"\x8303\x7FD4");

    // The text in the file is encoded in GBK
}

void OutputUnicode(FILE *f)
{
    _setmode(_fileno(f), _O_U16TEXT);
    fwprintf(f, L"%s", L"\x8303\x7FD4");

    // The text in the file is encoded in Unicode
}

For more information, please check the following post: http://blogs.msdn.com/michkap/archive/2008/03/18/8306597.aspx

Detect Shift Overflow

xiangfan — Sat, 13 Jun 2009 08:19:00 GMT

This is an intellectual exercise: when shifts a 32-bit unsigned integer in C++, how to detect whether the calculation overflows efficiently?

Here is the function prototype. shl_overflow will return true if v << cl overflows (cl is between 0 and 31. And we assume that sizeof(unsigned long) == 4 and sizeof(unsigned long long) == 8).

bool shl_overflow(unsigned long v, int cl)

The most natural way to implement this function is to extend v to 64-bit integer:

bool shl_overflow(unsigned long v, int cl)
{
unsigned long long vl = v;
return (vl << cl >> 32) != 0;
}

Now, let’s dig into the assembly world. We’ll limit the discussion on x86.

mov     eax, DWORD PTR _v$[esp-4]
mov     ecx, DWORD PTR _cl$[esp-4]
xor     edx, edx
call    __allshl
xor     eax, eax
or      eax, edx
jne     overflow

The implementation has to use three specific registers: eax, edx and ecx. And there is an expensive external function call.

If you step into __allshl in the debugger, you can find that it will use shld to shift 64-bit integer. VC provides some intrinsics which map to CPU instructions. For example, __ll_lshift will map to shld.

Because the high dword of vl is 0, we can simplify our code:

bool shl_overflow(unsigned long v, int cl)
{
unsigned long long vl = __ll_lshift(v, cl);
return (static_cast<unsigned long>(vl >> 32)) != 0;
}

The assembly looks like:

mov     eax, DWORD PTR _v$[esp-4]
mov     ecx, DWORD PTR _cl$[esp-4]
xor     edx, edx
shld    edx, eax, cl
test    edx
jne     overflow

Much better now.

Another approach is based on bit representation.

bool shl_overflow(unsigned long v, int cl)
{
    v = _rotl(v, cl);
    unsigned long index;
    return _BitScanForward(&index, v) ? index >= cl : false;
}

The idea is simple. If v << cl overflows, that means the most significant cl bits of v should contains "1".

There are two ways to test that.

1. Scan v from the least significant bits to the most, and test the index against 32 – cl. However, we have to handle the case when cl = 0.

2. Rotate v cl bits left first, so the most significant cl bits will be the least significant cl bits. Then we can scan and test the index against cl directly.

Notice that, the scan may fail if v is 0. The second way is simpler and more efficient.

The assembly looks like:

mov     ecx, DWORD PTR _cl$[esp-4]
mov     eax, DWORD PTR _v$[esp-4]
rol     eax, cl
bsf     eax, eax
je      notoverflow
cmp     eax, ecx
jl      overflow

It only uses two registers. It can also be extended to handle 64-bit shift. One drawback is an extra conditional jump (The extra jump can be replaced by "cmovz eax, ecx", but there is no way to ask the compiler to generate that)

Recursive Algorithm in C++

xiangfan — Sat, 23 May 2009 17:11:00 GMT

Many recursive algorithms have initial parameters. For example, Fibonacci Number is defined as: Fn = Fn-1 + Fn-2, with F1 = F2 = 1.

By giving different values to F1 and F2, we can generate different sequence of numbers.

1. If we implement the algorithm using functions, we have to either define these parameters as global variables or pass them in each recursive iteration.

int Fib(int n, int f1, int f2)
{
    if (n < 1) return 0;

    if (n >= 3) {
        return Fib(n - 1, f1, f2) + Fib(n - 2, f1, f2);
    } else if (n == 2) {
        return f2;
    } else {
        return f1;
    }
}

2. Recursive functor can store the initial parameters as the class data member. Here is the example.

struct FibFunctor
{
    FibFunctor(int f1, int f2) : m_f1(f1), m_f2(f2) {}
    int m_f1, m_f2;

    int operator()(int n) const {
        if (n < 1) return 0;

        if (n >= 3) {
            return (*this)(n - 1) + (*this)(n - 2);
        } else if (n == 2) {
            return m_f2;
        } else {
            return m_f1;
        }
    }
};

int Fib(int n, int f1, int f2)
{
    FibFunctor f(f1, f2);
    return f(n);
}

3. Lambda is a new core language feature introduced in C++0x to define implicit function object. However, "this" is not valid inside lambda expression.

This post Visual C++ Team Blog - Stupid Lambda Tricks shows how to write a recursive lambda. So the above functor can be rewritten as:

int Fib(int n, int f1, int f2)
{
    auto f = [&](int n) -> int {
        if (n < 1) return 0;

        if (n >= 3) {
            return f(n - 1) + f(n - 2);
        } else if (n == 2) {
            return f2;
        } else {
            return f1;
        }
    };
    return f(n);
}

Here, [&] is equivalent to [&f, &f1, &f2].

(According to C++03 3.3.1/1: "The point of declaration for a name is immediately after its complete declarator and before its initializer (if any).", it is legal to capture f in the lambda expression. That means:

struct A {
A(A*);
};

int main()
{
A a = A(&a); // OK
})

Measure Initialization Time of Global Variables

xiangfan — Tue, 12 May 2009 15:16:00 GMT

NOTICE: The technique describes in the article may not be supported in future release of VC. You should not use it in production code

There are two kinds of initialization in C++: static initialization and dynamic initialization.

According to the standard, static initialization shall always be performed before any dynamic initialization takes place.

In VC, static initialization is done at compile time. The value is stored in the data section of the generated executable. On the other hand, dynamic initialization happens at runtime. Before entering main, CRT will call the dynamic initializers of the global variables.

Sometimes you may need to measure the startup time of your program. However, dynamic initialization happens before main, so your measurement will not include it.

This article "CRT Initialization" describes the detailed information of how dynamic initialization works in VC.

VC provides pragma init_seg to fine-control the initialization process. It will place the dynamic initializers in the specific section. Besides predefined compiler, lib and user (the corresponding section names are ".CRT$XCC", ".CRT$XCL" and ".CRT$XCU"), you can also specify the section name explicitly.

With these knowledge, we can use the following code to measure the initialization time of global variables.

Let’s create two files: InitTime_Start.cpp and InitTime_End.cpp (we have to use two files because one file can only have one init_seg)

//InitTime_Start.cpp

#pragma warning(disable : 4075)

#pragma init_seg(".CRT$XCB")

Timer gInitTimer;

//InitTime_End.cpp

#pragma warning(disable : 4075)

#pragma init_seg(".CRT$XCY")

double gInitTime = gInitTimer.GetTime();

Here, "Timer" is a class which will start timing in ctor and return the elapsed time in member function GetTime. Because ".CRT$XCB" will be placed before ".CRT$XCC", the dynamic initializer of the timer will be called before any dynamic initializers of compiler generated global variables. Similarly, gInitTimer.GetTime will be called after all the dynamic initializers of user defined global variables. Then "gInitTime" will contain the initialization time of all global variables including those generated by compiler, library and user.

Enable syntax highlighting for TR1 headers in VS2008 SP1

xiangfan — Sat, 13 Dec 2008 06:39:00 GMT

Unfortunately, VS2008 SP1 doesn't recognize C++ tr1 headers. That means there are no syntax highlighting and no intellisense for these files.

This is a bug, but you can fix it by yourself. The trick is in the registry. VS maintains a list of extensionless files, and will treat them as cpp files.

It is under: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\Languages\Extensionless Files\{B2F072B0-ABC1-11D0-9D62-00C04FD9DFD9}

Just add tr1 headers into it and you'll now get full support of new tr1 features. Here is the reg file:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\Languages\Extensionless Files\{B2F072B0-ABC1-11D0-9D62-00C04FD9DFD9}]
"array"=""
"random"=""
"regex"=""
"tuple"=""
"type_traits"=""
"unordered_map"=""
"unordered_set"=""
"xawrap"=""
"xawrap0"=""
"xawrap1"=""
"xawrap2"=""
"xfwrap"=""
"xfwrap1"=""
"xrefwrap"=""
"xtr1common"=""
"xxbind0"=""
"xxbind1"=""
"xxcallfun"=""
"xxcallobj"=""
"xxcallpmf"=""
"xxcallwrap"=""
"xxfunction"=""
'xxmem_fn"=""
"xxpmfcaller"=""
"xxrefwrap"=""
"xxresult"=""
"xxtuple0"=""
"xxtuple1"=""
"xxtype_traits"=""

VC extensions list

xiangfan — Fri, 12 Dec 2008 01:36:00 GMT

MSDN has a page describing various VC extensions. But it is far from complete.

I've collected a list of nonstandard extensions provided by VC, some of them are evil. If you want to write standard conformant C++ code, you'd better be aware of these extensions which are on by default. Some commonly (mis-)used extensions are in bold.

W4001: nonstandard extension 'single line comment' was used
W4152: nonstandard extension, function/data pointer conversion in expression
W4200: nonstandard extension used : zero-sized array in struct/union
W4201: nonstandard extension used : nameless struct/union
W4202: nonstandard extension used : '...': prototype parameter in name list illegal
W4203: nonstandard extension used : union with static member variable
W4204: nonstandard extension used : non-constant aggregate initializer
W4205: nonstandard extension used : static function declaration in function scope
W4206: nonstandard extension used : translation unit is empty
W4207: nonstandard extension used : extended initializer form
W4208: nonstandard extension used : delete [exp] - exp evaluated but ignored
W4210: nonstandard extension used : function given file scope
W4211: nonstandard extension used : redefined extern to static
W4212: nonstandard extension used : function declaration used ellipsis
W4213: nonstandard extension used : cast on l-value
W4214: nonstandard extension used : bit field types other than int
W4215: nonstandard extension used : long float
W4216: nonstandard extension used : float long
W4218: nonstandard extension used : must specify at least a storage class or a type
W4221: nonstandard extension used : 'identifier' : cannot be initialized using address of automatic variable
W4223: nonstandard extension used : non-lvalue array converted to pointer
W4224: nonstandard extension used : formal parameter 'identifier' was previously defined as a type
W4226: nonstandard extension used : 'keyword' is an obsolete keyword
W4228: nonstandard extension used : qualifiers after comma in declarator list are ignored
W4231: nonstandard extension used : 'identifier' before template explicit instantiation
W4232: nonstandard extension used : 'identifier' : address of dllimport 'dllimport' is not static, identity not guaranteed
W4233: nonstandard extension used : 'keyword' keyword only supported in C++, not C
W4234: nonstandard extension used : 'keyword' keyword reserved for future use
W4235: nonstandard extension used : 'keyword' keyword not supported on this architecture
W4238: nonstandard extension used : class rvalue used as lvalue
W4239: nonstandard extension used : 'token' : conversion from 'type' to 'type'
W4240: nonstandard extension used : access to 'classname' now defined to be 'access specifier', previously it was defined to be 'access specifier'
W4288: nonstandard extension used : 'var' : loop control variable declared in the for-loop is used outside the for-loop scope; it conflicts with the declaration in the outer scope
W4289: nonstandard extension used : 'var' : loop control variable declared in the for-loop is used outside the for-loop scope
W4353: nonstandard extension used: constant 0 as function expression. Use '__noop' function intrinsic instead
W4480: nonstandard extension used: specifying underlying type for enum 'enum'
W4481: nonstandard extension used: override specifier 'keyword'
W4482: nonstandard extension used: enum 'enum' used in qualified name
W4509: nonstandard extension used: 'function' uses SEH and 'object' has destructor
W4836: nonstandard extension used : 'type' : local types or unnamed types cannot be used as template arguments

C2599: 'enum' : forward declaration of enum type is not allowed

Play with the C++ compiler - compile nightmare

xiangfan — Mon, 15 Sep 2008 17:49:00 GMT

Playing with the compiler is interesting. One challenge for the compiler writer is compilation performance. There're many kinds of C++ code which are the nightmare for the compiler. Now let's go!

1. Preprocess

a. Self Inclusion (GCC only)
Normally, the compiler will limit the nest level of inclusion. In order to challenge the compiler, you have to stop the inclusion at appropriate stage. GCC provides a useful macro __INCLUDE_LEVEL__ to do the work.
(We can write similar code for other compiler, although less elegant)

#if __INCLUDE_LEVEL__<199
#include __FILE__
#include __FILE__
#endif

b. Macro Expansion Explosion
By using macro, the code after the preprocess may reach O(2ⁿ). For example:

#define F1(x) x,x
#define F2(x) F1(x),F1(x)
#define F3(x) F2(x),F2(x)
#define F4(x) F3(x),F3(x)
#define F5(x) F4(x),F4(x)
#define F6(x) F5(x),F5(x)
#define F7(x) F6(x),F6(x)
#define F8(x) F7(x),F7(x)
#define F9(x) F8(x),F8(x)
#define G1(x) F9(x),F9(x)
#define G2(x) G1(x),G1(x)
#define G3(x) G2(x),G2(x)
#define G4(x) G3(x),G3(x)
#define G5(x) G4(x),G4(x)
#define G6(x) G5(x),G5(x)
#define G7(x) G6(x),G6(x)
#define G8(x) G7(x),G7(x)
#define G9(x) G8(x),G8(x)

int main()
{
G9(1);
}

Of course, different compilers handle this kind of overflow differently. GCC will fail directly (when all of the resource are eaten up), and VC will give ICE (Internal Compiler Error).

2. Template

a. Nesting
Similarly, there're nest level limit for template, but we can easily deceive the compiler.
Some version of GCC will enter infinite loop when compiles the following code:

#include <cstddef>
template <class T>
struct Test {
static const size_t Value=Test<Test<T> >::Value;
};

VC is wise enough to stop when the nest level limit is exceeded in the above code. But we can use one of VC's bug (you can also treat it as an extension) to write template which requires O(n^a) compilation time:

#include <cstddef>

#define INNER(A3,N3,A2,N2) \
template<size_t N3>\
struct A3\
{\
    enum {N=A3<N3-1>::N+1};\
};\
template<>\
struct A3<0>\
{\
    enum {N=A2<N2-1>::N};\
};

#define OUTER(A2,N2,A1,N1,A3,CONTENT) \
template<size_t N2>\
struct A2\
{\
    CONTENT\
    \
    enum {N=A3<N2>::N};\
};\
template<>\
struct A2<0>\
{\
    enum {N=A1<N1-1>::N};\
};

#define LEVEL2(a,b,c) INNER(A##b,N##b,A##a,N##a)
#define LEVEL3(a,b,c) OUTER(A##b,N##b,A##a,N##a,A##c,LEVEL2(a##1,b##1,c##1))
#define LEVEL4(a,b,c) OUTER(A##b,N##b,A##a,N##a,A##c,LEVEL3(a##1,b##1,c##1))
#define LEVEL5(a,b,c) OUTER(A##b,N##b,A##a,N##a,A##c,LEVEL4(a##1,b##1,c##1))

template<size_t N1>
struct A1
{
    LEVEL5(1,11,111)
    enum {N=A11<N1>::N};
};
template<>
struct A1<0>
{
    enum {N=0};
};

int main()
{
    return A1<20>::N;
}

The interesting thing is that, the standard doesn't allow this kind of specialization in template class. Is this a bug or extension of VC?
A member or a member template may be nested within many enclosing class templates. In an explicit specialization for such a member, the member declaration shall be preceded by a template<> for each enclosing class template that is explicitly specialized.

Based on similar idea, we can write portable version for both GCC and VC.

#define INNER(A3,N3) \
template<class N3>\
struct A3\
{\
    static void f() {}\
};

#define OUTER(A2,N2,A1,CONTENT) \
template<class N2>\
struct A2\
{\
    CONTENT\
    \
    static void f() {\
        A1<char>::f();\
        A1<short>::f();\
        A1<int>::f();\
        A1<long>::f();\
        A1<unsigned char>::f();\
        A1<unsigned short>::f();\
        A1<unsigned int>::f();\
        A1<unsigned long>::f();\
    }\
};

#define LEVEL2(a,b) INNER(A##a,T##a)
#define LEVEL3(a,b) OUTER(A##a,T##a,A##b,LEVEL2(a##1,b##1))
#define LEVEL4(a,b) OUTER(A##a,T##a,A##b,LEVEL3(a##1,b##1))
#define LEVEL5(a,b) OUTER(A##a,T##a,A##b,LEVEL4(a##1,b##1))
#define LEVEL6(a,b) OUTER(A##a,T##a,A##b,LEVEL5(a##1,b##1))
#define LEVEL7(a,b) OUTER(A##a,T##a,A##b,LEVEL6(a##1,b##1))
#define LEVEL8(a,b) OUTER(A##a,T##a,A##b,LEVEL7(a##1,b##1))

LEVEL8(1,11)

int main()
{
    A1<int>::f();
}

b. Multiple inheritance
C++ allows multiple inheritance, it can be used to hang the compiler. Try the following "Fibonacci" inheritance:

template<size_t N>
class Evil:virtual public Evil<N-1>,virtual public Evil<N-2>
{
public:
virtual ~Evil() {}
};

template<>
class Evil<1>
{
};
template<>
class Evil<0>
{
};

int main()
{
Evil<100> evil;
}

c. OLE (Output limit exceeded)
Normally, the compile time for template is O(n), but in contrast, the amount of error message will be O(n²). Taking advantage of this, you can make the error message "Output Limit Exceeded". For example:

#define ClassName A

template <int N>
class ClassName
{
enum {Value=ClassName<N-1>::Value};
};

int main()
{
int n=ClassName<0>::Value;
}

If I change "ClassName" to a very long name (which is widely supported by modern compilers), the error message will be easily OLE.
PS: The above code will cause ICE in VC8. This is the bug of VC8, and VC9 is happy with the code.

Debug vs Release

xiangfan — Sat, 30 Aug 2008 15:09:00 GMT

Someone may wonder what the difference is between Debug and Release mode, and whether it is possible to mix them.

Here is one example: http://forums.msdn.microsoft.com/en-US/vcgeneral/thread/775ce067-b225-4141-8b86-2d7e9b61db97/

syperk said: "As a result, I've switched to compiling my debug builds using the /MD switch. If you do this, you also have to undefine the _DEBUG preprocessor macro. Otherwise you get lots of errors about _CrtDebugReportW() not found. This function is apparently inserted into your code if _DEBUG is defined, but is only defined in the debug CRT. As long as optimization is off it seems to be perfectly possible to debug programs compiled this way, and there are no concerns about CRT incompatibility.".

But that is not true. There are many problems. In my opion, the most important one is that it is not a good idea to link different modules compiled with different compiler options in C++, especially when conditional compilation is involved.

In fact, "Debug" & "Release" are only two sets of predefined compiler flags and macros definitions provided by the IDE (_DEBUG and NDEBUG are two representing macros). The compiler doesn't aware of that (but it has some magic behind compile option "MD" and "MDd").
The main problem is that program compiled using "Debug" setting and the release runtime don't share the same compiler flags, and they are highly possible to be incompatible.

For example, if you change "MDd" -> "MD" in your debug configuration, and compile the following code (don't remove the _DEBUG macro, otherwise the "Debug" version is no longer a debug version (/MDd will define this macro implicitly, and IDE will explicitly define it via compiler flag)) :

#include <iostream>
#include <string>
using namespace std;
int main()
{
std::string str;
return 0;
}

You'll find that the program function properly. If you check the generated exe, you'll find that it refer to two dlls, one is msvcp90d.dll, the other is msvcr90.dll. Oops! You're mixing the debug and release runtime libraries! And that hide the potential incompatibility!

The magic is here (in use_ansi.h):

#ifdef _DEBUG
#pragma comment(lib,"msvcprtd")
#else /* _DEBUG */
#pragma comment(lib,"msvcprt")
#endif /* _DEBUG */

If you disable this trick, and link to msvcp90.dll, you'll get unresolved symbol error, that is because the implementation of string class is different in "Debug" and "Release" mode. This is one example of the incompatibility.

error LNK2019: unresolved external symbol "__declspec(dllimport) public: __thiscall std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >(struct std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >::_Has_debug_it)" (__imp_??0?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@QAE@U_Has_debug_it@01@@Z) referenced in function _main

It's fortunate that this is a linker error. Otherwise, you'll waste lots of time in debugging to find out the subtle incompatibility.

STL in VC9 devotes lots of efforts to prevent break like this, then you can disable macros like "_HAS_ITERATOR_DEBUGGING" as you like, and keep your code compatible with the runtime (which is compiled with the macro enabled). But it only applies when you stick to debug or stick to release. It is really tricky to guarantee that, and I think you should never rely on this.

Conformance macros in VC STL

xiangfan — Sat, 30 Aug 2008 11:01:00 GMT

In VC STL, there are two macros called "_HAS_IMMUTABLE_SETS" and "_HAS_STRICT_CONFORMANCE" which are defined in yvals.h. They are related to some defects in the C++ standard 2003.

1. _HAS_IMMUTABLE_SETS will influence the constness of set::iterator.

Associative containers require its elements to be sorted, so modify them will be dangerous. But the standard doesn't prohibit the programmer to do that.

In C++0x, the following statement is added for associative containter:
For associative containers where the value type is the same as the key type, both iterator and const_iterator are constant iterators
(In the standard, it is said:
Constant iterators do not satisfy the requirements for output iterators, and the result of the expression *i (for constant iterator i) cannot be used in an expression where an lvalue is required.)
That means you cannot modify the value of the elements using iterator any more.

Discussion: http://www.cpptalk.net/image-vp135370.html

2. _HAS_STRICT_CONFORMANCE will influence two places:

a. For set, multiset, map and multimap

void erase(const_iterator position);
size_type erase(const key_type& x);
void erase(const_iterator first, const_iterator last);

iterator erase(const_iterator position);
size_type erase(const key_type& x);
iterator erase(const_iterator first, const_iterator last);

In C++03, erase in associative containers will not return the iterator, which is incompatible with sequence containers. It is fixed in C++0x
VC's extension provides the return value for erase. _HAS_STRICT_CONFORMANCE will disable this extension.

Discussion: http://www.tech-archive.net/Archive/VC/microsoft.public.vc.stl/2005-10/msg00030.html
Further Discussion: http://www.open-std.org/jtc1/sc22/wg21/docs/lwg-defects.html#103

b. For codecvt::do_length(stateT& state, const externT* from, const externT* from_end, size_t max) const;

In C++03, codecvt<wchar_t, char, mbstate_t> and codecvt<char, char, mbstate_t> should return the lesser of max and (from_end-from).
This is weakened in C++0x. Only codecvt<char, char, mbstate_t> is required to behave like that.

BTW: VC's STL implementation is provided by Dinkumware. You can see "P.J. Plauger" at the end of each STL file.