As Raymond Chen pointed out last year (http://blogs.msdn.com/oldnewthing/archive/2004/01/29/64389.aspx), there is a potential integer overflow when calling operator::new. The C++ compiler in Visual Studio 2005 automatically generates defensive code to mitigate this potential vulnerability.

 

Code like this:

 

class CFoo {

public:

      CFoo() {m_p = NULL;}

private:

      void *m_p;

};

 

void *func(size_t count) {

      if (!count) return NULL;

      return new CFoo[count];

}

 

When compiled with VS2005, yields output like this:

 

; _count$ = esi

 

; 14   :    if (!count) return NULL;

 

  00000     85 f6       test esi, esi

  00002     74 36       je   SHORT $LN4@func

 

; 15   :    return new CFoo[count];

 

  00004     33 c9       xor  ecx, ecx

  00006     8b c6       mov  eax, esi

  00008     ba 04 00 00 00    mov  edx, 4

  0000d     f7 e2       mul  edx

  0000f     0f 90 c1    seto cl

  00012     f7 d9       neg  ecx

  00014     0b c8       or   ecx, eax

  00016     51          push ecx

  00017     e8 00 00 00 00    call ??2@YAPAXI@Z     ; operator new

 

As you can see (assuming you can read assembly code!) the multiply occurs (mul edx) and then the CL register is set or not set depending on the value if the overflow flag. Now here’s the funky part, ECX will end up being 0x00000000 or 0xFFFFFFFF, and because of the next operation (or ecx) the ECX register will either be 0xFFFFFFFF or the value held in EAX, which is result of the initial multiply. This is then passed to operator::new which will fail in the face of 2^N-1 by returning NULL or throwing an exception.

 

It’s better to fail, and potentially crash the application, than have an integer overflow that may lead to code execution.

 

Of course, the correct solution is get the code right in the first place by performing good arithmetic hygiene, and in this case, restrict the maximum number of objects you’ll create:

 

void *func(size_t count) {

      if (!count || count > MAX_FOO_ALLOWED) return NULL;

      return new CFoo[count];

}