Debug puzzler 0x00000002 “Attack of the crazy stack”

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

C++ Guy — Tue, 15 Apr 2008 00:49:19 GMT

Great puzzle!

I'll take a guess: somebody called atexit() and passed in a function pointer that's implemented in a DLL. But that DLL has been unloaded when _onexit() gets called.

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

molotov — Tue, 15 Apr 2008 05:05:09 GMT

Looks like crash2.exe is a 32-bit app running on x64 Vista+. EBP and EIP appear trashed; I'd suspect some kind of buffer overrun.

"e if our stack is larger enough This is more data for us"

I guess one thing I'd do is check the output of db esp...

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

Doug — Tue, 15 Apr 2008 06:33:55 GMT

Nice stack overflow. The ascii char are a dead give away.

"if our stack is larger enough This is more data for us"

I somewhat don't understand the "problem". Is it that it crashes or is it that it crashes under Windbg?

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

MSDNTST — Tue, 15 Apr 2008 07:28:29 GMT

This looks like the stack corruption to me.

The ebp has quite different value as esp register(their difference should be less than 1M):

esp=0017ff00 ebp=6f207473

The further "db" command confirms that 6f207473 points to invalid address.(Free VA region)

Since the ebp register is saved on the stack and later popped up from their I assume the garbage value "6f207473" is caused by corruption of "ret value" or "saved ebp value" on the stack.

Esp value looks ok, since the "dps esp" output makes sense.

The Access Violation is caused by "eip=65732074" which also points to the invalid VA region. This is expected because "ebp" value is corrupted, which provides incorrect offset return value for "eip".

Finally, to verify it, we can turn-on the /GS compiler option in the VS2003/2005/2008.

Jeffrey Tan

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

Tal Rosen — Tue, 15 Apr 2008 11:43:40 GMT

dds output looks very 'ASCII'

let start with

0:020> da esp

xxxxxxxxxx "fi eruo ats i kal sregro@."

Well, it a buffer overflow corrupting our stack.

since ecx=0042ecc8 (near out module code)

maybe this will give us a hint of the last call?

how about: ln 0042ecc8

?????????

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

Matthieu — Tue, 15 Apr 2008 17:18:12 GMT

eip=65732074 esp=0017ff00 ebp=6f207473

0017fefc 66692065 0x65732074

It means EIP value is from [ESP]

EBP is also erased. Then the overflow is probably caused by something like

pop ebp

retn XXXX

Then, retn opcode gives a bad return address to EIP. I guess the problem is to locate which function causes the buffer overflow right

ChildEBP RetAddr

WARNING: Frame IP not in any known module. Following frames may be wrong.

0017fefc 66692065 0x65732074

0017ff3c 0041b9a3 0x66692065

Is it possible to know the value at offset 0017ff044?

(Hypothesis-1)

There is a common dword between to string it looks like buffer from 0017FEFC is copied to 0017FF40.

It can be verified if dword at 0x17FF44 is equal to 0x65732074.

Debug dump also give the following information.

0017ff3c fffffffe

0017ff40 0041b9a3 crash2!_onexit+0x35

0017ff44 0041b9c2 crash2!atexit+0x9

Then, pointer to atexit()+0x9 offset may have been modified. It means when the program tried to return to atexit()+0x9 it returns to 0x65732074.

atexit offset = 0041b9c2-0x9

A call usualy is coded on 5 bytes, like 0xE8 then let's disassemble the code at 0x0041B9C2 - 0x5 (0x41B9BD)

In the dump we can see:

0041B9BD : e8 42 5c ff ff

Destination is : 0x41B9C2 + 0xFFFF5C42 = 0x0411604

If my logical is right the vulnerable function to the overflow is located at 0x0411604, but we don't see information or this address in the dump.

This answer depends if hypothesis-1 is right or not.

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

Skywing — Tue, 15 Apr 2008 18:12:02 GMT

I would concur that it's a stack overrun, probably in either a global/static class destructor, or an explicitly registered atexit handler.

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

Mark Steward — Wed, 16 Apr 2008 05:31:48 GMT

dds crash2!__onexitbegin

dds crash2!__onexitend

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

RichardRudek — Wed, 16 Apr 2008 07:54:01 GMT

My guess is that you've got a global function like described on the VC Blog, and that global is cause a stack overrun.

Basically, during _cinit() it goes through these, one by one:

...

atexit(_RTC_Terminate);

_initterm( __xc_a, __xc_z );

...

It happens after the atexit() call. That's why we see it in the dds esp output.

Just a guess... :)

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

RichardRudek — Wed, 16 Apr 2008 07:54:47 GMT

Oh, the blog entry is http://blogs.msdn.com/vcblog/archive/2006/10/20/crt-initialization.aspx

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

JM — Thu, 17 Apr 2008 13:37:30 GMT

@RichardRudek: I think you're on to something. In fact, what's happening is that the SEH epilog of msvcrt!_onexit (statically linked in this case) is failing due to stack corruption. I'm not experienced enough to know why or how this happens. :-)

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

Anonymous — Thu, 17 Apr 2008 18:05:28 GMT

Unfortunately, it looks like the nice idea of providing more data is too hard...

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

Jeffrey Tan — Fri, 18 Apr 2008 04:01:33 GMT

Hi JM,

Is there any clue that this is caused by the SEH overwrite instead of stack overflow for global variable? I am glad to hear the reason :-).

Anyway, we can turn-on /SafeSEH to give it a shoot.

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

Mark Steward — Fri, 18 Apr 2008 18:35:10 GMT

I forgot they're encoded. The quickest way to find the corruption would be to use ba on an address within the SEH frame from _onexit+0xc.

Jeffrey: the code at 0041b9a3 matches the return from __SEH_epilog4 in _onexit (in the latest non-debug msvcrt). So it's while clearing up the SEH that exposes the problem, but that doesn't mean it *is* the problem.

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

Mark Steward — Fri, 18 Apr 2008 19:50:25 GMT

Ahh. Optimisation bites again. None of the CRTs I can find optimises crash2!_onexit...

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

JM — Fri, 18 Apr 2008 23:05:43 GMT

@Jeffrey: Well, I obviously was wrong, but that's because I don't have enough experience decoding debugger output. For example, I did not understand how this:

0017ffa0 75a919f1 crash2!__tmainCRTStartup+0x15e [f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c @ 327]

0017ffac 7782d109 kernel32!BaseThreadInitThunk+0xe

0017ffec 00000000 ntdll!_RtlUserThreadStart+0x23

Turns into this on the next step:

0017ffa0 75a919f1 crash2!_onexit+0x35 [f:\sp\vctools\crt_bld\self_x86\crt\src\onexit.c @ 98]

0017ffac 7782d109 kernel32!BaseThreadInitThunk+0xe

0017ffec 00000000 ntdll!_RtlUserThreadStart+0x23

I thought the symbol indicated the symbol associated with the return address ("75a919f1") which I could have known was not the case from the initial start ("00000000" is certainly not ntdll!_RtlUserThreadStart+0x23). So ignore me, I'm an idiot.

(The SEH overwrite was based on disassembling my copy of _onexit(), which is executing the SEH epilog around +0x35).

re: Debug puzzler 0x00000002 “Attack of the crazy stack”

Mark Steward — Sat, 19 Apr 2008 03:19:32 GMT

Ignore everything I said this morning - this puzzle is a brilliant example of why you can't make assumptions when debugging!

JM: the current line (the symbol) is the return address from the line above. Windbg sees 0041b9a3 on the stack, and assumes that's the return address. It therefore doesn't find wmain's frame at 0017ff50, which is obvious when you check it manually.

The reason it's on the stack is that _cinit previously called atexit(_RTC_Terminate), which uses SEH to do a check on the stack on termination. But these aren't valid frames - they just haven't been overwritten yet, because wmain has allocated 40 bytes of unused local stack space.

Unfortunately, onexit's name confused everyone except RichardRudek: onexit is not called while exiting! As soon as you know that, the problem is just "stack overflow".

If debugging outside a blog, I'd always check the most reliable-looking frame by doing ub 00412ac4 l1, which would show it had called wmain, not _cinit.

P.S. If you actually build this program, you need /GS- to avoid getting a STATUS_STACK_BUFFER_OVERRUN exception!