Hi NTDebuggers, I have another puzzler for you. We started crash2.exe under windbg and it crashed. Go figure! Sometimes we have a very limited amount of data available to figure out what went wrong. That being said, this week’s puzzler only gives you a few clues. Given this week’s debugger output, what do you suspect the problem is? What would you do to further isolate the issue or prove your theory?
If there is more data you need to solve it, post a comment / request and I will provide the data for you. We will post all comments during the week and provide our answer on Friday. We look forward to your comments.
CommandLine: crash2.exe
Symbol search path is: srv*C:\symbols*\\symbols\symbols
Executable search path is:
ModLoad: 00400000 00438000 crash2.exe
ModLoad: 779b0000 77b00000 ntdll.dll
ModLoad: 76180000 76290000 C:\Windows\syswow64\kernel32.dll
(15d0.1688): Break instruction exception - code 80000003 (first chance)
eax=00000000 ebx=00000000 ecx=cd7b0000 edx=00000000 esi=fffffffe edi=77a90094
eip=779c0004 esp=0017faf8 ebp=0017fb28 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!DbgBreakPoint:
779c0004 cc int 3
0:000> g
(15d0.1688): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=0017feec ebx=7619140f ecx=0042ecc8 edx=00000000 esi=00000002 edi=00001770
eip=65732074 esp=0017ff00 ebp=6f207473 iopl=0 nv up ei pl nz ac pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010216
65732074 ?? ???
0:000> k 123
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
0017fefc 66692065 0x65732074
0017ff3c 0041b9a3 0x66692065
*** WARNING: Unable to verify checksum for crash2.exe
0017ffa0 762019f1 crash2!_onexit+0x35
0017ffac 77a2d109 kernel32!BaseThreadInitThunk+0xe
0017ffec 00000000 ntdll!_RtlUserThreadStart+0x23
0:000> lm
start end module name
00400000 00438000 crash2 C (private pdb symbols) D:\CPRRAMP\source\crash2\debug\crash2.pdb
76180000 76290000 kernel32 (private pdb symbols) C:\symbols\wkernel32.pdb\20F7BB5ED22344A2910B27CA7252AE792\wkernel32.pdb
779b0000 77b00000 ntdll (private pdb symbols) C:\symbols\wntdll.pdb\7099E4B6A6984FD08CBC90A4EDD40FD12\wntdll.pdb
0:000> db 66692065
66692065 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
66692075 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
66692085 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
66692095 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
666920a5 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
666920b5 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
666920c5 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
666920d5 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
0:000> db 0x65732074
65732074 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
65732084 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
65732094 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
657320a4 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
657320b4 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
657320c4 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
657320d4 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
657320e4 ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ?? ????????????????
0:000> db 0041b9a3
0041b9a3 c3 e8 42 57 ff ff c3 cc-cc cc cc cc cc cc cc cc ..BW............
0041b9b3 cc cc cc cc cc cc ff 74-24 04 e8 42 5c ff ff f7 .......t$..B\...
0041b9c3 d8 1b c0 f7 d8 59 48 c3-cc cc cc cc 8b 44 24 04 .....YH......D$.
0041b9d3 a3 78 3c 43 00 a3 7c 3c-43 00 a3 80 3c 43 00 a3 .x<C..|<C...<C..
0041b9e3 84 3c 43 00 c3 cc cc cc-cc cc cc 6a 10 68 58 16 .<C........j.hX.
0041b9f3 43 00 e8 cc 5d ff ff 33-ff 57 e8 0f 59 ff ff 59 C...]..3.W..Y..Y
0041ba03 89 7d fc 39 7d 08 75 1c-be 78 3c 43 00 ff 35 78 .}.9}.u..x<C..5x
0041ba13 3c 43 00 e8 19 5e ff ff-89 45 e4 c7 45 e0 02 00 <C...^...E..E...
0:000> db 0x66692065
0:000> dd esp
0017ff00 66692065 72756f20 61747320 69206b63
0017ff10 616c2073 72656772 6f6e6520 54686775
0017ff20 20736968 6d207369 2065726f 61746164
0017ff30 726f6620 00737520 6f76e0ae fffffffe
0017ff40 0041b9a3 0041b9c2 00411181 004146d9
0017ff50 0017ffa0 00412ac4 00000001 002620a0
0017ff60 002620d8 6f220936 00000000 00000000
0017ff70 7efde000 0017ff9c 00000000 00000006
0:000> dds esp
0017ff00 66692065
0017ff04 72756f20
0017ff08 61747320
0017ff0c 69206b63
0017ff10 616c2073
0017ff14 72656772
0017ff18 6f6e6520
0017ff1c 54686775
0017ff20 20736968
0017ff24 6d207369
0017ff28 2065726f
0017ff2c 61746164
0017ff30 726f6620
0017ff34 00737520
0017ff38 6f76e0ae
0017ff3c fffffffe
0017ff40 0041b9a3 crash2!_onexit+0x35
0017ff44 0041b9c2 crash2!atexit+0x9
0017ff48 00411181 crash2!ILT+380(__RTC_Terminate)
0017ff4c 004146d9 crash2!_cinit+0x49
0017ff50 0017ffa0
0017ff54 00412ac4 crash2!__tmainCRTStartup+0x15e
0017ff58 00000001
0017ff5c 002620a0
0017ff60 002620d8
0017ff64 6f220936
0017ff68 00000000
0017ff6c 00000000
0017ff70 7efde000
0017ff74 0017ff9c
0017ff78 00000000
0017ff7c 00000006
Good luck and happy debugging!
Jeff-
[Update: more debugger output, per reader request. Posted 4/17/2008]
Note this cashes without windbg also.
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
ModLoad: 777b0000 77900000 ntdll.dll
ModLoad: 75a10000 75b20000 C:\Windows\syswow64\kernel32.dll
(1324.16d0): Break instruction exception - code 80000003 (first chance)
eax=00000000 ebx=00000000 ecx=73fe0000 edx=00000000 esi=fffffffe edi=77890094
eip=777c0004 esp=0017faf8 ebp=0017fb28 iopl=0 nv up ei pl zr na pe nc
777c0004 cc int 3
(1324.16d0): Access violation - code c0000005 (first chance)
eax=0017feec ebx=75a2140f ecx=0042ecc8 edx=00000000 esi=00000002 edi=00001770
0:000> db esp
0017ff00 65 20 69 66 20 6f 75 72-20 73 74 61 63 6b 20 69 e if our stack i
0017ff10 73 20 6c 61 72 67 65 72-20 65 6e 6f 75 67 68 54 s larger enoughT
0017ff20 68 69 73 20 69 73 20 6d-6f 72 65 20 64 61 74 61 his is more data
0017ff30 20 66 6f 72 20 75 73 00-9b 7d 6b d3 fe ff ff ff for us..}k.....
0017ff40 a3 b9 41 00 c2 b9 41 00-81 11 41 00 d9 46 41 00 ..A...A...A..FA.
0017ff50 a0 ff 17 00 c4 2a 41 00-01 00 00 00 98 21 03 00 .....*A......!..
0017ff60 d0 21 03 00 03 94 3f d3-00 00 00 00 00 00 00 00 .!....?.........
0017ff70 00 e0 fd 7e 9c ff 17 00-00 00 00 00 06 00 00 00 ...~............
0:000> ln 0042ecc8
(0042ecac) crash2!`string'+0x1c | (0042eccc) crash2!`string'
0:000> .formats 0017ff044
Evaluate expression:
Hex: 017ff044
Decimal: 25161796
Octal: 00137770104
Binary: 00000001 01111111 11110000 01000100
Chars: ..D
Time: Mon Oct 19 01:23:16 1970
Float: low 4.70085e-038 high 0
Double: 1.24316e-316
0:000> dd 0017ff044
017ff044 ???????? ???????? ???????? ????????
017ff054 ???????? ???????? ???????? ????????
017ff064 ???????? ???????? ???????? ????????
017ff074 ???????? ???????? ???????? ????????
017ff084 ???????? ???????? ???????? ????????
017ff094 ???????? ???????? ???????? ????????
017ff0a4 ???????? ???????? ???????? ????????
017ff0b4 ???????? ???????? ???????? ????????
0:000> dds crash2!__onexitbegin
00434354 59c96d56
00434358 00000001
0043435c 00000000
00434360 00000000
00434364 00000000
00434368 00000000
0043436c 00000000
00434370 00033760
00434374 00000000
00434378 00000000
0043437c 00000000
00434380 00000000
00434384 00000000
00434388 00000000
0043438c 00000000
00434390 00000000
00434394 00000000
00434398 00000000
0043439c 00000000
004343a0 00000000
004343a4 00000000
004343a8 00000000
004343ac 00000000
004343b0 00000000
004343b4 00000000
004343b8 00000000
004343bc 00000000
004343c0 00000000
004343c4 00000000
004343c8 00000000
004343cc 00000000
004343d0 00000000
0:000> dds crash2!__onexitend
00434350 59c94d56
[Update: our answer. Posted 4/18/2008]
This week we had a lot of people that realized this was a buffer overrun. You guys are so good I’m going to make next week’s puzzler a little harder!
Good work all! They are not all listed but some of the responses I liked the best were:
moltov
Matthieu
Doug
Tal Rosen
This week’s official response
Let’s start off with wmain. You can see here that we push a pointer to a string onto the stack and call fun1 004117a3
0:000> uf crash2!wmain
crash2!wmain [d:\cprramp\source\crash2\crash2\crash2.cpp @ 11]:
11 00412520 55 push ebp
11 00412521 8bec mov ebp,esp
11 00412523 83ec40 sub esp,40h
11 00412526 53 push ebx
11 00412527 56 push esi
11 00412528 57 push edi
12 00412529 686cec4200 push offset crash2!`string' (0042ec6c) << Pushing param onto the stack
12 0041252e e870f2ffff call crash2!ILT+1950(?fun1YAXPADZ) (004117a3) << Making call to fun1
12 00412533 83c404 add esp,4
13 00412536 33c0 xor eax,eax
14 00412538 5f pop edi
14 00412539 5e pop esi
14 0041253a 5b pop ebx
14 0041253b 8be5 mov esp,ebp
14 0041253d 5d pop ebp
14 0041253e c3 ret
Lets dump out the value we are passing.
0:000> da 0042ec6c
0042ec6c "This is a test ot see if our sta"
0042ec8c "ck is larger enough"
0:000> uf 004117a3
crash2!fun1 [d:\cprramp\source\crash2\crash2\crash2.cpp @ 17]:
17 00412550 55 push ebp
17 00412551 8bec mov ebp,esp
17 00412553 83ec4c sub esp,4Ch
17 00412556 53 push ebx
17 00412557 56 push esi
17 00412558 57 push edi
19 00412559 8b4508 mov eax,dword ptr [ebp+8] << Here we are moving EBP+8 into eax. This is basically just loading the address of parameter one into eax
19 0041255c 50 push eax << We push it onto the stack to make our call to strcpy
19 0041255d 8d4df4 lea ecx,[ebp-0Ch] << Now we are loading the address of a local on the stack.
Note the local is ebp-c That means that we can only write 0xC bytes to
this location before we end up overwriting things on the stack like our base pointer and return address.
19 00412560 51 push ecx << Now we push our local variable address onto the stack for our call to strcpy
19 00412566 83c408 add esp,8
20 00412569 8d45f4 lea eax,[ebp-0Ch]
20 0041256c 50 push eax
19 00412561 e8c3eeffff call crash2!ILT+1060(_strcpy) (00411429) << This is where things go WRONG, within the call to strcpy we have overwritten our return address with string data.
Let’s look at the before and after.
0:000> bp 00412560
Breakpoint 2 hit
eax=0042ec6c ebx=75a2140f ecx=0017feec edx=00000000 esi=00000002 edi=00001770
eip=00412560 esp=0017fe9c ebp=0017fef8 iopl=0 nv up ei pl nz ac pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216
crash2!fun1+0x10:
00412560 51 push ecx
0:000> k
0017fef8 00412533 crash2!fun1+0x10 [d:\cprramp\source\crash2\crash2\crash2.cpp @ 19] << The return address is ok here!
0017ff50 00412ac4 crash2!wmain+0x13 [d:\cprramp\source\crash2\crash2\crash2.cpp @ 12]
0017ffa0 75a919f1 crash2!__tmainCRTStartup+0x15e [f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c @ 327]
0017ffac 7782d109 kernel32!BaseThreadInitThunk+0xe
0:000> p << Lets step over the call to strcpy and look again.
eax=0017feec ebx=75a2140f ecx=0042eca0 edx=00686775 esi=00000002 edi=00001770
eip=00412569 esp=0017fea0 ebp=0017fef8 iopl=0 nv up ei pl nz ac pe nc
crash2!fun1+0x19:
00412569 8d45f4 lea eax,[ebp-0Ch]
0017fef8 65732074 crash2!fun1+0x19 [d:\cprramp\source\crash2\crash2\crash2.cpp @ 20]
0017ff3c 0041b9a3 0x65732074 << This is not going to be pretty when we ret out of fun1. We will basically return to nowhere.
0017ffa0 75a919f1 crash2!_onexit+0x35 [f:\sp\vctools\crt_bld\self_x86\crt\src\onexit.c @ 98]
0:000> da 0017fef8 if we do a da on the location of the stack frame we can see what is at that location.
0017fef8 "st ot see if our stack is larger" << It’s our string!
0017ff18 " enough"
Finally we have the C code.
#include <windows.h>
void fun1(char * szData);
void fun2(char * szData);
int _tmain(int argc, _TCHAR* argv[])
{
fun1("This is a test ot see if our stack is larger enough");
return 0;
}
void fun1(char * szdata)
char szData1[10];
strcpy(szData1,szdata);
fun2(szData1);
void fun2(char *szData)
printf("Hello from fun2");
strcat(szData,"This is more data for us");
Have a great weekend, Good luck, and happy debugging!
Great puzzle!
I'll take a guess: somebody called atexit() and passed in a function pointer that's implemented in a DLL. But that DLL has been unloaded when _onexit() gets called.
Looks like crash2.exe is a 32-bit app running on x64 Vista+. EBP and EIP appear trashed; I'd suspect some kind of buffer overrun.
"e if our stack is larger enough This is more data for us"
I guess one thing I'd do is check the output of db esp...
Nice stack overflow. The ascii char are a dead give away.
"if our stack is larger enough This is more data for us"
I somewhat don't understand the "problem". Is it that it crashes or is it that it crashes under Windbg?
This looks like the stack corruption to me.
The ebp has quite different value as esp register(their difference should be less than 1M):
esp=0017ff00 ebp=6f207473
The further "db" command confirms that 6f207473 points to invalid address.(Free VA region)
Since the ebp register is saved on the stack and later popped up from their I assume the garbage value "6f207473" is caused by corruption of "ret value" or "saved ebp value" on the stack.
Esp value looks ok, since the "dps esp" output makes sense.
The Access Violation is caused by "eip=65732074" which also points to the invalid VA region. This is expected because "ebp" value is corrupted, which provides incorrect offset return value for "eip".
Finally, to verify it, we can turn-on the /GS compiler option in the VS2003/2005/2008.
Jeffrey Tan
dds output looks very 'ASCII'
let start with
0:020> da esp
xxxxxxxxxx "fi eruo ats i kal sregro@."
Well, it a buffer overflow corrupting our stack.
since ecx=0042ecc8 (near out module code)
maybe this will give us a hint of the last call?
how about: ln 0042ecc8
?????????
eip=65732074 esp=0017ff00 ebp=6f207473
It means EIP value is from [ESP]
EBP is also erased. Then the overflow is probably caused by something like
pop ebp
retn XXXX
Then, retn opcode gives a bad return address to EIP. I guess the problem is to locate which function causes the buffer overflow right
Is it possible to know the value at offset 0017ff044?
(Hypothesis-1)
There is a common dword between to string it looks like buffer from 0017FEFC is copied to 0017FF40.
It can be verified if dword at 0x17FF44 is equal to 0x65732074.
Debug dump also give the following information.
Then, pointer to atexit()+0x9 offset may have been modified. It means when the program tried to return to atexit()+0x9 it returns to 0x65732074.
atexit offset = 0041b9c2-0x9
A call usualy is coded on 5 bytes, like 0xE8 then let's disassemble the code at 0x0041B9C2 - 0x5 (0x41B9BD)
In the dump we can see:
0041B9BD : e8 42 5c ff ff
Destination is : 0x41B9C2 + 0xFFFF5C42 = 0x0411604
If my logical is right the vulnerable function to the overflow is located at 0x0411604, but we don't see information or this address in the dump.
This answer depends if hypothesis-1 is right or not.
I would concur that it's a stack overrun, probably in either a global/static class destructor, or an explicitly registered atexit handler.
dds crash2!__onexitbegin
dds crash2!__onexitend
Ta
My guess is that you've got a global function like described on the VC Blog, and that global is cause a stack overrun.
Basically, during _cinit() it goes through these, one by one:
...
atexit(_RTC_Terminate);
_initterm( __xc_a, __xc_z );
It happens after the atexit() call. That's why we see it in the dds esp output.
Just a guess... :)
Oh, the blog entry is http://blogs.msdn.com/vcblog/archive/2006/10/20/crt-initialization.aspx
@RichardRudek: I think you're on to something. In fact, what's happening is that the SEH epilog of msvcrt!_onexit (statically linked in this case) is failing due to stack corruption. I'm not experienced enough to know why or how this happens. :-)
Unfortunately, it looks like the nice idea of providing more data is too hard...
Hi JM,
Is there any clue that this is caused by the SEH overwrite instead of stack overflow for global variable? I am glad to hear the reason :-).
Anyway, we can turn-on /SafeSEH to give it a shoot.
I forgot they're encoded. The quickest way to find the corruption would be to use ba on an address within the SEH frame from _onexit+0xc.
Jeffrey: the code at 0041b9a3 matches the return from __SEH_epilog4 in _onexit (in the latest non-debug msvcrt). So it's while clearing up the SEH that exposes the problem, but that doesn't mean it *is* the problem.
Ahh. Optimisation bites again. None of the CRTs I can find optimises crash2!_onexit...