NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

Matt — Tue, 29 Apr 2008 00:01:03 GMT

tcpip!DeliverToUser+0x114:

ba50df27 640fb60d51000000 movzx ecx,byte ptr fs:[51h]

ba50df2f 898880010000 mov dword ptr [eax+180h],ecx

----

tcpip!DeliverToUser+0x114:

ba5024a5 e979ba0000 jmp tcpip!DeliverToUser+0x110 (ba50df23)

Are you sure your code snippet isnt bugged?

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

Matt — Tue, 29 Apr 2008 00:01:45 GMT

f78b6544 ba502493 badb0d00 00000001 00000000 nt!_KiTrap0E+0x2a7 (FPO: [0,0] TrapFrame @ f78b6544)

Bad return address when KiTrap0E is called. It should be ba502494 instead of ba502493.

tcpip!DeliverToUser+0x103:

ba502494 8b15f84154ba mov edx,dword ptr [tcpip!_imp__KeNumberProcessors (ba5441f8)]

[This was the first close response we got, and was a very good start. Is there a specific problem that could cause us to go to ba502493 instead of ba502494?]

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

MSDNTST — Wed, 30 Apr 2008 06:44:59 GMT

I suspect this may be a code corruption.

It is the code address "ba502493" == tcpip!GetAddrType+0x19f that caused the bugcheck.

However, from the "ub ba50d9d8" output, it should call tcpip!DeliverToUser function instead of "tcpip!GetAddrType".

Further searching with "ba5024" in the assembly output shows that the following instruction in tcpip!DeliverToUser will jump to one byte after the crashing address:

"jne tcpip!DeliverToUser+0x103 (ba502494)"

So, I believe this code instruction may be corrupted that the execution jumps one byte before the target "ba502493" which is the garbage instruction "add byte ptr [ebx+5441F815h],cl". It is this garbage instruction that caused the bugcheck.

Jeffrey Tan

[This answer is very close, but there is more that can be said about this problem. How can we jump to ba502493 instead of ba502494?]

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

Ryanman — Wed, 30 Apr 2008 23:36:01 GMT

Was this a bitflip error causing the jump to the improper address?

kd> .formats ba502493

10111010 01010000 00100100 10010011 <--last bit flipped?

should have been:

.formats ba502494

10111010 01010000 00100100 10010100

code path should have been:

ba50df1d 0f857145ffff jne tcpip!DeliverToUser+0x103 (ba502494)

tcpip!DeliverToUser+0x103:

ba502494 8b15f84154ba mov edx,dword ptr [tcpip!_imp__KeNumberProcessors (ba5441f8)]

But instead:

We jump to the wrong location and interpret the instruction as:

008b15f84154 <--Should have started at 8b15 instead. So we end up with the garbage below:

tcpip!GetAddrType+0x19f:

ba502493 008b15f84154 add byte ptr [ebx+5441F815h],cl ds:0023:e074281d=??

[Good effort, you’re getting very close to the answer. However, this example would require that three bits be flipped not just the last bit.]

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

Ryanman — Wed, 30 Apr 2008 23:51:41 GMT

To add to what I posted before, bitfip errors are usually (or is it always??) hardware related. I recommend running a memory diagnostic utility (such as the one on the 2008/Vista DVD or running hardware diagnostics from the vendor.

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

Infro — Thu, 01 May 2008 21:36:35 GMT

From 0f857145ffff jne ba502494

Its a relative jump of FFFF4571 make that FFFF4570 and you'll jump back one more (negative numbers count downward) to ba502493. So you might have a memmory error at that location. Another possibility that could occur is that because you are jumping into another page the page mechanism might have modified EIP if it was programmed incorrectly, but it shouldn't ever touch EIP.

[WE HAVE A WINNER! Congratulations, you are the first person to identify and post the root cause of this problem.]

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

Ryanman — Thu, 01 May 2008 23:13:07 GMT

I see now that it's 3 bits differing instead of just 1 (I should slow down and look at the data more carefully).

Did an interrupt occur and then following the ISR the context was restored incorrectly?

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

Ben Voigt — Fri, 02 May 2008 02:32:15 GMT

Although the hamming distance in the resultant address is 3 bits, the instruction appears to be using relative addressing, and in the relative address only one bit needs to be flipped.

Anyway, tcpip!DeliverToUser+0x103 (ba502494) seems somewhat strange. Aren't entrypoints typically word aligned?

Oh, and the other strangeness -- there are two addresses which resolve to a name of tcpip!DeliverToUser apparently, perhaps taking error handling code out of the fast path or an overloaded name with the mangled part stripped.

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

Mo — Fri, 02 May 2008 06:00:43 GMT

I'll take a guess of why it jump to ba502493 instead of ba502494.

This line

ba50df0d 640fb61551000000 movzx edx,byte ptr fs:[51h]

is this:

FORCEINLINE

ULONG

NTAPI

KeGetCurrentProcessorNumber(VOID)

{

__asm { movzx eax, _PCR KPCR.Number }

}

The .trap ecx=edx=1 which explains why it did not jump on ba50df1d. It smells like SMP related issue.

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

dmitri — Fri, 02 May 2008 12:08:01 GMT

Seems that the bitflip hit the instruction itself this time:

0:000> .formats ffff4571

Evaluate expression:

Hex: ffff4571

Decimal: -47759

Octal: 37777642561

Binary: 11111111 11111111 01000101 01110001

Chars: ..Eq

Time: ***** Invalid

Float: low -1.#QNAN high 0

Double: 2.12197e-314

0:000> .formats ffff4570

Evaluate expression:

Hex: ffff4570

Decimal: -47760

Octal: 37777642560

Binary: 11111111 11111111 01000101 01110000

Chars: ..Ep

Time: ***** Invalid

Float: low -1.#QNAN high 0

Double: 2.12197e-314

original instruction:

ba50df1d 0f857145ffff jne tcpip!DeliverToUser+0x103 (ba502494)

faulty instruction:

ba50df1d 0f857045ffff jne tcpip!GetAddrType+0x19f (ba502493)

re: NTDebugging Puzzler 0x00000004: This didn’t puzzle the Debug Ninja, how about you?

Infro — Sat, 03 May 2008 02:23:13 GMT

Well it took me some time to figure it out because my time was being spent in other places.

My process for doing it was simple, the trapped instruction didn't make any sense, it was a 32bit processor with an ebx+offset > 4GB, doesn't seem like an array over run but a bad value loaded into ebx, so I decided to find where ebx was loaded, so I searched for the binary 8b15f841 and found it an enterly intresting place, a byte off from where it should of been (or so I thought), but then I had to do other things, when I came back I started using livekd and unassembling GetAddrType, got distracted again, when I came back I started over and searched for the instruction again, found it and seen that the way it got there was from a jump, so I looked at the jne, thought well its a relative jump, maybe an interrupt happened and modified eip, after a while of other crazy thoughts (processor error with carry flag? a page monitoring program that screwed up after hooking the paging mechanism?). Then I took a look at the comments and seen "bit flipping" comment and it dawned on my, hardware error, something I don't consider very often, to be honest and as it was very close it had to be the answer, flipped bit, eip/jne offset, well as the comment said

.formats ba502493

.formats ba502494

Can't be that, 3 to 4 is a multibit flip 100, 011, and one less than FFFF4571 is a single bit flip.

Transcript of Windows NT Debugging Blog Live Chat

Microsoft Advanced Windows Debugging and Troubleshooting — Thu, 14 Aug 2008 19:55:39 GMT

For those of you that could not make the live chat on 8/13, here is the transcript of the chat session....