Spat's WebLog (Steve Patrick)

When things go wrong...

Dr Watson knows best..

Dr Watson knows best..

Rate This
  • Comments 3

 

 

I recently worked on a case which involved a crash  ( it was the MMC ) which didn’t have a lot of info attached to it.  There was no crash dump file, and no live debug to go on. All that I had was a Dr Watson log.

Who’s that you say?  See here for some history of the good doctor.

 

If you run drwtsn32.exe ( not in  Vista  ) you will see some configuration options  

 watson

 

One is a path to the log file, and the other is a path to where it will store the dump.

 

If you look in your machine, I’ll bet you have more than a few entries in your Watson log as well J

 

Here is what this one looked like… ( a few things changed.. bonus points to someone with enough time to waste to tell me the real module )

 

Application exception occurred:

App: C:\WINDOWS\system32\mmc.exe (pid=3444)

When: 10/30/2007 @ 15:35:22.324

                       Exception number: c0000005 (access violation)

*----> System Information <----*

        Computer Name: MyComputer

        User Name: spatdsg

         Terminal Session Id: 1

        Number of Processors: 4

        Processor Type: x86 Family 15 Model 4 Stepping 10

        Windows Version: 5.2

        Current Build: 3790

        Service Pack: 1

        Current Type: Multiprocessor Free

        Registered Organization:  redmond

        Registered Owner: Spatdsg

*----> Task List <----*

   0 System Process

   4 System

 452 smss.exe

 560 csrss.exe

932 svchost.exe

1008 svchost.exe

1068 svchost.exe

1088 svchost.exe

<snip>

---> Module List <----*

000000000400000 - 0000000000409000: C:\WINDOWS\system32\Normaliz.dll

0000000000b20000 - 0000000000b37000: C:\WINDOWS\system32\odbcint.dll

0000000001000000 - 00000000010d2000: C:\WINDOWS\system32\mmc.exe

00000000013d0000 - 00000000013dc000: C:\Program Files\Exchsrvr\bin\pttrace.dll

0000000076dc0000 - 0000000076de7000: C:\WINDOWS\system32\Foo.dll

<snip>

 

 

Now, when I get these I generally open them in notepad,  start from the bottom of the file and work my way up  - looking for the word “FAULT”

 

The file keeps a record of oldest to newest crashes  ( newest being at the bottom of  the file ) .

 

Once I locate the most recent crash I examine the stack which crashed.

 

 

 

eax=00000000 ebx=00000080 ecx=7c83005c edx=00000000 esi=77e6c24b edi=40000000

eip=76ddcb3c esp=0007e850 ebp=0007e868 iopl=0         nv up ei pl zr na po nc

cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246

 

 

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for

C:\WINDOWS\system32\Foo.dll -

function: Foo!SchemaIsClassAContainer

76ddcb1e 08ff             or      bh,bh

76ddcb20 d6               ???

76ddcb21 83f8ff           cmp     eax,0xffffffff

76ddcb24 7556            jnz Foo!SchemaIsClassAContainer+0xce3 (76ddcb7c)

76ddcb26 ff157c10dc76     call    dword ptr [Foo+0x107c (76dc107c)]

76ddcb2c 83f803           cmp     eax,0x3

76ddcb2f 7544            jnz Foo!SchemaIsClassAContainer+0xcdc (76ddcb75)

76ddcb31 6a5c             push    0x5c

76ddcb33 ff7508           push    dword ptr [ebp+0x8]

76ddcb36 ff154411dc76     call    dword ptr [Foo+0x1144 (76dc1144)]

FAULT ->76ddcb3c 66832000         and     word ptr [eax],0x0    ds:0023:00000000=???

76ddcb41 59               pop     ecx

76ddcb42 6a00             push    0x0

76ddcb44 ff7508           push    dword ptr [ebp+0x8]

76ddcb47 8945fc           mov     [ebp-0x4],eax

76ddcb4a ff15cc10dc76     call    dword ptr [Foo+0x10cc (76dc10cc)]

76ddcb50 85c0             test    eax,eax

76ddcb52 741b             jz Foo!SchemaIsClassAContainer+0xcd6 (76ddcb6f)

76ddcb54 8b45fc           mov     eax,[ebp-0x4]

76ddcb57 66c7005c00       mov     word ptr [eax],0x5c

 

 

 

 

As you can see -  there are no symbols present for this crash , so we can’t trust the function it thinks it was in – which  is “Foo!SchemaIsClassAContainer”.   You can however, trust the module but not the function name

 

So .. where and why did we really crash?

 

 

We call some funtion blah() which appears to return a pointer .

 

76ddcb36 ff154411dc76     call    dword ptr [Foo+0x1144 (76dc1144)]

 

Then we dereference this pointer and try to set that to zero via a bitwise AND.

 

However – the pointer  located in EAX is… zero.

 

76ddcb3c 66832000         and     word ptr [eax],0x0    ds:0023:00000000=????

 

Obviously we aren’t checking for NULL on some return.  

 

How to find this source code and what real function we’re calling? If I had the exact same version, and it loaded in the same spot, I could just do an ‘ln’ or start looking at the address 76ddcb3c. But, I don’t and I can’t. 

 

I don’t have the dump, just this text file.

 

There are some hardcoded, and hopefully somewhat unique, interesting opcodes we see in the Watson log.

I pasted them below:

 

76ddcb2c 83f803           cmp     eax,0x3

 

And this sequence:

 

76ddcb31 6a5c             push    0x5c

76ddcb33 ff7508           push    dword ptr [ebp+0x8]

 

So let’s try and look for some opcode pattern  like this:

5c 6a 75 ff

 

When I start the MMC and load this dll - my DLL loads at:

 

0000000076dc0000 - 0000000076de7000: C:\WINDOWS\system32\Foo.dll

 

 

I get that information from the  “lm”  command.

 

Then I determine how much room we need to search through.

 

0:000> ? 0000000076de7000- 0000000076dc0000

Evaluate expression: 159744 = 00027000

 

Then I can search via the ‘s’  command:

 

WORD..

0:000> s -w  76dc0000     L0x27000  5c6a

76de1f04  5c6a 75ff ff08 5815 dc10 6676 2083 5900  j\.u...X..vf. .Y

 

Or:

 

DWORD..

 

0:003>  s -d   76dc0000     L0x27000  75ff5c6a

76de1f04  75ff5c6a 5815ff08 6676dc10 59002083  j\.u...X..vf. .Y

 

 

Wow amazing that we only have one match ... looks like a nice match.

 

0:000> u 76de1f04 

Foo!StoreSchemaInfoToFileW+0x37:

76de1f04 6a5c            push    5Ch

76de1f06 ff7508          push    dword ptr [ebp+8]

76de1f09 ff155810dc76    call    dword ptr [Foo!_imp__wcsrchr (76dc1058)]   ;  there was no match here – returned NULL

76de1f0f 66832000        and     word ptr [eax],0

76de1f13 59              pop     ecx

76de1f14 59              pop     ecx

76de1f15 6a00            push    0

76de1f17 ff7508          push    dword ptr [ebp+8]

76de1f1a 8945fc          mov     dword ptr [ebp-4],eax

76de1f1d ff155412dc76    call    dword ptr [Foo!_imp__CreateDirectoryW (76dc1254)]

76de1f23 85c0            test    eax,eax

76de1f25 741b            je      Foo!StoreSchemaInfoToFileW+0x75 (76de1f42)

76de1f27 8b45fc          mov     eax,dword ptr [ebp-4]

76de1f2a 66c7005c00      mov     word ptr [eax],5Ch

76de1f2f 33c0            xor     eax,eax

76de1f31 50              push    eax

 

So now we know the real function – StoreSchemaInfoToFileW and can go examine it, perhaps change it to check the return.

 

I've used this same method a number of times - no symbols ( some of our reskit tools ) or times like this where there is just the watson log.

 

Maybe it will help someone figure out something similar .. or maybe there was an easier way for me to do this?

 

spatdsg

 

Leave a Comment
  • Please add 2 and 2 and type the answer here:
  • Post
  • I know that one of the Bugslayer columns had a program where you gave it a dll or exe to load and a crash offset and a load address and it figured out the symbol for you.  I've used it before, can't seem to remember the name now though.

  • You can save yourself all this trouble by creating and using map files.

  • yes - but the whole point of the post is what to do when you lack symbols or map files.. and only have a few pieces of assembly to go from.

Page 1 of 1 (3 items)