In Win2008 R2 some Multi-threaded terminal server application may crashes with access violation in the test  eax, eax cpu instruction with following symptoms. This issue is very intermittent.

1.         

2.       You may find following 2 threads,

  • Executing a test instruction and causing AV.
  • Trying to change the protection of exactly same code memory page which the first thread is causing access violation ( Not waiting ).
  • If DEP for the process is turned off application is working fine.

Following are the analysis details,

The stored exception information can be accessed via .ecxr.

(428.b50): Access violation - code c0000005 (first/second chance not available)

eax=00000000 ebx=7740f85c ecx=74e92dd9 edx=00000000 esi=00000000 edi=00000001

eip=741c17cd esp=0018a4dc ebp=0018a508 iopl=0         nv up ei pl nz na pe nc

cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206

mswsock!SockWaitForSingleObject+0x3a:

741c17cd 85c0            test    eax,eax

 

 

0:000> k100

ChildEBP RetAddr 

0018a508 741c678c mswsock!SockWaitForSingleObject+0x3a

0018a5f4 75f04a20 mswsock!WSPSelect+0x3a6

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for DvZediRimac002.dll -

0018a674 4b6f447d ws2_32!select+0x494

WARNING: Stack unwind information not available. Following frames may be wrong.

0018e598 4b6e5a80 ThirdParty!ThirdParty +0xac

0018e6ac 774bfd6e ThirdParty!ThirdParty +0x2fc

0018e698 00000000 ntdll!RtlpValidateHeap+0x20

 

0:000> !analyze -v

*******************************************************************************

*                                                                             *

*                        Exception Analysis                                   *

*                                                                             *

*******************************************************************************

 

Debugger CompCtrlDb Connection::Open failed 80004005

Debugger CompCtrlDb Connection::Open failed 80004005

Debugger CompCtrlDb Connection::Open failed 80004005

Debugger Dbgportaldb Connection::Open failed 80040e4d

Database Dbgportaldb not connected

 

FAULTING_IP:

mswsock!SockWaitForSingleObject+3a

741c17cd 85c0            test    eax,eax

 

EXCEPTION_RECORD:  ffffffff -- (.exr ffffffffffffffff)

ExceptionAddress: 741c17cd (mswsock!SockWaitForSingleObject+0x0000003a)

   ExceptionCode: c0000005 (Access violation)

  ExceptionFlags: 00000000

NumberParameters: 2

   Parameter[0]: 00000008

   Parameter[1]: 741c17cd

Attempt to execute non-executable address 741c17cd

 

PROCESS_NAME:  KAREWE.exe

 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

 

WRITE_ADDRESS:  741c17cd

 

BUGCHECK_STR:  ACCESS_VIOLATION

 

LAST_CONTROL_TRANSFER:  from 741c678c to 741c17cd

 

STACK_TEXT: 

0018a508 741c678c 0000025c 00000264 00000001 mswsock!SockWaitForSingleObject+0x3a

0018a5f4 75f04a20 00000000 0018a6a0 00000000 mswsock!WSPSelect+0x3a6

0018a674 4b6f447d 00000000 0018a6a0 00000000 ws2_32!select+0x494

WARNING: Stack unwind information not available. Following frames may be wrong.

0018e598 4b6e5a80 0018e6fc 00000200 00000001 ThirdParty!ThirdParty +0xac

0018e6ac 774bfd6e 00000000 00000044 00310150 ThirdParty!ThirdParty +0x2fc

0018e698 00000000 02c30ac8 23040027 0000000d ntdll!RtlpValidateHeap+0x20

 

 

STACK_COMMAND:  ~0s; .ecxr ; kb

 

PRIMARY_PROBLEM_CLASS:  SOFTWARE_NX_FAULT_FALSE_POSITIVE

 

DEFAULT_BUCKET_ID:  SOFTWARE_NX_FAULT_FALSE_POSITIVE

 

FAULTING_THREAD:  00000b50

 

FOLLOWUP_IP:

mswsock!SockWaitForSingleObject+3a

741c17cd 85c0            test    eax,eax

 

SYMBOL_STACK_INDEX:  0

 

FOLLOWUP_NAME:  wsstress

 

MODULE_NAME:  mswsock

 

DEBUG_FLR_IMAGE_TIMESTAMP:  4a5bda77

 

SYMBOL_NAME:  mswsock!SockWaitForSingleObject+3a

 

IMAGE_NAME:  mswsock.dll

 

FAILURE_BUCKET_ID:  ACCESS_VIOLATION_mswsock!SockWaitForSingleObject+3a

 

BUCKET_ID:  ACCESS_VIOLATION_mswsock!SockWaitForSingleObject+3a

 

Followup: wsstress

---------

 

 

 

·         The application is failing at

741c17cd 85c0            test    eax,eax

 

due to access violation.

·         I checked if any dll is tampered with

0:000> !for_each_module !chkimg @#ModuleName

 

found that the dll under question(mswsock) is not tampered.

  • Above instruction (test) don't access any memory location other than where EIP points.
  • So for the above instruction to generate an AV, the EIP should point to an address in a non executable page of memory.
  • Again from !address command we can see that this a code region and it is from a mapped dll which should ideally an executable page.

0:000> !address eip

 TEB 7efdd000 in range 7efdb000 7efde000

 TEB 7efda000 in range 7efd8000 7efdb000

 ProcessParametrs 003213f0 in range 00320000 0038f000

 Environment 00320810 in range 00320000 0038f000

    741c0000 : 741c1000 - 00001000

                    Type     01000000 MEM_IMAGE

                    Protect  00000004 PAGE_READWRITE

                    State    00001000 MEM_COMMIT

                    Usage    RegionUsageImage

                    FullPath C:\Windows\System32\mswsock.dll

 

On further debugging I found the following as well, which should be true in at least some cases of execution.

 

Only two threads in the application,

Ø  Executing the test instruction and causing  AV.

Ø  Trying to change the protection of exactly same code memory page which the first thread is causing access violation.

0:000> ~

.  0  Id: 428.b50 Suspend: 1 Teb: 7efdd000 Unfrozen

   1  Id: 428.900 Suspend: 1 Teb: 7efda000 Unfrozen

0:000> ~1s

eax=00000000 ebx=80000000 ecx=00000000 edx=00000000 esi=741c1098 edi=741c0000

eip=7740ffea esp=030afacc ebp=030afb08 iopl=0         nv up ei pl nz na po nc

cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

ntdll!NtProtectVirtualMemory+0x12:

7740ffea 83c404          add     esp,4

 

0:001> k100

ChildEBP RetAddr  

030afacc 7416aa2c ntdll!NtProtectVirtualMemory+0x12

030afb08 7416ac22 TSAPPCMP!TsRedirectRegisteredImage+0xcf

030afb58 7416af58 TSAPPCMP!TsWalkProcessDlls+0xf5

030afb8c 4b740f0c TSAPPCMP!TLoadLibraryA+0x3a

WARNING: Stack unwind information not available. Following frames may be wrong.

030afbe8 4b73f737 ThirdParty!ThirdParty +0x38a9

030afc84 774236fa ThirdParty!ThirdParty +0x20d4

030afd78 4b6b83c3 ntdll!RtlpFreeHeap+0xb7a

030afde0 774236fa ThirdParty!ThirdParty +0x9b

030afddc 50000063 ntdll!RtlpFreeHeap+0xb7a

030afec4 774236fa 0x50000063

030affd4 77429d45 ntdll!RtlpFreeHeap+0xb7a

030affec 00000000 ntdll!_RtlUserThreadStart+0x1b

 

 

Looking at that stack based on following link and trying to sleuth the parameters, ( NtProtectVirtualMemory has FPO and is written in assembly so normal methods don't work )

 

http://undocumented.ntinternals.net/UserMode/Undocumented%20Functions/Memory%20Management/Virtual%20Memory/NtProtectVirtualMemory.html

 

NTSYSAPI

NTSTATUS

NTAPI

NtProtectVirtualMemory(

 

  IN HANDLE               ProcessHandle,

  IN OUT PVOID            *BaseAddress,

  IN OUT PULONG           NumberOfBytesToProtect,

  IN ULONG                NewAccessProtection,

  OUT PULONG              OldAccessProtection );

 

Picking up the function arguments from stack

 

0:001> dds esp

030afacc  7740ffea ntdll!NtProtectVirtualMemory+0x12

030afad0  7416aa2c TSAPPCMP!TsRedirectRegisteredImage+0xcf

030afad4  ffffffff

030afad8  030afafc

030afadc  030afb00

030afae0  00000004

030afae4  030afaf8

030afae8  774f020c ntdll!PebLdr+0xc

030afaec  0038af48

030afaf0  0038be08

030afaf4  00000004

030afaf8  00000020

030afafc  741c1098 mswsock!_imp__NtOpenKey

030afb00  00000004

030afb04  741c0000 mswsock!_imp__OutputDebugStringA <PERF> (mswsock+0x0)

030afb08  030afb58

030afb0c  7416ac22 TSAPPCMP!TsWalkProcessDlls+0xf5

030afb10  741e4818 mswsock!DNSAPI_NULL_THUNK_DATA_DLB+0x8

030afb14  00000000

030afb18  00000001

030afb1c  3c144767

030afb20  00000000

030afb24  4b7a203c DvZediRimac002!CWatchedProxyObj::`vftable'+0x30280

030afb28  00000000

030afb2c  001a0018

030afb30  74162684 TSAPPCMP!`string'

030afb34  030afb54

030afb38  741c00d8 mswsock!_imp__OutputDebugStringA <PERF> (mswsock+0xd8)

030afb3c  0038be08

030afb40  030afb1c

030afb44  00000000

030afb48  030afb7c

 

 

Unassembleing the call to NtProtectVirtualMemory.

 

0:001> ub 7416aa2c L10

TSAPPCMP!TsRedirectRegisteredImage+0xaa [d:\w7rtm\termsrv\tsappcmp\register.c @ 1368]:

7416aa07 59              pop     ecx

7416aa08 59              pop     ecx

7416aa09 e9c2000000      jmp     TSAPPCMP!TsRedirectRegisteredImage+0x173 (7416aad0)

7416aa0e 6a04            push    4

7416aa10 58              pop     eax

7416aa11 8d4df0          lea     ecx,[ebp-10h]

7416aa14 51              push    ecx

7416aa15 50              push    eax

7416aa16 8945f8          mov     dword ptr [ebp-8],eax

7416aa19 8d45f8          lea     eax,[ebp-8]

7416aa1c 50              push    eax

7416aa1d 8d45f4          lea     eax,[ebp-0Ch]

7416aa20 50              push    eax

7416aa21 6aff            push    0FFFFFFFFh

7416aa23 8975f4          mov     dword ptr [ebp-0Ch],esi

7416aa26 ff151c131674    call    dword ptr [TSAPPCMP!_imp__NtProtectVirtualMemory (7416131c)]

 

What address is trying to change the protection.

 

0:001> dc 030afafc

030afafc  741c1098 00000004 741c0000 030afb58  ...t.......tX...

030afb0c  7416ac22 741e4818 00000000 00000001  "..t.H.t........

030afb1c  3c144767 00000000 4b7a203c 00000000  gG.<....< zK....

030afb2c  001a0018 74162684 030afb54 741c00d8  .....&.tT......t

030afb3c  0038be08 030afb1c 00000000 030afb7c  ..8.........|...

030afb4c  7416e80a 4b084bdf 00000000 030afb8c  ...t.K.K........

030afb5c  7416af58 3c1447b3 00000000 4b7a203c  X..t.G.<....< zK

030afb6c  00000000 611a0000 030afb60 7499fa2e  .......a`......t

 

 

Location being changed by NtProtectVirtualMemory.

 

0:001> !address 741c1098

TEB 7efdd000 in range 7efdb000 7efde000

TEB 7efda000 in range 7efd8000 7efdb000

ProcessParametrs 003213f0 in range 00320000 0038f000

Environment 00320810 in range 00320000 0038f000

    741c0000 : 741c1000 - 00001000

                    Type     01000000 MEM_IMAGE

                    Protect  00000004 PAGE_READWRITE

                    State    00001000 MEM_COMMIT

                    Usage    RegionUsageImage

                    FullPath C:\Windows\System32\mswsock.dll

 

Failing EIP in the other thread.

 

0:001> !address 741c17cd

TEB 7efdd000 in range 7efdb000 7efde000

TEB 7efda000 in range 7efd8000 7efdb000

ProcessParametrs 003213f0 in range 00320000 0038f000

Environment 00320810 in range 00320000 0038f000

    741c0000 : 741c1000 - 00001000

                    Type     01000000 MEM_IMAGE

                    Protect  00000004 PAGE_READWRITE

                    State    00001000 MEM_COMMIT

                    Usage    RegionUsageImage

                    FullPath C:\Windows\System32\mswsock.dll

 

 

Conclusion

 

This is an issue in TSAPPCMP.dll

 

Until the fix is release if you happen to run in to this issue following is the solution.

Set the following registry value:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server]

to

"IAT"=dword:00000001