Debug Fundamentals Exercise 3: Calling conventions

Debug Fundamentals Exercise 3: Calling conventions

  • Comments 19

 

Today’s exercise will focus on x86 function calling conventions.  The calling convention of a function describes the following:

 

·         The order in which parameters are passed

·         Where parameters are placed (pushed on the stack or placed in registers)

·         Whether the caller or the callee is responsible for unwinding the stack on return

 

While debugging, an understanding of calling conventions is helpful when you need to determine why certain values are placed in registers or on the stack before a function call.

 

Standard x86 calling convention on Windows:

Name

Arguments

Unwinds stack

Win32 (Stdcall)

pushed onto stack from right to left

callee

Native C++ (Thiscall)

pushed onto stack from right to left, "this" pointer in ecx

callee

COM (Stdcall for C++)

pushed onto stack from right to left, then "this" is pushed

callee

Fastcall

arg1 in ecx, arg2 in edx, remaining args pushed onto stack from right to left

callee

Cdecl

pushed onto stack from right to left

caller

 

 

Question:

Below are calls to 5 functions.  Each function takes two DWORD parameters.  Based on the code that calls each function, identify the calling convention used.

 

// Call to Function1

01002ffe 8b08            mov     ecx,dword ptr [eax]

01003000 53              push    ebx

01003001 687c2c0001      push    offset 01002c7c

01003006 50              push    eax

01003007 ff11            call    dword ptr [ecx]

 

// Call to Function2

01002490 50              push    eax

01002491 688c110001      push    offset 0100118c

01002496 e82a020000      call    dbgex4!Function2 (010026c5)

0100249b 59              pop     ecx

0100249c 59              pop     ecx

 

// Call to Function3

0100248e 8bd0            mov     edx,eax

01002490 8bcf            mov     ecx,edi

01002492 e8aeffffff      call    dbgex4!Function3 (01002445)

 

// Call to Function4

00413586 8b450c          mov     eax,dword ptr [ebp+0Ch]

00413589 50              push    eax

0041358a 8b4d08          mov     ecx,dword ptr [ebp+8]

0041358d 51              push    ecx

0041358e 8b4dec          mov     ecx,dword ptr [ebp-14h]

00413591 e86fdfffff      call    dbgex4!Function4 (00411505)

 

// Call to Function5

01003540 56              push    esi

01003541 8d85d4f9ffff    lea     eax,[ebp-62Ch]

01003547 50              push    eax

01003548 ff1558100001    call    dbgex4!Function5 (01001058)]

 

 

Bonus: describe the calling convention used for x64.

 

 


[Update: our answer. Posted 12/18/2008]

 

Function1 - COM (Stdcall for C++)

 

Function2 - cdecl

 

Function3 - fastcall

 

Function4 - Native C++ (Thiscall)

 

Function5 - Win32 (Stdcall)

 

 

Bonus: describe the calling convention used for x64:  

http://msdn.microsoft.com/en-us/library/ms794533.aspx

 

Leave a Comment
  • Please add 4 and 6 and type the answer here:
  • Post
  • Call to Function1 : COM (Stdcall for C++)

    Call to Function2 : Cdecl

    Call to Function3 : Fastcall

    Call to Function4 : Native C++ (Thiscall)

    Call to Function5 : Win32 (Stdcall)

    --rc

  • function 1 => COM

    function 2 => cdecl

    function 3 => fastcall

    function 4 => thiscall

    function 5 => stdcall

  • function1 - COM (3 pushes)

    function2 - Cdecl (2 pops)

    function3 - fastcall (no pushes)

    function4 - C++ stdcall (2 pushes & mov ecx)

    function5 - Win32 stdcall (2 pushes, no ecx)

  • Function1 - StdCall

    Function2 - Cdecl

    Function3 - FastCall

    Function4 - ThisCall

    Function5 - StdCall (COM if ebp-62Ch holds 'this')

    Thank you very much again. keep it going!

    Moshe.

  • 1:  COM

    2:  Cdecl

    3:  Fastcall

    4:  Native C++

    5:  Win32

    x64:  http://msdn.microsoft.com/en-us/library/zthk2dkh.aspx

  • 1: COM

    2: cdecl

    3: fastcall

    4: thiscall

    5: stdcall

  • Function1 uses "COM (Stdcall for C++)"

    Function2 uses "Cdecl"

    Function3 uses "Fastcall"

    Function4 uses "Native C++ (Thiscall)"

    Function5 uses "Win32 (Stdcall)"

    x64 uses the fastcall calling convention - registers are used for the first four args (rcx, rdx, r8, r9), and the stack is used for additional args.

  • Function 1 is COM (Stdcall for C++)

    Function 2 is Cdecl

    Function 3 is Fastcall

    Function 4 is Native C++ (Thiscall)

    Function 5 is Win32 (Stdcall)

  • Function1:

    COM (Stdcall for C++)

    pComObject->QueryInterface

    Function2:

    __cdecl

    Function3:

    Microsoft __fastcall (not Borland)

    Function4:

    thiscall

    Function5:

    __stdcall

  • Function1 - COM

    Function2 - Cdecl

    Function3 - FastCall

    Function4 - C++

    Function5 - stdcall

  • Com, Cdecl, Fastcall, stdcall, thiscall

    Also, it should be noted that in X64, fast call uses rcx(r6c), rdx(r7x), r8x, r9x, then the stack. Not to mention that how some view register preservation is/should be part of calling conventions.

  • Too Bad we can't edit comments, oh well.

    Corrected:

    COM, CDelc, FastCall, Thiscall, StdCall

  • fn1 = COM

    fn2 = Cdecl

    fn3 = fastcall

    fn4 = Native C++

    fn5 = Win32

    Bonus: rcx = arg1, rdx = arg2, r8 = arg3, r9 = arg4, rest pushed on stack right to left. Stack spill space always reserved for the parameters passed through registers.

  • Hi,

    My answers are:

    1 - COM (Stdcall for C++)

    2 - Cdecl

    3 - Fastcall

    4 - Native C++ (Thiscall)

    5 - Win32 (Stdcall)

    The x64 calling convention passes the first four parameters in registers and the remaining parameters are pushed onto stack.

    -George

  • Function1  - COM

    Function2  - cdecl

    Function3  - fastcall

    Function4  - thiscall

    Function5  - stdcall

    On x64, the parameters are passed to rcx, rdx, r8, r9 and then the stack as in rsp+8, rsp+0x10, and so on.

    rcx stores the this pointer where available.

Page 1 of 2 (19 items) 12