Calling Conventions in C/C++

In C/C++ programming, a calling convention is a set of rules that specify how a function will be called. You might have seen keywords like __cdecl or __stdcall when you get linking errors. For example:

error LNK2019: unresolved external symbol "void __cdecl A(void)" (?A@@YAXXZ) referenced in function _main

Here, we see __cdecl being used. It is one of the calling conventions in C/C++. You can also see code like this in 3rd party libraries. For example:

extern __m128i __cdecl _mm256_mask_cvtepi32_epi16(__m128i, __mmask8, __m256i);

What are caller and callee?

When we call a function, a stack frame is allocated for that function, arguments are passed to the function, and after the function does its work, the allocated stack frame is deallocated and control is passed to the calling function.

The function that calls the subroutine is called the caller. The function that gets called(i.e. subroutine) by the caller is called the callee.

// C++ Program to illustrate the caller and callee
#include <iostream>

// callee
void func() { std::cout << "Geeks"; }

// caller
int main()
{

    // function call
    func();

    return 0;
}

Geeks

In the above code, main() is the caller and func() is the callee.

Calling Conventions

The Calling Conventions in C/C++ are the guidelines that determine:

• How the arguments are passed onto the stack.
• Who will clear the stack, caller or callee?
• What registers will be used and how.

Syntax

The following syntax shows how to use the calling convention:

return_type calling_convention function_name {
    // statements
}

The C++ code gets converted to object code at the end of the compilation stage. Then we get the object file. The object files are linked together to create a binary file(exe, lib, dll).

Before the creation of the object file, we can tell the compiler to stop and give us the .asm file. This is the assembly file which will get converted to an object file. Different calling conventions produce different assembly codes. For GCC, the -S can be used for this purpose. Just pass this flag while compiling the code.

gcc -S sourceFileName.c

Different Calling Conventions

There are many calling conventions for different platforms. We are going to look at 32-bit x86 calling conventions.

1. __cdecl
2. __stdcall
3. __fastcall
4. __thiscall (C++ only)

Example of Calling Convention

The following program illustrates how to use the calling convention.

// C++ Program to demonstrate the calling convention
#include <iostream>

// __cdecl calling convention
int __cdecl cdeclAdd(int a, int b)
{
    int c = a + b;
    return c;
}

// __stdcall calling convention
int __stdcall stdcallAdd(int a, int b)
{
    int c = a + b;
    return c;
}

// __fastcall calling convention
int __fastcall fastcallAdd(int a, int b, int c, int d)
{
    int e = a + b + c + d;
    return e;
}

// __thiscall calling convention
class Temp {
public:
    int __thiscall thiscallAdd(int a, int b)
    {
        int c = a + b;
        return c;
    }
};

// driver code
int main()
{
    int result;
    Temp obj;

    // Function calls and output
    result = cdeclAdd(1, 2);
    std::cout << "Result: " << result << std::endl;

    result = stdcallAdd(3, 4);
    std::cout << "Result: " << result << std::endl;

    result = fastcallAdd(7, 8, 9, 10);
    std::cout << "Result: " << result << std::endl;

    result = obj.thiscallAdd(5, 6);
    std::cout << "Result: " << result << std::endl;
}

Output

Result: 3
Result: 7
Result: 34
Result: 11

Here we have 4 functions: cdeclAdd(), stdcallAdd(), fastcallAdd(), and thiscallAdd() with calling conventions __cdecl, __stdcall, __fastcall, and __thiscall respectively. They are being called by the caller i.e. main(). Let's understand each of them one by one.

__cdecl

The __cdecl calling convention is the default calling convention in C/C++. In this calling convention:

• The arguments are pushed from Right to Left (so that the first argument is nearest to the top-of-stack).
• Caller cleans the stack.
• Creates larger executables than __stdcall, because it requires each function call to include stack cleanup code.

As the caller cleans the stack in this convention, we can provide variable arguments to the functions with the __cdecl calling convention.

Compiling the above program in Debug mode using assembly code, we get:

main()::result = cdeclAdd(1, 2);

line 42:    
    push    2
    push    1
    call    ?cdeclAdd@@YAHHH@Z            ; cdeclAdd
    add    esp, 8
    mov    DWORD PTR _result$[ebp], eax

Note: If you use Release Mode, you might not see the variables passing onto the stack like this as the compiler will produce optimized code.

The function with calling convention __cdecl is cdeclAdd() which is called on line 42 in the main function.

cdeclAdd()

?cdeclAdd@@YAHHH@Z PROC                    ; cdeclAdd, COMDAT
; File c:\users\ruchit\documents\code\visual studio\visual studio\source.cpp
; Line 6
    push    ebp
    mov    ebp, esp
    sub    esp, 204                ; 000000ccH
    push    ebx
    push    esi
    push    edi
    lea    edi, DWORD PTR [ebp-204]
    mov    ecx, 51                    ; 00000033H
    mov    eax, -858993460                ; ccccccccH
    rep stosd
    mov    ecx, OFFSET __F81044A6_source@cpp
    call    @__CheckForDebuggerJustMyCode@4
; Line 7
    mov    eax, DWORD PTR _a$[ebp]
    add    eax, DWORD PTR _b$[ebp]
    mov    DWORD PTR _c$[ebp], eax
; Line 8
    mov    eax, DWORD PTR _c$[ebp]
; Line 9
    pop    edi
    pop    esi
    pop    ebx
    add    esp, 204                ; 000000ccH
    cmp    ebp, esp
    call    __RTC_CheckEsp
    mov    esp, ebp
    pop    ebp
    ret    0
?cdeclAdd@@YAHHH@Z ENDP                    ; cdeclAdd

Explanation: When we look into the assembly code of cdeclAdd(), the last statement is: ret 0. This means that the callee will do nothing to the stack pointer and the control returns to the main().

If we look at the assembly code in main(), after the function call, there is another statement: add esp, 8. When the control is returned to the main, it will add in stack pointer(esp) 8 bytes. Adding in memory means that we are popping from the stack or clearing up the stack. If we subtract from esp, this means we are pushing to the stack. The stack grows in reverse order with respect to memory.

Note: Here we are incrementing 8 bytes in esp because we are popping 2 integer variables passed. Since each integer variable in the 32-bit arch is 4 bytes, we are clearing 8-byte memory for both variables.

This means that the caller (i.e. main) will clear the stack and not the callee (i.e. cdeclAdd).

__stdcall

This is a Microsoft-specific calling convention used by Win32 API functions. In this convention:

• Arguments are pushed from Right to Left.
• Callee cleans the stack.

main()::result = stdcallAdd(3, 4);

line 45:    
    push    4
    push    3
    call    ?stdcallAdd@@YGHHH@Z            ; stdcallAdd
    mov    DWORD PTR _result$[ebp], eax

Like cdeclAdd(), we see that arguments are passed from Right to Left.

In line 42 in the assembly Code, we see that arguments are passed from Right to Left for function call: stdcallAdd(3, 4)

The function with calling convention __stdcall is stdcallAdd() which is called on line 42 in the main function. If we look at the assembly code of main(), after the function call, there is no statement for the stack pointer. It ends with the function call.

stdcallAdd()

?stdcallAdd@@YGHHH@Z PROC                ; stdcallAdd, COMDAT
; File c:\users\ruchit\documents\code\visual studio\visual studio\source.cpp
; Line 13
    push    ebp
    mov    ebp, esp
    sub    esp, 204                ; 000000ccH
    push    ebx
    push    esi
    push    edi
    lea    edi, DWORD PTR [ebp-204]
    mov    ecx, 51                    ; 00000033H
    mov    eax, -858993460                ; ccccccccH
    rep stosd
    mov    ecx, OFFSET __F81044A6_source@cpp
    call    @__CheckForDebuggerJustMyCode@4
; Line 14
    mov    eax, DWORD PTR _a$[ebp]
    add    eax, DWORD PTR _b$[ebp]
    mov    DWORD PTR _c$[ebp], eax
; Line 15
    mov    eax, DWORD PTR _c$[ebp]
; Line 16
    pop    edi
    pop    esi
    pop    ebx
    add    esp, 204                ; 000000ccH
    cmp    ebp, esp
    call    __RTC_CheckEsp
    mov    esp, ebp
    pop    ebp
    ret    8
?stdcallAdd@@YGHHH@Z ENDP                ; stdcallAdd

Explanation: When we look into the assembly code of stdcallAdd(), the last statement is: ret 8. This means that it increments the stack pointer by 8 bytes and pops the 2 variables that were passed to it. Then, the control returns to the main().

So, this means that callee (i.e. stdcallAdd) will clear the stack and not the caller (i.e. main).

__fastcall

In __fastcall calling convention, the arguments are passed to the register if possible.

• The first two arguments are passed in register ECX and EDX. The remaining arguments are passed on the stack from Right to Left.
• Callee cleans the stack.

Going back to the assembly code of the main function, the statement

main()::return = fastcallAdd(7, 8, 9, 10);

; Line 48
    push    10                    ; 0000000aH
    push    9
    mov    edx, 8
    mov    ecx, 7
    call    ?fastcallAdd@@YIHHHHH@Z            ; fastcallAdd
    mov    DWORD PTR _result$[ebp], eax

In fastcallAdd, the arguments are pushed onto the stack in Right to Left order, then the 2 arguments that remain are pushed onto registers EDX and ECX respectively. (Or we can say the first 2 arguments are passed onto ECX and EDX in Left to Right order)

The function with the calling convention __fastcall is fastcallAdd(), which is called on line 48 in the main function. We have seen how the arguments are passed onto the stack. We know that since the caller i.e. main is not doing anything about the stack pointer, it is the callee's job to clean up the stack.

fastcallAdd()

?fastcallAdd@@YIHHHHH@Z PROC                ; fastcallAdd, COMDAT
; _a$ = ecx
; _b$ = edx
; File c:\users\ruchit\documents\code\visual studio\visual studio\source.cpp
; Line 20
    push    ebp
    mov    ebp, esp
    sub    esp, 228                ; 000000e4H
    push    ebx
    push    esi
    push    edi
    push    ecx
    lea    edi, DWORD PTR [ebp-228]
    mov    ecx, 57                    ; 00000039H
    mov    eax, -858993460                ; ccccccccH
    rep stosd
    pop    ecx
    mov    DWORD PTR _b$[ebp], edx
    mov    DWORD PTR _a$[ebp], ecx
    mov    ecx, OFFSET __F81044A6_source@cpp
    call    @__CheckForDebuggerJustMyCode@4
; Line 21
    mov    eax, DWORD PTR _a$[ebp]
    add    eax, DWORD PTR _b$[ebp]
    add    eax, DWORD PTR _c$[ebp]
    add    eax, DWORD PTR _d$[ebp]
    mov    DWORD PTR _e$[ebp], eax
; Line 22
    mov    eax, DWORD PTR _e$[ebp]
; Line 23
    pop    edi
    pop    esi
    pop    ebx
    add    esp, 228                ; 000000e4H
    cmp    ebp, esp
    call    __RTC_CheckEsp
    mov    esp, ebp
    pop    ebp
    ret    8
?fastcallAdd@@YIHHHHH@Z ENDP                ; fastcallAdd

Explanation

Here, we can see that, the stack pointer is incremented with 8 bytes only. We are only popping two 4-byte integers from the stack. This is because the other 2 arguments are passed into ECX and EDX registers. So, we only need to pop the 2 arguments that are on the stack.

__thiscall

The __thiscall calling convention is the default calling convention used by methods inside a class. That is why it is only possible in C++ but not in C. In this convention:

• Arguments are pushed on the stack from Right to Left.
• The this pointer is passed via register ECX, and not on the stack.
• Since we pass this pointer as well, we cannot use this calling convention for non-member functions.
• Callee cleans the stack.

Let us look at the assembly code for main():

result = obj.thiscallAdd(5, 6);

; Line 51
    push    6
    push    5
    lea    ecx, DWORD PTR _obj$[ebp]
    call    ?thiscallAdd@Temp@@QAEHHH@Z        ; Temp::thiscallAdd
    mov    DWORD PTR _result$[ebp], eax

Here, we can see in thiscallAdd, first, the arguments are passed in Right to Left order. Then, the 'this' pointer(which refers to the object itself) is passed on to the ECX register.

The function with calling convention __thiscall is thiscalllAdd() inside the Temp class, which is called on line 51 in the main function.

We need to create a class with a member function 'thiscallAdd' to use the __thiscall convention. This is because __thiscall can only be used for member functions of a class. In fact, you cannot even use the keyword static when using this calling convention. So, __thiscall can only appear on non-static member functions.

thiscallAdd()

?thiscallAdd@Temp@@QAEHHH@Z PROC            ; Temp::thiscallAdd, COMDAT
; _this$ = ecx
; File c:\users\ruchit\documents\code\visual studio\visual studio\source.cpp
; Line 29
    push    ebp
    mov    ebp, esp
    sub    esp, 216                ; 000000d8H
    push    ebx
    push    esi
    push    edi
    push    ecx
    lea    edi, DWORD PTR [ebp-216]
    mov    ecx, 54                    ; 00000036H
    mov    eax, -858993460                ; ccccccccH
    rep stosd
    pop    ecx
    mov    DWORD PTR _this$[ebp], ecx
    mov    ecx, OFFSET __F81044A6_source@cpp
    call    @__CheckForDebuggerJustMyCode@4
; Line 30
    mov    eax, DWORD PTR _a$[ebp]
    add    eax, DWORD PTR _b$[ebp]
    mov    DWORD PTR _c$[ebp], eax
; Line 31
    mov    eax, DWORD PTR _c$[ebp]
; Line 32
    pop    edi
    pop    esi
    pop    ebx
    add    esp, 216                ; 000000d8H
    cmp    ebp, esp
    call    __RTC_CheckEsp
    mov    esp, ebp
    pop    ebp
    ret    8
?thiscallAdd@Temp@@QAEHHH@Z ENDP            ; Temp::thiscallAdd

Explanation: The 'this' in classes is a self-reference and thus, the name of the convention: __thiscall.

When we look at the last statement executed, i.e. ret 8, we see that we are adding 8 bytes to the stack pointer and then returning the control to the main. The callee has popped the 2 arguments passed (4 bytes each)and has cleared up the stack.

Advantages of Calling Conventions

The following are the advantages of calling conventions in C/C++:

• The calling convention helps in the standardization of the compiler's way of function invoking and parameter passing.
• Standardization leads to interoperability between programming languages.
• Calling conventions implements efficient methods to invoke functions according to the requirements.