Calling Conventions in C/C++

In C/C++ programming, a calling convention is a set of rules that specify how a function will be called. You might have seen keywords like __cdecl or __stdcall when you get linking errors. For example:

error LNK2019: unresolved external symbol "void __cdecl A(void)" (?A@@YAXXZ) referenced in function _main

Here, we see __cdecl being used. It is one of the calling conventions in C/C++. You can also see code like this in 3rd party libraries.

For example:

extern __m128i __cdecl _mm256_mask_cvtepi32_epi16(__m128i, __mmask8, __m256i);

What are caller and callee?

When we call a function, a stack frame is allocated for that function, arguments are passed to the function, and after the function does its work, the allocated stack frame is deallocated and control is passed to the calling function.

The function that calls the subroutine is called the caller. The function that gets called(i.e. subroutine) by the caller is called the callee.

Geeks

In the above code, main() is the caller and func() is the callee.

Calling Conventions

The Calling Conventions in C/C++ are the guidelines that determine:

• How the arguments are passed onto the stack.
• Who will clear the stack, caller or callee?
• What registers will be used and how.

Syntax

The following syntax shows how to use the calling convention:

return_type calling_convention function_name {
    // statements
}

The C++ code gets converted to object code at the end of the compilation stage. Then we get the object file. The object files are linked together to create a binary file(exe, lib, dll).

Before the creation of the object file, we can tell the compiler to stop and give us the .asm file. This is the assembly file which will get converted to an object file. Different calling conventions produce different assembly codes. For GCC, the -S can be used for this purpose. Just pass this flag while compiling the code.

gcc -S sourceFileName.c

Different Calling Conventions

There are many calling conventions for different platforms. We are going to look at 32-bit x86 calling conventions.

1. __cdecl
2. __stdcall
3. __fastcall
4. __thiscall (C++ only)

Example of Calling Convention

The following program illustrates how to use the calling convention.

Output

Result: 3
Result: 7
Result: 34
Result: 11

Here we have 4 functions: cdeclAdd(), stdcallAdd(), fastcallAdd(), and thiscallAdd() with calling conventions __cdecl, __stdcall, __fastcall, and __thiscall respectively. They are being called by the caller i.e. main(). Let’s understand each of them one by one.

__cdecl

The __cdecl calling convention is the default calling convention in C/C++. In this calling convention:

• The arguments are pushed from Right to Left (so that the first argument is nearest to the top-of-stack).
• Caller cleans the stack.
• Creates larger executables than __stdcall, because it requires each function call to include stack cleanup code.

As the caller cleans the stack in this convention, we can provide variable arguments to the functions with the __cdecl calling convention.

Compiling the above program in Debug mode using assembly code, we get:

main()::result = cdeclAdd(1, 2);

Note: If you use Release Mode, you might not see the variables passing onto the stack like this as the compiler will produce optimized code.

The function with calling convention __cdecl is cdeclAdd() which is called on line 42 in the main function.

cdeclAdd()

Explanation: When we look into the assembly code of cdeclAdd(), the last statement is: ret 0. This means that the callee will do nothing to the stack pointer and the control returns to the main().

If we look at the assembly code in main(), after the function call, there is another statement: add esp, 8. When the control is returned to the main, it will add in stack pointer(esp) 8 bytes. Adding in memory means that we are popping from the stack or clearing up the stack. If we subtract from esp, this means we are pushing to the stack. The stack grows in reverse order with respect to memory.

Note: Here we are incrementing 8 bytes in esp because we are popping 2 integer variables passed. Since each integer variable in the 32-bit arch is 4 bytes, we are clearing 8-byte memory for both variables.

This means that the caller (i.e. main) will clear the stack and not the callee (i.e. cdeclAdd).

__stdcall

This is a Microsoft-specific calling convention used by Win32 API functions. In this convention:

• Arguments are pushed from Right to Left.
• Callee cleans the stack.

main()::result = stdcallAdd(3, 4);

Like cdeclAdd(), we see that arguments are passed from Right to Left.

In line 42 in the assembly Code, we see that arguments are passed from Right to Left for function call: stdcallAdd(3, 4)

The function with calling convention __stdcall is stdcallAdd() which is called on line 42 in the main function. If we look at the assembly code of main(), after the function call, there is no statement for the stack pointer. It ends with the function call.

stdcallAdd()

Explanation: When we look into the assembly code of stdcallAdd(), the last statement is: ret 8. This means that it increments the stack pointer by 8 bytes and pops the 2 variables that were passed to it. Then, the control returns to the main().

So, this means that callee (i.e. stdcallAdd) will clear the stack and not the caller (i.e. main).

__fastcall

In __fastcall calling convention, the arguments are passed to the register if possible.

• The first two arguments are passed in register ECX and EDX. The remaining arguments are passed on the stack from Right to Left.
• Callee cleans the stack.

Going back to the assembly code of the main function, the statement

main()::return = fastcallAdd(7, 8, 9, 10);

In fastcallAdd, the arguments are pushed onto the stack in Right to Left order, then the 2 arguments that remain are pushed onto registers EDX and ECX respectively. (Or we can say the first 2 arguments are passed onto ECX and EDX in Left to Right order)

The function with the calling convention __fastcall is fastcallAdd(), which is called on line 48 in the main function. We have seen how the arguments are passed onto the stack. We know that since the caller i.e. main is not doing anything about the stack pointer, it is the callee’s job to clean up the stack.

fastcallAdd()

Explanation

Here, we can see that, the stack pointer is incremented with 8 bytes only. We are only popping two 4-byte integers from the stack. This is because the other 2 arguments are passed into ECX and EDX registers. So, we only need to pop the 2 arguments that are on the stack.

__thiscall

The __thiscall calling convention is the default calling convention used by methods inside a class. That is why it is only possible in C++ but not in C. In this convention:

• Arguments are pushed on the stack from Right to Left.
• The this pointer is passed via register ECX, and not on the stack.
• Since we pass this pointer as well, we cannot use this calling convention for non-member functions.
• Callee cleans the stack.

Let us look at the assembly code for main():

result = obj.thiscallAdd(5, 6);

Here, we can see in thiscallAdd, first, the arguments are passed in Right to Left order. Then, the ‘this’ pointer(which refers to the object itself) is passed on to the ECX register.

The function with calling convention __thiscall is thiscalllAdd() inside the Temp class, which is called on line 51 in the main function.

We need to create a class with a member function ‘thiscallAdd’ to use the __thiscall convention. This is because __thiscall can only be used for member functions of a class. In fact, you cannot even use the keyword static when using this calling convention. So, __thiscall can only appear on non-static member functions.

thiscallAdd()

Explanation: The ‘this’ in classes is a self-reference and thus, the name of the convention: __thiscall.

When we look at the last statement executed, i.e. ret 8, we see that we are adding 8 bytes to the stack pointer and then returning the control to the main. The callee has popped the 2 arguments passed (4 bytes each)and has cleared up the stack.

Advantages of Calling Conventions

The following are the advantages of calling conventions in C/C++:

• The calling convention helps in the standardization of the compiler’s way of function invoking and parameter passing.
• Standardization leads to interoperability between programming languages.
• Calling conventions implements efficient methods to invoke functions according to the requirements.