Chapter 3 - Assembly/3.5 CallingConventions.md
When a function is called you could, theoretically, pass parameters via registers, the stack, or even on disk. You just need to be sure that the function you are calling knows how you are calling it. This isn't too big of a problem if you are using your own functions, but things would get messy when you start using libraries. To solve this problem we have calling conventions that define how parameters are passed to a function, who allocates space for local variables, and who cleans up the stack.
Callee refers to the function being called, and the caller is the function making the call.
There are several different calling conventions including cdecl, syscall, stdcall, fastcall, and many more. Because we are going to be reverse engineering on x64 Windows we will mostly focus on x64 fastcall. When we get into the DLL chapter we will also have to know cdecl. If you do plan on reversing on other platforms, be sure to learn the calling convention(s).
You will usually see a double underscore prefix (__) before a calling convention's name. For example:__fastcall. I won't be doing this because it's not necessary.
Fastcall is the calling convention for x64 Windows. Windows uses a four-register fastcall calling convention by default. Fastcall pushes function parameters backward (right to left). When talking about calling conventions you will hear about something called the "Application Binary Interface" (ABI). The ABI defines various rules for programs such as calling conventions, parameter handling, and more.
void func(int a, int b, float c, int d , float e){
//do something...
return;
}
And i passed on these values:
int main(){
func(1,2,0.1f,4,0.5f);
}
float xmm0 to xmm3). The argument is placed according to the position of the argument within that type's sequence(in our example, 0.1f is third in position and is passed onto the third register xmm2 ).d (which has a value of 4), we examine its data type, identified as int. We then note its position in the argument list, which is fourth. Consequently, we store the value 4 in the fourth integer register, namely r9.All parameters, even ones passed through registers, have space reserved on the stack for them. Also, there is always space made for 4 parameters on the stack even if there are no parameters passed. This space isn't completely wasted because the compiler can, and often will, use it. This space is referred to as shadow space or home space. The Windows x64 calling convention mandates a space of 32 (0x20) bytes be left before every function call.
In assembly one might see this line:
SUB RSP, 0x20
This is the shadow space allocation happening and this indicates to us that a function is going to be called. The reason why its sub instead of add is because the stack grows towards lower addresses as we discussed in [[3.2 MemoryLayout]].
Data on the stack such as local variables and function parameters are often accessed with RBP or RSP. On x64 it's extremely common to see RSP used instead of RBP for parameters. Remember that the first four parameters, even though they are passed via registers, still have space reserved for them on the stack (shadow space). This space is going to be 32 bytes (0x20), 8 bytes for each of the 4 registers.
Here is a very simple example where the numbers 1 to 8 are passed from one function to another function. Notice the order they are put in.
MOV DWORD PTR SS:[RSP + 0x38], 0x8
MOV DWORD PTR SS:[RSP + 0x30], 0x7
MOV DWORD PTR SS:[RSP + 0x28], 0x6
MOV DWORD PTR SS:[RSP + 0x20], 0x5
MOV R9D, 0x4
MOV R8D, 0x3
MOV EDX, 0x2
MOV ECX, 0x1
CALL function
As you can see, the first few parameters are passed via ECX (because it doesn't need the full RCX register), EDX, R8D, and R9D as usual. It passes the rest of the parameters through RSP+0x20, RSP+0x28, etc.
The calling convention mandates one more crucial thing: the stack pointer rsp must be 16-byte aligned (0x10) before every function call (specifically, before a call instruction). This means that the value of rsp must always be a multiple of 16. Initially, when a program starts, rsp is typically aligned correctly, so there's no immediate issue.
However, the call instruction pushes an 8 byte address onto the stack. If rsp is 16 byte aligned before the call, pushing 8 bytes will make rsp 8 byte aligned . This breaks the 16 byte alignment condition. So, you might want to add 8 bytes to rsp so that it's aligned again (8 pushed by call + 8 manually subtracted = 16 bytes). You have to do this if your going to execute call again, which usually you have to.
This is here as extra knowledge, so now whenever you reverse programs and encounter this after
call:sub rsp,8, you know why.
That's the x64 Windows fastcall calling convention for you. Learning your first calling convention is like learning your first programming language. It seems complex and daunting at first, but it's really quite simple. It's typically harder to learn your first calling convention than it is your second or third.
If you want to learn more about this calling convention you can here:
https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=vs-2019
https://docs.microsoft.com/en-us/cpp/build/x64-software-conventions?view=vs-2019
If this lesson was confusing, read through 3.3 Instructions.md then re-read this lesson. I apologize for this but there really isn't a good order to teach this stuff in since it all goes together.
That wasn't too hard, was it? Cdecl is pretty easy and this is your second calling convention so I can exclude many details.
<- Previous Lesson
Next Lesson ->