This educational note is prepared to enhance the reader's understanding why there are prolog and epilog in a subroutine, as well as the changes in stack. Only slight understanding of x86 assembly is assumed.
Prepared by Globeriz (Globeriz Project)
It all starts with the stack.
The stacks works in a LIFO(Last-in First-out) basis. This means the last value pushed onto stack will be the first value to pop from the stack. e.g.:
push 3 ;push 0x00000003 to stack
push 2 ;push 0x00000002 to stack
push 1 ;push 0x00000001 to stack
pop eax ;pop last pushed value from stack (i.e. 1) to eax
First, familiarize yourself with push and pop instructions.
push - for pushing a value onto stack
It first decreases esp(stack pointer register) by 4 ( sizeof (DWORD) ), then the value is assigned to the dereferenced value of esp.
In opcodes, push eax is equivalent to:
sub esp, 4 ;esp = esp - 4 (i.e. esp-4 => esp)
mov [esp], eax ;*(DWORD *)esp = eax
Meanwhile in stack:
| Memory Address | Relative to esp | Value | Memory Address | Relative to esp | Value | |
|---|---|---|---|---|---|---|
| X-4 | esp-4 | ? | X-4 | esp | eax | |
| X | esp | a | X | esp+4 | a | |
| X+4 | esp+4 | b | X+4 | esp+8 | b | |
| BEFORE push eax | AFTER push eax | |||||
pop - for popping (getting the last pushed value) from stack
It firsts assign memory/register with the last pushed value on stack ( i.e. [esp] ), then increases esp by 4
In opcodes, pop eax is equivalent to:
mov eax, [esp] ; eax = *(DWORD *)esp
add esp, 4 ; esp = esp + 4
Note how pop eax cancels out the effect of push eax
Meanwhile in stack:
| Memory Address | Relative to esp | Value | Memory Address | Relative to esp | Value | |
|---|---|---|---|---|---|---|
| X-8 | esp-4 | ? | X-8 | esp-8 | ? | |
| X-4 | esp | eax | X-4 | esp-4 | eax | |
| X | esp+4 | a | X | esp | a | |
| X+4 | esp+8 | b | X+4 | esp+4 | b | |
| BEFORE pop eax | AFTER pop eax | |||||
Now, after understanding the push and pop instructions and their relations with the stack, it is time to introduce the C calling convention in x86 assembly.
There are TWO parties involved in a call, namely the caller (main routine) and the callee (subroutine).
The C calling convention of caller is as follows:
STEP P1> Saving the registers (by pushing them onto stack) that will be altered in the subroutine, such that it is possible to restore them after returning from the subroutine. Common examples are eax ecx edx.
STEP P2> Passing the necessary parameters to the subroutine. The parameters are pushed onto stack just before the call instruction, where the pushing order of the parameters (according to C calling convention) should be reverse of the order in the subroutine function (i.e. the LAST parameter of the function should be pushed FIRST)
STEP P3> Using the call instruction to invoke the subroutine function. The call instruction first pushes the return address (eip, or offset to the instruction right after call in caller) onto stack, and then it branches to the subroutine address to execute code in callee.
In opcodes, call eax is equivalent to:
push eip
jmp eax
Then the CPU enters callee (subroutine).
After returning from the subroutine function, (the return value from subroutine can be found in eax register)
STEP E1> Remove (deallocate) the parameters from stack by increasing esp by the total size of parameters.
STEP E2> Restoring saved registers (by popping them from stack) as the registers are believed to be altered in the subroutine N.B. the popping of registers should be the REVERSE of the order they are pushed
The following assembly code demonstrates the role of caller:
...
push eax ;saving registers
push ecx ;
push edx ;
push param3 ;passing parameters
push param2 ;
push param1 ;first parameter pushed last
call _subroutine ;invoke _subroutine(param1, param2, param3)
add esp, 0x0C ;restoring stack to the state before the call
pop edx ;restoring registers
pop ecx ;
pop eax ;first saved register pops last
...
The rules for callee is as follows:
STEP P1> Preserving ebp (base pointer) and then copy the value of esp to ebp. Inside the subroutine, ebp is a copy of initial stack pointer. It is going to be used as a reference point for parameters and local variables in subroutine. The reason behind preserving ebp is that the caller does not expect ebp to be altered in the subroutine, therefore it is necessarily to be done in order to restore its value later on.
STEP P2> Allocating space for local variables, this is done by decreasing esp by the size of local variables required. After this, parameters and local variables can be located at known constant offsets (See Appendix A) from ebp.
STEP P3> Preserve the registers that will be used by the subroutine. Common examples are ebx edi esi
After the steps above (prolog), the CPU proceeds to the main body of subroutine. After executing the main body code, the following steps are followed to return properly to the caller (main routine).
STEP E1> Copy the return value to eax.
STEP E2> Restore the registers saved in prolog by pop instructions (REVERSE of order pushed)
STEP E3> Remove (deallocate) local variables by copying ebp to esp (simply reversing STEP P2)
STEP E4> Right before ret instruction, restore the ebp of caller by popping ebp from stack.
STEP E5> Using the ret instruction to return to caller (main routine). The ret instruction pops the return address from stack and branches to the main routine address to continue executing code in caller.
These steps are known as epilog. After epilog, the CPU returns to caller.
The following assembly code demonstrates the role of callee:
_subroutine:
; prolog
push ebp ;preserving ebp
mov ebp, esp ;preparing ebp as a reference point
sub esp, 0x08 ;allocate space for local variables
push ebx ;preserving registers
push edi
push esi
; end of prolog
... ; main body
;epilog
pop esi ;restoring registers
pop edi
pop ebx
mov esp, ebp ;deallocate local variables
pop ebp ; restoring ebp
ret ; return to main routine
;end of epilog
Appendix A (stack inside subroutine main body, after prolog and before epilog)
| Memory relative to ebp (hex) | Value |
|---|---|
| ebp-14 | esi (saved) |
| ebp-10 | edi (saved) |
| ebp-C | ebx (saved) |
| ebp-8 | local_var2 |
| ebp-4 | local_var1 |
| ebp | ebp (saved) |
| ebp+4 | eip (return address) |
| ebp+8 | param1 |
| ebp+C | param2 |
| ebp+10 | param3 |
Appendix B (entire stack change for calling process)
...;caller
push eax ;saving registers
push ecx ;
push edx ;
push param3 ;passing parameters
push param2 ;
push param1 ;first parameter pushed last
call _subroutine ;invoke _subroutine(param1, param2, param3)
| Memory Address | Relative to esp | Value | Memory Address | Relative to esp | Value | |
|---|---|---|---|---|---|---|
| X-4 | esp-4 | ? | X-4 | esp | eax | |
| X | esp | ? | X | esp+4 | ? | |
| Original state | AFTER push eax | |||||
| Memory Address | Relative to esp | Value | Memory Address | Relative to esp | Value | |
|---|---|---|---|---|---|---|
| X-C | esp-4 | ? | X-C | esp | edx | |
| X-8 | esp | ecx | X-8 | esp+4 | ecx | |
| X-4 | esp+4 | eax | X-4 | esp+8 | eax | |
| X | esp+8 | ? | X | esp+C | ? | |
| AFTER push ecx | AFTER push edx | |||||
| Memory Address | Relative to esp | Value | Memory Address | Relative to esp | Value | |
|---|---|---|---|---|---|---|
| X-18 | esp-8 | ? | X-18 | esp-4 | ? | |
| X-14 | esp-4 | ? | X-14 | esp | param2 | |
| X-10 | esp | param3 | X-10 | esp+4 | param3 | |
| X-C | esp+4 | edx | X-C | esp+8 | edx | |
| X-8 | esp+8 | ecx | X-8 | esp+C | ecx | |
| X-4 | esp+C | eax | X-4 | esp+10 | eax | |
| X | esp+10 | ? | X | esp+14 | ? | |
| AFTER push parma3 | AFTER push param2 | |||||
| Memory Address | Relative to esp | Value | Memory Address | Relative to esp | Value | |
|---|---|---|---|---|---|---|
| X-1C | esp-4 | ? | X-1C | esp | eip | |
| X-18 | esp | param1 | X-18 | esp+4 | param1 | |
| X-14 | esp+4 | param2 | X-14 | esp+8 | param2 | |
| X-10 | esp+8 | param3 | X-10 | esp+C | param3 | |
| X-C | esp+C | edx | X-C | esp+10 | edx | |
| X-8 | esp+10 | ecx | X-8 | esp+14 | ecx | |
| X-4 | esp+14 | eax | X-4 | esp+18 | eax | |
| X | esp+18 | ? | X | esp+1C | ? | |
| AFTER push param1 | AFTER call _subroutine | |||||
; prolog
push ebp ;preserving ebp
mov ebp, esp ;preparing ebp as a reference point
sub esp, 0x08 ;allocate space for local variables
push ebx ;preserving registers
push edi
push esi
; end of prolog
...
| Memory Address | Relative to esp | Value | Memory Address | Relative to esp | Relative to ebp | Value | |
|---|---|---|---|---|---|---|---|
| X-24 | esp-4 | ? | X-24 | esp-4 | ebp-4 | ? | |
| X-20 | esp | ebp | X-20 | esp | ebp | ebp | |
| X-1C | esp+4 | eip | X-1C | esp+4 | ebp+4 | eip | |
| X-18 | esp+8 | param1 | X-18 | esp+8 | ebp+8 | param1 | |
| X-14 | esp+C | param2 | X-14 | esp+C | ebp+C | param2 | |
| X-10 | esp+10 | param3 | X-10 | esp+10 | ebp+10 | param3 | |
| X-C | esp+14 | edx | X-C | esp+14 | ebp+14 | edx | |
| X-8 | esp+18 | ecx | X-8 | esp+18 | ebp+18 | ecx | |
| X-4 | esp+1C | eax | X-4 | esp+1C | ebp+1C | eax | |
| X | esp+20 | ? | X | esp+20 | ebp+20 | ? | |
| AFTER push ebp | AFTER mov ebp, esp | ||||||
| Memory Address | Relative to esp | Relative to ebp | Value | Memory Address | Relative to esp | Relative to ebp | Value | |
|---|---|---|---|---|---|---|---|---|
| X-2C | esp-4 | ebp-C | ? | X-2C | esp | ebp-C | ebx | |
| X-28 | esp | ebp-8 | ? | X-28 | esp+4 | ebp-8 | ?(var2) | |
| X-24 | esp+4 | ebp-4 | ? | X-24 | esp+8 | ebp-4 | ?(var1) | |
| X-20 | esp+8 | ebp | ebp | X-20 | esp+C | ebp | ebp | |
| X-1C | esp+C | ebp+4 | eip | X-1C | esp+10 | ebp+4 | eip | |
| X-18 | esp+10 | ebp+8 | param1 | X-18 | esp+14 | ebp+8 | param1 | |
| X-14 | esp+14 | ebp+C | param2 | X-14 | esp+18 | ebp+C | param2 | |
| X-10 | esp+18 | ebp+10 | param3 | X-10 | esp+1C | ebp+10 | param3 | |
| X-C | esp+1C | ebp+14 | edx | X-C | esp+20 | ebp+14 | edx | |
| X-8 | esp+20 | ebp+18 | ecx | X-8 | esp+24 | ebp+18 | ecx | |
| X-4 | esp+24 | ebp+1C | eax | X-4 | esp+28 | ebp+1C | eax | |
| X | esp+28 | ebp+20 | ? | X | esp+2C | ebp+20 | ? | |
| AFTER sub esp, 0x08 | AFTER push ebx | |||||||
| Memory Address | Relative to esp | Relative to ebp | Value | Memory Address | Relative to esp | Relative to ebp | Value | |
|---|---|---|---|---|---|---|---|---|
| X-38 | esp-8 | ebp-18 | ? | X-38 | esp-4 | ebp-18 | ? | |
| X-34 | esp-4 | ebp-14 | ? | X-34 | esp | ebp-14 | esi | |
| X-30 | esp | ebp-10 | edi | X-30 | esp+4 | ebp-10 | edi | |
| X-2C | esp+4 | ebp-C | ebx | X-2C | esp+8 | ebp-C | ebx | |
| X-28 | esp+8 | ebp-8 | var2 | X-28 | esp+C | ebp-8 | var2 | |
| X-24 | esp+C | ebp-4 | var1 | X-24 | esp+10 | ebp-4 | var1 | |
| X-20 | esp+10 | ebp | ebp | X-20 | esp+14 | ebp | ebp | |
| X-1C | esp+14 | ebp+4 | eip | X-1C | esp+18 | ebp+4 | eip | |
| X-18 | esp+18 | ebp+8 | param1 | X-18 | esp+1C | ebp+8 | param1 | |
| X-14 | esp+1C | ebp+C | param2 | X-14 | esp+20 | ebp+C | param2 | |
| X-10 | esp+20 | ebp+10 | param3 | X-10 | esp+24 | ebp+10 | param3 | |
| X-C | esp+24 | ebp+14 | edx | X-C | esp+28 | ebp+14 | edx | |
| X-8 | esp+28 | ebp+18 | ecx | X-8 | esp+2C | ebp+18 | ecx | |
| X-4 | esp+2C | ebp+1C | eax | X-4 | esp+30 | ebp+1C | eax | |
| X | esp+30 | ebp+20 | ? | X | esp+34 | ebp+20 | ? | |
| AFTER push edi | AFTER push esi | |||||||
...
;epilog
pop esi ;restoring registers
pop edi
pop ebx
mov esp, ebp ;deallocate local variables
pop ebp ; restoring ebp
ret ; return to main routine
;end of epilog
| Memory Address | Relative to esp | Relative to ebp | Value | Memory Address | Relative to esp | Relative to ebp | Value | |
|---|---|---|---|---|---|---|---|---|
| X-38 | esp-8 | ebp-18 | ? | X-38 | esp-C | ebp-18 | ? | |
| X-34 | esp-4 | ebp-14 | esi(x) | X-34 | esp-8 | ebp-14 | esi(x) | |
| X-30 | esp | ebp-10 | edi | X-30 | esp-4 | ebp-10 | edi(x) | |
| X-2C | esp+4 | ebp-C | ebx | X-2C | esp | ebp-C | ebx | |
| X-28 | esp+8 | ebp-8 | var2 | X-28 | esp+4 | ebp-8 | var2 | |
| X-24 | esp+C | ebp-4 | var1 | X-24 | esp+8 | ebp-4 | var1 | |
| X-20 | esp+10 | ebp | ebp | X-20 | esp+C | ebp | ebp | |
| X-1C | esp+14 | ebp+4 | eip | X-1C | esp+10 | ebp+4 | eip | |
| X-18 | esp+18 | ebp+8 | param1 | X-18 | esp+14 | ebp+8 | param1 | |
| X-14 | esp+1C | ebp+C | param2 | X-14 | esp+18 | ebp+C | param2 | |
| X-10 | esp+20 | ebp+10 | param3 | X-10 | esp+1C | ebp+10 | param3 | |
| X-C | esp+24 | ebp+14 | edx | X-C | esp+20 | ebp+14 | edx | |
| X-8 | esp+28 | ebp+18 | ecx | X-8 | esp+24 | ebp+18 | ecx | |
| X-4 | esp+2C | ebp+1C | eax | X-4 | esp+28 | ebp+1C | eax | |
| X | esp+30 | ebp+20 | ? | X | esp+2C | ebp+20 | ? | |
| AFTER pop esi | AFTER pop edi | |||||||
| Memory Address | Relative to esp | Relative to ebp | Value | Memory Address | Relative to esp | Relative to ebp | Value | |
|---|---|---|---|---|---|---|---|---|
| X-34 | esp-C | ebp-14 | esi(x) | X-34 | esp-14 | ebp-14 | esi(x) | |
| X-30 | esp-8 | ebp-10 | edi(x) | X-30 | esp-10 | ebp-10 | edi(x) | |
| X-2C | esp-4 | ebp-C | ebx(x) | X-2C | esp-C | ebp-C | ebx(x) | |
| X-28 | esp | ebp-8 | var2 | X-28 | esp-8 | ebp-8 | var2(x) | |
| X-24 | esp+4 | ebp-4 | var1 | X-24 | esp-4 | ebp-4 | var1(x) | |
| X-20 | esp+8 | ebp | ebp | X-20 | esp | ebp | ebp | |
| X-1C | esp+C | ebp+4 | eip | X-1C | esp+4 | ebp+4 | eip | |
| X-18 | esp+10 | ebp+8 | param1 | X-18 | esp+8 | ebp+8 | param1 | |
| X-14 | esp+14 | ebp+C | param2 | X-14 | esp+C | ebp+C | param2 | |
| X-10 | esp+18 | ebp+10 | param3 | X-10 | esp+10 | ebp+10 | param3 | |
| X-C | esp+1C | ebp+14 | edx | X-C | esp+14 | ebp+14 | edx | |
| X-8 | esp+20 | ebp+18 | ecx | X-8 | esp+18 | ebp+18 | ecx | |
| X-4 | esp+24 | ebp+1C | eax | X-4 | esp+1C | ebp+1C | eax | |
| X | esp+28 | ebp+20 | ? | X | esp+20 | ebp+20 | ? | |
| AFTER pop ebx | AFTER mov esp, ebp | |||||||
| Memory Address | Relative to esp | Value | Memory Address | Relative to esp | Value | |
|---|---|---|---|---|---|---|
| X-2C | esp-10 | ebx(x) | X-2C | esp-14 | ebx(x) | |
| X-28 | esp-C | var2(x) | X-28 | esp-10 | var2(x) | |
| X-24 | esp-8 | var1(x) | X-24 | esp-C | var1(x) | |
| X-20 | esp-4 | ebp(x) | X-20 | esp-8 | ebp(x) | |
| X-1C | esp | eip | X-1C | esp-4 | eip(x) | |
| X-18 | esp+4 | param1 | X-18 | esp | param1 | |
| X-14 | esp+8 | param2 | X-14 | esp+4 | param2 | |
| X-10 | esp+C | param3 | X-10 | esp+8 | param3 | |
| X-C | esp+10 | edx | X-C | esp+C | edx | |
| X-8 | esp+14 | ecx | X-8 | esp+10 | ecx | |
| X-4 | esp+18 | eax | X-4 | esp+14 | eax | |
| X | esp+1C | ? | X | esp+18 | ? | |
| AFTER pop ebp | AFTER ret | |||||
add esp, 0x0C ;restoring stack to the state before the call
pop edx ;restoring registers
pop ecx ;
pop eax ;first saved register pops last
...
| Memory Address | Relative to esp | Value | Memory Address | Relative to esp | Value | |
|---|---|---|---|---|---|---|
| X-1C | esp-10 | eip(x) | X-1C | esp-14 | eip(x) | |
| X-18 | esp-C | param1(x) | X-18 | esp-10 | param1(x) | |
| X-14 | esp-8 | param2(x) | X-14 | esp-C | param2(x) | |
| X-10 | esp-4 | param3(x) | X-10 | esp-8 | param3(x) | |
| X-C | esp | edx | X-C | esp-4 | edx(x) | |
| X-8 | esp+4 | ecx | X-8 | esp | ecx | |
| X-4 | esp+8 | eax | X-4 | esp+4 | eax | |
| X | esp+C | ? | X | esp+8 | ? | |
| AFTER sub esp, 0x0C | AFTER pop edx | |||||
| Memory Address | Relative to esp | Value | Memory Address | Relative to esp | Value | |
|---|---|---|---|---|---|---|
| X-C | esp-8 | edx(x) | X-C | esp-C | edx(x) | |
| X-8 | esp-4 | ecx(x) | X-8 | esp-8 | ecx(x) | |
| X-4 | esp | eax | X-4 | esp-4 | eax(x) | |
| X | esp+4 | ? | X | esp | ? | |
| AFTER pop ecx | AFTER pop eax (Original state) | |||||
No comments:
Post a Comment