One major part of x86 assembly is the ability to invoke subrountines.
This educational note is prepared to enhance the reader's understanding why there are prolog and epilog in a subroutine, as well as the changes in stack. Only slight understanding of x86 assembly is assumed.
Prepared by Globeriz (Globeriz Project)
It all starts with the stack.
The stacks works in a LIFO(Last-in First-out) basis. This means the last value pushed onto stack will be the first value to pop from the stack. e.g.:
push 3 ;push 0x00000003 to stack
push 2 ;push 0x00000002 to stack
push 1 ;push 0x00000001 to stack
pop eax ;pop last pushed value from stack (i.e. 1) to eax
First, familiarize yourself with
push and
pop instructions.
push - for pushing a value onto stack
It first decreases
esp(stack pointer register) by 4 ( sizeof (DWORD) ), then the value is assigned to the dereferenced value of
esp.
In opcodes,
push eax is equivalent to:
sub esp, 4 ;esp = esp - 4 (i.e. esp-4 => esp)
mov [esp], eax ;*(DWORD *)esp = eax
Meanwhile in stack:
| Memory Address |
Relative to esp |
Value |
|
Memory Address |
Relative to esp |
Value |
| X-4 |
esp-4 |
? |
|
X-4 |
esp |
eax |
| X |
esp |
a |
|
X |
esp+4 |
a |
| X+4 |
esp+4 |
b |
|
X+4 |
esp+8 |
b |
| BEFORE push eax |
|
AFTER push eax |
N.B.
push eax does not alter the value of
eax after execution.
pop - for popping (getting the last pushed value) from stack
It firsts assign memory/register with the last pushed value on stack ( i.e.
[esp] ), then increases
esp by 4
In opcodes,
pop eax is equivalent to:
mov eax, [esp] ; eax = *(DWORD *)esp
add esp, 4 ; esp = esp + 4
Note how
pop eax cancels out the effect of
push eax
Meanwhile in stack:
| Memory Address |
Relative to esp |
Value |
|
Memory Address |
Relative to esp |
Value |
| X-8 |
esp-4 |
? |
X-8 |
esp-8 |
? |
| X-4 |
esp |
eax |
X-4 |
esp-4 |
eax |
| X |
esp+4 |
a |
X |
esp |
a |
| X+4 |
esp+8 |
b |
X+4 |
esp+4 |
b |
| BEFORE pop eax |
AFTER pop eax |
Now, after understanding the
push and
pop instructions and their relations with the
stack, it is time to introduce the
C calling convention in x86 assembly.
There are TWO parties involved in a call, namely the
caller (main routine) and the
callee (subroutine).
The C calling convention of
caller is as follows:
STEP
P1> Saving the registers (
by pushing them onto stack) that will be altered in the subroutine, such that it is possible to restore them after returning from the subroutine. Common examples are
eax ecx edx.
STEP
P2> Passing the necessary parameters to the subroutine. The parameters are pushed onto stack just before the
call instruction, where the pushing order of the parameters (according to C calling convention) should be reverse of the order in the subroutine function (i.e.
the LAST parameter of the function should be pushed FIRST)
STEP
P3> Using the
call instruction to invoke the subroutine function. The
call instruction first pushes the return address (
eip, or offset to the instruction right after
call in caller) onto stack, and then it branches to the subroutine address to execute code in callee.
In opcodes,
call eax is equivalent to:
push eip
jmp eax
Then the CPU enters
callee (subroutine).
After returning from the subroutine function, (the
return value from subroutine can be found in
eax register)
STEP
E1> Remove (deallocate) the parameters from stack by
increasing esp by the total size of parameters.
STEP
E2> Restoring saved registers (
by popping them from stack) as the registers are believed to be altered in the subroutine
N.B. the popping of registers should be the REVERSE of the order they are pushed
The following assembly code demonstrates the role of
caller:
...
push eax ;saving registers
push ecx ;
push edx ;
push param3 ;passing parameters
push param2 ;
push param1 ;first parameter pushed last
call _subroutine ;invoke _subroutine(param1, param2, param3)
add esp, 0x0C ;restoring stack to the state before the call
pop edx ;restoring registers
pop ecx ;
pop eax ;first saved register pops last
...
The rules for
callee is as follows:
STEP
P1> Preserving
ebp (base pointer) and then copy the value of
esp to
ebp. Inside the subroutine,
ebp is a copy of initial stack pointer. It is going to be used as a reference point for parameters and local variables in subroutine. The reason behind preserving ebp is that the caller does not expect
ebp to be altered in the subroutine, therefore it is necessarily to be done in order to restore its value later on.
STEP
P2> Allocating space for local variables, this is done by decreasing
esp by the size of local variables required. After this,
parameters and local variables can be located at
known constant offsets (See
Appendix A) from
ebp.
STEP
P3> Preserve the registers that will be used by the subroutine. Common examples are
ebx edi esi
After the steps above (
prolog), the CPU proceeds to the main
body of subroutine. After executing the main body code, the following steps are followed to
return properly to the
caller (main routine).
STEP
E1> Copy the
return value to
eax.
STEP
E2> Restore the registers saved in prolog by
pop instructions (REVERSE of order pushed)
STEP
E3> Remove (deallocate) local variables by copying
ebp to
esp (simply reversing STEP P2)
STEP
E4> Right before
ret instruction, restore the
ebp of
caller by popping
ebp from stack.
STEP
E5> Using the
ret instruction to return to
caller (main routine). The
ret instruction
pops the return address from stack and branches to the main routine address to continue executing code in
caller.
These steps are known as
epilog. After epilog, the CPU returns to
caller.
The following assembly code demonstrates the role of
callee:
_subroutine:
; prolog
push ebp ;preserving ebp
mov ebp, esp ;preparing ebp as a reference point
sub esp, 0x08 ;allocate space for local variables
push ebx ;preserving registers
push edi
push esi
; end of prolog
... ; main body
;epilog
pop esi ;restoring registers
pop edi
pop ebx
mov esp, ebp ;deallocate local variables
pop ebp ; restoring ebp
ret ; return to main routine
;end of epilog
Appendix A (stack inside subroutine main body, after prolog and before epilog)
| Memory relative to ebp (hex) |
Value |
| ebp-14 |
esi (saved) |
| ebp-10 |
edi (saved) |
| ebp-C |
ebx (saved) |
| ebp-8 |
local_var2 |
| ebp-4 |
local_var1 |
| ebp |
ebp (saved) |
| ebp+4 |
eip (return address) |
| ebp+8 |
param1 |
| ebp+C |
param2 |
| ebp+10 |
param3 |
Appendix B (entire stack change for calling process)
...;caller
push eax ;saving registers
push ecx ;
push edx ;
push param3 ;passing parameters
push param2 ;
push param1 ;first parameter pushed last
call _subroutine ;invoke _subroutine(param1, param2, param3)
| Memory Address |
Relative to esp |
Value |
|
Memory Address |
Relative to esp |
Value |
| X-4 |
esp-4 |
? |
|
X-4 |
esp |
eax |
| X |
esp |
? |
|
X |
esp+4 |
? |
| Original state |
|
AFTER push eax |
| Memory Address |
Relative to esp |
Value |
|
Memory Address |
Relative to esp |
Value |
| X-C |
esp-4 |
? |
|
X-C |
esp |
edx |
| X-8 |
esp |
ecx |
|
X-8 |
esp+4 |
ecx |
| X-4 |
esp+4 |
eax |
|
X-4 |
esp+8 |
eax |
| X |
esp+8 |
? |
|
X |
esp+C |
? |
| AFTER push ecx |
|
AFTER push edx |
| Memory Address |
Relative to esp |
Value |
|
Memory Address |
Relative to esp |
Value |
| X-18 |
esp-8 |
? |
|
X-18 |
esp-4 |
? |
| X-14 |
esp-4 |
? |
|
X-14 |
esp |
param2 |
| X-10 |
esp |
param3 |
|
X-10 |
esp+4 |
param3 |
| X-C |
esp+4 |
edx |
|
X-C |
esp+8 |
edx |
| X-8 |
esp+8 |
ecx |
|
X-8 |
esp+C |
ecx |
| X-4 |
esp+C |
eax |
|
X-4 |
esp+10 |
eax |
| X |
esp+10 |
? |
|
X |
esp+14 |
? |
| AFTER push parma3 |
|
AFTER push param2 |
| Memory Address |
Relative to esp |
Value |
|
Memory Address |
Relative to esp |
Value |
| X-1C |
esp-4 |
? |
|
X-1C |
esp |
eip |
| X-18 |
esp |
param1 |
|
X-18 |
esp+4 |
param1 |
| X-14 |
esp+4 |
param2 |
|
X-14 |
esp+8 |
param2 |
| X-10 |
esp+8 |
param3 |
|
X-10 |
esp+C |
param3 |
| X-C |
esp+C |
edx |
|
X-C |
esp+10 |
edx |
| X-8 |
esp+10 |
ecx |
|
X-8 |
esp+14 |
ecx |
| X-4 |
esp+14 |
eax |
|
X-4 |
esp+18 |
eax |
| X |
esp+18 |
? |
|
X |
esp+1C |
? |
| AFTER push param1 |
|
AFTER call _subroutine |
Inside subroutine:
; prolog
push ebp ;preserving ebp
mov ebp, esp ;preparing ebp as a reference point
sub esp, 0x08 ;allocate space for local variables
push ebx ;preserving registers
push edi
push esi
; end of prolog
...
| Memory Address |
Relative to esp |
Value |
|
Memory Address |
Relative to esp |
Relative to ebp |
Value |
| X-24 |
esp-4 |
? |
X-24 |
esp-4 |
ebp-4 |
? |
| X-20 |
esp |
ebp |
X-20 |
esp |
ebp |
ebp |
| X-1C |
esp+4 |
eip |
X-1C |
esp+4 |
ebp+4 |
eip |
| X-18 |
esp+8 |
param1 |
X-18 |
esp+8 |
ebp+8 |
param1 |
| X-14 |
esp+C |
param2 |
X-14 |
esp+C |
ebp+C |
param2 |
| X-10 |
esp+10 |
param3 |
X-10 |
esp+10 |
ebp+10 |
param3 |
| X-C |
esp+14 |
edx |
X-C |
esp+14 |
ebp+14 |
edx |
| X-8 |
esp+18 |
ecx |
X-8 |
esp+18 |
ebp+18 |
ecx |
| X-4 |
esp+1C |
eax |
X-4 |
esp+1C |
ebp+1C |
eax |
| X |
esp+20 |
? |
X |
esp+20 |
ebp+20 |
? |
| AFTER push ebp |
AFTER mov ebp, esp |
| Memory Address |
Relative to esp |
Relative to ebp |
Value |
|
Memory Address |
Relative to esp |
Relative to ebp |
Value |
| X-2C |
esp-4 |
ebp-C |
? |
X-2C |
esp |
ebp-C |
ebx |
| X-28 |
esp |
ebp-8 |
? |
X-28 |
esp+4 |
ebp-8 |
?(var2) |
| X-24 |
esp+4 |
ebp-4 |
? |
X-24 |
esp+8 |
ebp-4 |
?(var1) |
| X-20 |
esp+8 |
ebp |
ebp |
X-20 |
esp+C |
ebp |
ebp |
| X-1C |
esp+C |
ebp+4 |
eip |
X-1C |
esp+10 |
ebp+4 |
eip |
| X-18 |
esp+10 |
ebp+8 |
param1 |
X-18 |
esp+14 |
ebp+8 |
param1 |
| X-14 |
esp+14 |
ebp+C |
param2 |
X-14 |
esp+18 |
ebp+C |
param2 |
| X-10 |
esp+18 |
ebp+10 |
param3 |
X-10 |
esp+1C |
ebp+10 |
param3 |
| X-C |
esp+1C |
ebp+14 |
edx |
X-C |
esp+20 |
ebp+14 |
edx |
| X-8 |
esp+20 |
ebp+18 |
ecx |
X-8 |
esp+24 |
ebp+18 |
ecx |
| X-4 |
esp+24 |
ebp+1C |
eax |
X-4 |
esp+28 |
ebp+1C |
eax |
| X |
esp+28 |
ebp+20 |
? |
X |
esp+2C |
ebp+20 |
? |
| AFTER sub esp, 0x08 |
AFTER push ebx |
| Memory Address |
Relative to esp |
Relative to ebp |
Value |
|
Memory Address |
Relative to esp |
Relative to ebp |
Value |
| X-38 |
esp-8 |
ebp-18 |
? |
X-38 |
esp-4 |
ebp-18 |
? |
| X-34 |
esp-4 |
ebp-14 |
? |
X-34 |
esp |
ebp-14 |
esi |
| X-30 |
esp |
ebp-10 |
edi |
X-30 |
esp+4 |
ebp-10 |
edi |
| X-2C |
esp+4 |
ebp-C |
ebx |
X-2C |
esp+8 |
ebp-C |
ebx |
| X-28 |
esp+8 |
ebp-8 |
var2 |
X-28 |
esp+C |
ebp-8 |
var2 |
| X-24 |
esp+C |
ebp-4 |
var1 |
X-24 |
esp+10 |
ebp-4 |
var1 |
| X-20 |
esp+10 |
ebp |
ebp |
X-20 |
esp+14 |
ebp |
ebp |
| X-1C |
esp+14 |
ebp+4 |
eip |
X-1C |
esp+18 |
ebp+4 |
eip |
| X-18 |
esp+18 |
ebp+8 |
param1 |
X-18 |
esp+1C |
ebp+8 |
param1 |
| X-14 |
esp+1C |
ebp+C |
param2 |
X-14 |
esp+20 |
ebp+C |
param2 |
| X-10 |
esp+20 |
ebp+10 |
param3 |
X-10 |
esp+24 |
ebp+10 |
param3 |
| X-C |
esp+24 |
ebp+14 |
edx |
X-C |
esp+28 |
ebp+14 |
edx |
| X-8 |
esp+28 |
ebp+18 |
ecx |
X-8 |
esp+2C |
ebp+18 |
ecx |
| X-4 |
esp+2C |
ebp+1C |
eax |
X-4 |
esp+30 |
ebp+1C |
eax |
| X |
esp+30 |
ebp+20 |
? |
X |
esp+34 |
ebp+20 |
? |
| AFTER push edi |
AFTER push esi |
Then it enters main body of subroutine. After the main body:
...
;epilog
pop esi ;restoring registers
pop edi
pop ebx
mov esp, ebp ;deallocate local variables
pop ebp ; restoring ebp
ret ; return to main routine
;end of epilog
| Memory Address |
Relative to esp |
Relative to ebp |
Value |
|
Memory Address |
Relative to esp |
Relative to ebp |
Value |
| X-38 |
esp-8 |
ebp-18 |
? |
X-38 |
esp-C |
ebp-18 |
? |
| X-34 |
esp-4 |
ebp-14 |
esi(x) |
X-34 |
esp-8 |
ebp-14 |
esi(x) |
| X-30 |
esp |
ebp-10 |
edi |
X-30 |
esp-4 |
ebp-10 |
edi(x) |
| X-2C |
esp+4 |
ebp-C |
ebx |
X-2C |
esp |
ebp-C |
ebx |
| X-28 |
esp+8 |
ebp-8 |
var2 |
X-28 |
esp+4 |
ebp-8 |
var2 |
| X-24 |
esp+C |
ebp-4 |
var1 |
X-24 |
esp+8 |
ebp-4 |
var1 |
| X-20 |
esp+10 |
ebp |
ebp |
X-20 |
esp+C |
ebp |
ebp |
| X-1C |
esp+14 |
ebp+4 |
eip |
X-1C |
esp+10 |
ebp+4 |
eip |
| X-18 |
esp+18 |
ebp+8 |
param1 |
X-18 |
esp+14 |
ebp+8 |
param1 |
| X-14 |
esp+1C |
ebp+C |
param2 |
X-14 |
esp+18 |
ebp+C |
param2 |
| X-10 |
esp+20 |
ebp+10 |
param3 |
X-10 |
esp+1C |
ebp+10 |
param3 |
| X-C |
esp+24 |
ebp+14 |
edx |
X-C |
esp+20 |
ebp+14 |
edx |
| X-8 |
esp+28 |
ebp+18 |
ecx |
X-8 |
esp+24 |
ebp+18 |
ecx |
| X-4 |
esp+2C |
ebp+1C |
eax |
X-4 |
esp+28 |
ebp+1C |
eax |
| X |
esp+30 |
ebp+20 |
? |
X |
esp+2C |
ebp+20 |
? |
| AFTER pop esi |
AFTER pop edi |
| Memory Address |
Relative to esp |
Relative to ebp |
Value |
|
Memory Address |
Relative to esp |
Relative to ebp |
Value |
| X-34 |
esp-C |
ebp-14 |
esi(x) |
|
X-34 |
esp-14 |
ebp-14 |
esi(x) |
| X-30 |
esp-8 |
ebp-10 |
edi(x) |
|
X-30 |
esp-10 |
ebp-10 |
edi(x) |
| X-2C |
esp-4 |
ebp-C |
ebx(x) |
|
X-2C |
esp-C |
ebp-C |
ebx(x) |
| X-28 |
esp |
ebp-8 |
var2 |
|
X-28 |
esp-8 |
ebp-8 |
var2(x) |
| X-24 |
esp+4 |
ebp-4 |
var1 |
|
X-24 |
esp-4 |
ebp-4 |
var1(x) |
| X-20 |
esp+8 |
ebp |
ebp |
|
X-20 |
esp |
ebp |
ebp |
| X-1C |
esp+C |
ebp+4 |
eip |
|
X-1C |
esp+4 |
ebp+4 |
eip |
| X-18 |
esp+10 |
ebp+8 |
param1 |
|
X-18 |
esp+8 |
ebp+8 |
param1 |
| X-14 |
esp+14 |
ebp+C |
param2 |
|
X-14 |
esp+C |
ebp+C |
param2 |
| X-10 |
esp+18 |
ebp+10 |
param3 |
|
X-10 |
esp+10 |
ebp+10 |
param3 |
| X-C |
esp+1C |
ebp+14 |
edx |
|
X-C |
esp+14 |
ebp+14 |
edx |
| X-8 |
esp+20 |
ebp+18 |
ecx |
|
X-8 |
esp+18 |
ebp+18 |
ecx |
| X-4 |
esp+24 |
ebp+1C |
eax |
|
X-4 |
esp+1C |
ebp+1C |
eax |
| X |
esp+28 |
ebp+20 |
? |
|
X |
esp+20 |
ebp+20 |
? |
| AFTER pop ebx |
|
AFTER mov esp, ebp |
| Memory Address |
Relative to esp |
Value |
|
Memory Address |
Relative to esp |
Value |
| X-2C |
esp-10 |
ebx(x) |
X-2C |
esp-14 |
ebx(x) |
| X-28 |
esp-C |
var2(x) |
X-28 |
esp-10 |
var2(x) |
| X-24 |
esp-8 |
var1(x) |
X-24 |
esp-C |
var1(x) |
| X-20 |
esp-4 |
ebp(x) |
X-20 |
esp-8 |
ebp(x) |
| X-1C |
esp |
eip |
X-1C |
esp-4 |
eip(x) |
| X-18 |
esp+4 |
param1 |
X-18 |
esp |
param1 |
| X-14 |
esp+8 |
param2 |
X-14 |
esp+4 |
param2 |
| X-10 |
esp+C |
param3 |
X-10 |
esp+8 |
param3 |
| X-C |
esp+10 |
edx |
X-C |
esp+C |
edx |
| X-8 |
esp+14 |
ecx |
X-8 |
esp+10 |
ecx |
| X-4 |
esp+18 |
eax |
X-4 |
esp+14 |
eax |
| X |
esp+1C |
? |
X |
esp+18 |
? |
| AFTER pop ebp |
AFTER ret |
Then CPU returns to main routine to continue execution:
add esp, 0x0C ;restoring stack to the state before the call
pop edx ;restoring registers
pop ecx ;
pop eax ;first saved register pops last
...
| Memory Address |
Relative to esp |
Value |
|
Memory Address |
Relative to esp |
Value |
| X-1C |
esp-10 |
eip(x) |
X-1C |
esp-14 |
eip(x) |
| X-18 |
esp-C |
param1(x) |
X-18 |
esp-10 |
param1(x) |
| X-14 |
esp-8 |
param2(x) |
X-14 |
esp-C |
param2(x) |
| X-10 |
esp-4 |
param3(x) |
X-10 |
esp-8 |
param3(x) |
| X-C |
esp |
edx |
X-C |
esp-4 |
edx(x) |
| X-8 |
esp+4 |
ecx |
X-8 |
esp |
ecx |
| X-4 |
esp+8 |
eax |
X-4 |
esp+4 |
eax |
| X |
esp+C |
? |
X |
esp+8 |
? |
| AFTER sub esp, 0x0C |
AFTER pop edx |
| Memory Address |
Relative to esp |
Value |
|
Memory Address |
Relative to esp |
Value |
| X-C |
esp-8 |
edx(x) |
|
X-C |
esp-C |
edx(x) |
| X-8 |
esp-4 |
ecx(x) |
|
X-8 |
esp-8 |
ecx(x) |
| X-4 |
esp |
eax |
|
X-4 |
esp-4 |
eax(x) |
| X |
esp+4 |
? |
|
X |
esp |
? |
| AFTER pop ecx |
|
AFTER pop eax (Original state) |