how do procedure calls work in assembler?

int3 picture int3 · Aug 9, 2009 · Viewed 16.6k times · Source

I just started tinkering with ASM and I'm not sure if my understanding of procedure calls is correct.

say at some point in the code there is a procedure call

call dword ptr[123]

and the procedure consists of only one command, ret:

ret 0004

what would be the effect of this procedure call, and where would the return value be stored? I read somewhere that a return value of 2 bytes would be stored in AX, but when I replace the procedure call by

mov AX, 0004

(together with the necessary NOPs) the program crashes.

Answer

Nathan Fellman picture Nathan Fellman · Aug 9, 2009

in x86 assembler the parameter to the ret instruction means:

RET immediate

Return to calling procedure and pop immediate bytes from the stack.

(quoting from Intel® 64 and IA-32 Architectures Software Developer's Manuals Vol 2B)

So when you type:

ret 0004

You're telling the CPU to return to the instruction immediately after the call, and to pop 4 bytes off the stack. This is great if you pushed 4 bytes onto the stack before the call.

push eax
call dword ptr[123]

Note that this has nothing to do with the return value. In fact, a procedure in Assembly has no way of specifying that a value is a return value. This is all done by convention. Most compilers of which I am aware will use EAX to hold the return value, but this is true only because the calling function will expect the result there.

So your calling code would be:

call dword ptr [123]
mov dword ptr [result], eax

and your function that returns the value 4 would be:

mov eax, 4
ret