Programming

There are various approaches, but the most common one in an x86 environment is overwriting the return address that was pushed onto the stack.

When you call a function, the compiler generally maps it to a CALL instruction at the machine language level.

At the time that a CALL instruction is invoked, the current instruction pointer gets pushed onto the stack.

kagis

https://www.felixcloutier.com/x86/call

When executing a near call, the processor pushes the value of the EIP register (which contains the offset of the instruction following the CALL instruction) on the stack (for use later as a return-instruction pointer). The processor then branches to the address in the current code segment specified by the target operand.

A function-local, fixed-length array will also live on the stack. If it's possible to induce the code in the called function to overflow such an array, it can overwrite that instruction pointer saved on the stack. When the function returns, it hits a RET instruction, which will pop that saved instruction pointer off the stack and jump to it:

https://www.felixcloutier.com/x86/ret

Transfers program control to a return address located on the top of the stack. The address is usually placed on the stack by a CALL instruction, and the return is made to the instruction that follows the CALL instruction.

If what was overwriting the saved instruction pointer on the stack was a function pointer to malicious code, it will now be executing.

If you're wanting to poke at this, I'd suggest familiarizing yourself with a low-level debugger so that you can actually see what's happening first, as doing this blindly from source without being able to see what's happening at the instruction level and being able to inspect the stack is going to be a pain. On Linux, probably gdb. On Windows, I'm long out of date, but SoftICE was a popular low-level debugger last time I was touching Windows.

You'll want to be able to at least set breakpoints, disassemble code around a given instruction to show the relevant machine language, display memory at a given address in various formats, and single step at the machine language level.

I'd also suggest familiarizing yourself with the calling convention for your particular environment, which is what happens at a machine language level surrounding the call and return from a subroutine. Such buffer overflow attacks involve also overwriting other data on the stack, and understanding what is being overwritten is going to be necessary to understand such an attack.