Home›Assembly Language in 5 Days›Day 5
Day 5 of 5
⏱ ~60 minutes
Assembly Language in 5 Days — Day 5

Reverse Engineering & Security

Assembly knowledge is the foundation of reverse engineering and binary exploitation. Today covers reading compiler-generated assembly, using GDB and radare2, and understanding how classic buffer overflow vulnerabilities work at the machine level.

Reading Compiler Output

Compilers translate C to assembly — reading that output reveals what the CPU actually does. Compile with 'gcc -O0 -S file.c' for unoptimized assembly. 'gcc -O2 -S' shows how the compiler optimizes: dead code elimination, loop unrolling, inlining, and auto-vectorization. Tools: Godbolt Compiler Explorer (godbolt.org) shows C source alongside assembly in real time. Every security researcher uses it daily.

GDB and radare2 for Reverse Engineering

GDB (GNU Debugger): 'disas main' disassembles a function, 'x/10i $rip' shows next 10 instructions, 'info registers' shows all register values, 'x/8gx $rsp' shows 8 qwords on the stack. radare2 is more powerful for static analysis: r2 binary, aaa (analyze), [email protected] (disassemble main), VV (visual flow graph). Both are essential for understanding unknown binaries.

Buffer Overflows at the Assembly Level

A stack buffer overflow overwrites data beyond an array's bounds. If a char buf[64] is on the stack and you write 80 bytes, you overwrite the saved RBP (8 bytes) and the return address (next 8 bytes). By controlling the return address, you redirect execution to your shellcode or a ROP gadget. Modern mitigations: stack canaries (detect overflow before return), ASLR (randomize addresses), NX (non-executable stack), and PIE (position-independent executable).

bash
# Compile C to assembly (Godbolt-style, locally)
gcc -O0 -S -masm=intel example.c -o example.s
cat example.s

# GDB: step through a binary
gdb ./program
(gdb) break main
(gdb) run
(gdb) layout asm
(gdb) info registers
(gdb) x/8gx $rsp      # view stack
(gdb) ni              # next instruction

# radare2: analyze a binary
r2 ./program
[0x00400530]> aaa     # analyze all
[0x00400530]> afl     # list functions
[0x00400530]> pdf @ main  # disassemble main
[0x00400530]> VV      # visual graph view

# Compile a vulnerable program (disable mitigations for learning)
gcc -fno-stack-protector -z execstack -no-pie vuln.c -o vuln
# Find offset to return address:
python3 -c 'print("A"*72 + "B"*8)' | ./vuln
💡
Install pwndbg or gef as a GDB plugin. They add colored output, automatic stack/register display on each step, and built-in commands for finding ROP gadgets and pattern matching.
📝 Day 5 Exercise
Analyze a Binary with GDB
  1. Write a simple C program with a function and compile with -O0 -g
  2. Open in GDB, set a breakpoint at the function entry, and run
  3. Use 'info registers' to see all register values at the breakpoint
  4. Use 'x/8gx $rsp' to view the first 8 quadwords on the stack
  5. Step through 10 instructions with 'ni' and watch registers change

Day 5 Summary

  • Compiler Explorer (godbolt.org) shows C-to-assembly translation interactively
  • GDB: disas, x/i $rip, info registers, x/gx $rsp are the essential commands
  • radare2: aaa, afl, [email protected] provide static analysis
  • Buffer overflows overwrite the saved return address to redirect execution
  • Modern mitigations: stack canaries, ASLR, NX bit, PIE
Challenge

Write a 50-line C program with 3 functions. Compile it and analyze the assembly output. Document: how each function's stack frame is set up, how arguments are passed, and where the return value is placed.

← Previous
Day 4: SIMD & Floating Point
Course complete
Back to course overview
→
Finished this lesson?