UofTCTF 2026: BabyBOF

This challenge was a classic buffer overflow exploitation, involving overwriting the return address of a function to jump to the “win” function.

We are provided a binary executable called chall and a remote server (where we have to retrieve the flag).

I ran the following command to perform reconnaissance on the binary file.

chmod +x ./chall
file ./chall
checksec --file=./chall
strings ./chall

First, I made the binary executable. I played around with the executable, testing inputs, etc. Then I ran the file command on the binary to get an initial idea of the file that I was working with. Then, I used checksec to see what security protections are in place, which helps narrow down the attack vectors. I also used the strings command to check if there were any glaring strings just out in the open (flags, functions, etc).

From the file command:

1. We were able to determine that the binary is an ELF (Executable and Linkable Format), which basically tells us that it is a standard Linux binary.
2. We were able to determine the architecture, x84-64, which means it is a 64-bit system, addresses are 8 bytes instead of 4 bytes, registers are in the format of RAX, RBX, etc, rather than EAX.
3. Most importantly, the program is not stripped. This means debugging symbols are present, variable/function names are preserved, and it makes it infinitely easier to reverse engineer compared to a stripped program, which removes all unnecessary metadata and only preserves the bare minimum so that it is runnable.

Then, we ran checksec on the binary to check if there are any exploit mitigations in place.

What is relevant in this CTF is the following:

1. No stack canary - Stack canaries are checks in place to make sure that the stack is not overwritten. If we were to overwrite a buffer and overflow it, if there was a stack canary, then the program would detect that it has been modified and would exit the program. No stack canary means we can overwrite return addresses without consequences.
2. No PIE - PIE stands for Position Independent Executable, which is a security feature that randomizes addresses within the binary. Since there is no PIE, the addresses are static and will be the same every time the binary is run.

After doing general recon, we have an idea - this program is vulnerable to an overflow attack. I then used Ghidra, a software reverse engineering (SRE) framework, to decompile and reverse engineer the binary.

Immediately, we see that there is a “win” function right above the main function. Our goal is to then manipulate the main function so that we can call the win function.

Here is the decompiled main function. In very simple terms, the program asks you to input your name. However, if the length of your input is greater than 14 letters, it will print “That’s suspicious.” Otherwise, it prints “Hi, (your name input).”

The vulnerability lies in the method gets(). This method does not perform any bounds-checking; it reads the user input stdin without considering the size of the buffer it can actually write to. This means we can use gets() to overflow the buffer, allowing us to overwrite nearby addresses. Additionally, the strlen() comparison check comes way too late to prevent us from overflowing the buffer.

When main is called, the stack is set up so that we have space for the buffer/local variables (16 bytes), followed by 8 bytes for the RBP pointer, and finally another 8 bytes for the return address pointer. Memory addresses always go from lower memory addresses to higher memory addresses.

However, the stack is actually very counterintuitive; it actually grows in the opposite direction. When pushing an item to the stack, it goes from a higher memory address to a lower memory address. That means the first items pushed into the stack will be at the highest memory address, while the most recently pushed-in item will be at a lower memory address.

We first push the return address into the stack, and then the base pointer RBP, and then the space for the buffer (16 bytes). The RSP, stack pointer, points to the top of the stack, and will actually decrement when we push things into the stack (decrementing = going to a lower memory address).

In this case, we have something like this.

| 16 bytes buffer | 8 bytes RBP | 8 bytes Return Address |
Lower Memory Address                      Higher Memory Address

Now, the exploit is to utilize gets() to overwrite the return address to get us to win(). From Ghidra, we know that the win function is located at 0x4011f6.

However, directly jumping to this location would cause stack alignment issues. For a 64-bit system, the RSP must be 16-byte aligned before a call (this means it has to end in 0).

If we jump directly to 0x4011f6, we will execute push rbp, which means that RSP will decrease by 8 (since we are adding items to the stack, the RSP goes up and goes to a LOWER memory address). To skip this potential stack misalignment, we should directly jump to 0x4011fb.

Here is the script I used to perform the buffer overflow and get an interactive shell on the remote server.

from pwn import *

p = remote('REMOTE SERVER IP ADDRESS', 5000)

payload = b'\x00'*24 + p64(0x4011fb)

p.sendline(payload)

p.interactive()

The payload sends 24 null bytes - filling up the 16-byte buffer and the 8-byte RBP, and then overwriting the return address with 0x4011fb. This works because strlen() stops reading at a null byte, and if we send 24 null bytes, it stops reading - assumes strlen to be 0, 0 < 14, program doesn’t exit since it fulfills the comparison check, allowing us to spawn a shell and view the flag.