gcc (Ghost C Compiler) - Reverse Engineering

Points: 436 | Flag: BITSCTF{n4n0m1t3s_4nd_s3lf_d3struct_0ur0b0r0s} | Solved by: Smothy @ 0xN1umb

what we got

Single binary ghost_compiler - claims to be a "safe C compiler" that's "as fast as gcc". README says to try it with hello world. Sounds totally legit right.

Stripped binary, uses system, fwrite, unlink, memset, chmod... bunch of file manipulation stuff. sus.

first look

Ran it on a hello world and it... actually works? Compiles fine, output runs fine, identical to regular gcc output. So what's the "ghost" part??

literally the same binary as gcc lmao. but then i noticed something - the ghost_compiler's timestamp changed after running it. the binary modified itself. re-extracted from the zip and sure enough:

different hashes. this thing is self-modifying. the ghost ate itself fr

the solve

diffed the original and post-run binaries:

orig = open('ghost_compiler', 'rb').read()
ran = open('ghost_compiler_ran', 'rb').read()
diffs = [(i, orig[i], ran[i]) for i in range(len(orig)) if orig[i] != ran[i]]
# 64 bytes different at file offset 0x3020-0x305f
# Original: 9aa522e81efa9190...
# Modified: 0000000000000000...  (all zeros)

64 bytes at offset 0x3020 got zeroed out. that's the .data section (VA 0x4020). so the binary had some hidden data that it wiped after first execution. classic ouroboros behavior.

reversed the binary and figured out the full flow:

1. func_1349(argv[0]) - searches its OWN binary for the 8-byte pattern stored at 0x4020
2. func_14b5(argv[0], offset) - computes FNV-1a hash of the entire binary, SKIPPING the 64 flag bytes. XORs result with 0xcafebabe00000000 to get the decryption key
3. func_1583(buffer, offset, key) - XOR decrypts the 64 bytes using the key (rotating right by 1 bit each byte), checks if result starts with BITSCTF{
4. if valid: wipes the 64 bytes with memset(0), deletes itself with unlink(), rewrites itself without the flag, then runs gcc normally
5. if invalid (already wiped): returns error

ngl the XOR cipher with the rotating key was a nice touch. the key derivation:

# FNV-1a hash (skipping the 64 flag bytes)
FNV_OFFSET_BASIS = 0xcbf29ce484222325
FNV_PRIME = 0x100000001b3

h = FNV_OFFSET_BASIS
for i in range(len(data)):
    if 0x3020 <= i <= 0x305f:
        continue  # skip the encrypted flag
    h = (h ^ data[i]) * FNV_PRIME & 0xFFFFFFFFFFFFFFFF

key = 0xcafebabe00000000 ^ h  # = 0x5145dd89c16375d8

the key stays the same whether the flag bytes are encrypted or zeroed because the hash SKIPS them. so we can compute it from the wiped binary too. then just apply the XOR:

encrypted = bytes.fromhex('9aa522e81efa91901b8eb35e5a2af9f510ee6c42725476b1ad862f5caf3d5361fca716eee899048bbfde058b2e53178b45a25128148a45a25128140a85c261b0')
key = 0x5145dd89c16375d8

k = key
flag = b''
for byte in encrypted:
    flag += bytes([byte ^ (k & 0xFF)])
    k = ((k >> 1) | ((k & 1) << 63)) & 0xFFFFFFFFFFFFFFFF

# BITSCTF{n4n0m1t3s_4nd_s3lf_d3struct_0ur0b0r0s}

the name "ouroboros" in the flag is lowkey perfect - the serpent eating its own tail, just like this binary consuming its own flag data after each run

flag

BITSCTF{n4n0m1t3s_4nd_s3lf_d3struct_0ur0b0r0s}

smothy out ✌️