El Diablo - Reverse

Points: 498 | Flag: BITSCTF{l4y3r_by_l4y3r_y0u_unr4v3l_my_53cr375} | Solved by: Smothy @ 0xN1umb

ogres have layers

"ogres have layers. onions have layers. THIS BINARY has layers."

what we got

Single binary, description says "I bought this program but I lost the license file". So we gotta reverse a DRM/license checker. ez right? lmao no.

$ file challenge
challenge: ELF 64-bit LSB shared object, x86-64, Packer: Packed with UPX

the solve

layer 1: UPX

ok this one was free

bash

$ upx -d challenge -o challenge_unpacked

layer 2: anti-debug

Binary has like 4 different anti-debug checks:

ptrace(PTRACE_TRACEME)
/proc/self/status TracerPid check
Parent process name check (looks for lldb, strace, ida64, radare2, etc)
/proc/version WSL/Microsoft check

Quick LD_PRELOAD bypass:

// noptrace.c
long ptrace(int request, pid_t pid, void *addr, void *data) { return 0; }

bash

$ gcc -shared -o noptrace.so noptrace.c -fPIC
$ LD_PRELOAD=./noptrace.so ./challenge_unpacked test.lic

but here's the sneaky part - the anti-debug isn't just protection. each check that passes (no debugger) ORs a value into a global variable:

DAT_0010c290 = 0;
if (!ptrace_detected)    DAT_0010c290 |= 0xa623000000000000;
if (!tracerpid_detected) DAT_0010c290 |= 0x73a700000000;
if (!proc_check_clean)   DAT_0010c290 |= 0x70290000;
if (!parent_check_clean) DAT_0010c290 |= 0x31c7;

final key when clean: 0xa62373a7702931c7 - this becomes the AES decryption key for the next layer. if you're debugging, you get a wrong key and garbage bytecode. big brain move ngl

layer 3: AES-128-CBC encrypted VM bytecode

928 bytes of VM bytecode sitting in the binary, encrypted with AES-128-CBC:

Key: c7312970a77323a6c7312970a77323a6 (derived from the anti-debug value)
IV: all zeros

python

from Crypto.Cipher import AES

key_val = 0xa62373a7702931c7
aes_key = bytes([(key_val >> (i*8)) & 0xff for i in range(16)])
cipher = AES.new(aes_key, AES.MODE_CBC, b'\x00'*16)
decrypted = cipher.decrypt(encrypted_bytecode)
# boom, readable VM instructions

layer 4: custom VM

the decrypted bytecode runs on a custom VM with:

10 registers (r0-r9), each holding either int or string
Z-flag for conditional jumps
full instruction set: arithmetic, XOR, string ops, jumps, comparisons

decompiled the opcode dispatch table from IDA, mapped all ~35 opcodes. key ones:

0x01: LOAD_IMM reg, imm16
0x20: XOR dst, src1, src2
0x82: GET_LICENSE_BYTE dst, idx_reg
0x84: PRINT_FLAG_CHAR reg

layer 5: the actual flag logic

once you disassemble the VM bytecode the pattern is dead simple. literally just repeating this 46 times:

LOAD_IMM r0, <constant>       ; hardcoded XOR value
LOAD_IMM r3, <index>          ; license byte index (0-9, cycling)
GET_LICENSE_BYTE r1, r3        ; r1 = license[index]
XOR r0, r0, r1                 ; r0 = constant ^ license_byte
PRINT_FLAG_CHAR r0             ; putchar(r0)

so flag[i] = constant[i] ^ license[i % 10]

since we know the flag starts with BITSCTF{ that gives us 8 of the 10 license key bytes immediately. the remaining 2 were trivial to guess from context (l4y3r pattern).

oh and one more gotcha - PRINT_FLAG_CHAR only works if the PRINT_FLAG_CHAR environment variable is set. without it the opcode is a silent nop. sneaky.

final run

$ echo -n "LICENSE-99f5671124d520d5f63c" > license.lic
$ LD_PRELOAD=./noptrace.so PRINT_FLAG_CHAR=1 ./challenge_unpacked license.lic
[i] loaded license file
processing... please wait...
[i] running program...
The flag lies here somewhere...
BITSCTF{l4y3r_by_l4y3r_y0u_unr4v3l_my_53cr375}

matrix hacking

flag

BITSCTF{l4y3r_by_l4y3r_y0u_unr4v3l_my_53cr375}

5 layers deep: UPX -> anti-debug -> AES encryption -> custom VM -> XOR cipher. the flag literally tells you what the challenge is about lol. fr tho the anti-debug-as-key-derivation trick was clean, respect to the author.

smothy out ✌️

El Diablo