Discount VMProtect (465pt)
Description: Seeing how easy new software is cracked I decided to use a Virtual Machine to protect my license check, because VMs can not be cracked, right? (Author: avlad171)
TLDR: The binary reads a flag and runs two stack-based programs. The first program checks the length of the flag and the second validates it against a static string.
From the title of the challenge we know that this binary will implement some sort of VM. After opening main, we see:
The password is read into a global buffer
flag_dat and we see two function that operate on some global data.
Then it compares the value of a variable and tells us if our flag is correct.
Upon opening the function, we see some light obfuscation:
The return value gets mangled to
0x40087F which implements the VM.
Inside the VM function, we see a familiar switch table that implements our program opcodes:
After dynamic analysis, we can see that the program input is stored at
Looking at the first few instructions, it is clear that
[rbp-4] must be some sort of program counter.
At this point, I started reversing the opcodes and wrote a simple disassembler:
def decode(prog, pc): '''disassemble a single opcode, returns (name, len)''' a = prog[pc] if a == 48: return 'ret', 1 elif a == 49: return 'push(res[%d])' % prog[pc+1], 2 elif a == 50: return 'dup', 1 elif a == 51: return 'push(flag[pop()])', 1 elif a == 52: return 'jz 0x%x' % prog[pc+1], 2 elif a == 53: return 'ror', 1 elif a == 54: return 'push 0x%x' % prog[pc+1], 2 elif a == 55: return 'xor', 1 elif a == 56: return 'add', 1 elif a == 57: return 'sub', 1 elif a == 97: return 'not', 1 elif a == 98: return 'dbg', 1 elif a == 99: return 'strcpy', 1 elif a == 100: return 'res[%d] = pop()' % prog[pc+1], 2 elif a == 101: return 'get res[top]', 1 else: return 'unk', 1 def disasm(prog): pc = 0 while pc < len(prog): d,sz = decode(prog, pc) print('[%02X] :: %s' % (pc, d)) pc += sz
This is a pretty simple stack-based language.
There are several math operations (
not) that operate on stack values.
dup duplicates the top of the stack.
dbg is an opcode that calls ptrace and potentially fails (if we are running in a debugger for example).
Several opcodes operate on flag data and a separate global buffer that is shared between programs.
strcpy opcode, sets
1 and copies a static buffer into
res[0xa:...] (accessible to the VM programs).
main, we know that the value
res must be
1 after both programs are done for the flag to be correct.
At this point, I ran the disassembler on both the programs and reversed them by hand. I’ve annotated them below with the stack layout and potential labels:
__start:  :: push(res) | r  :: strcpy __loop:  :: dup | r r  :: dup | r r r  :: res = pop() | r r  :: push(flag[pop()]) | r f[r]  :: jz 0x12 [0A] :: push 0x1 | r 1 [0C] :: add | (r+1) [0D] :: dup | (r+1) (r+1) [0E] :: dup | (r+1) (r+1) (r+1) [0F] :: sub | (r+1) (r+1)  :: jz 0x3 | (r+1) __end_of_flag:  :: push 0x23 | len 0x23  :: sub | (len - 0x23)  :: jz 0x1b __incorrect_length:  :: push 0x0  :: res = pop() | res = 0 __correct_length: [1B] :: push 0x0 [1D] :: res = pop() | res = 0 [1F] :: ret
This program simply loops over the flag and finds the first null byte.
If the length is not
res is set to
0 (indicating that our flag is incorrect).
__start:  :: push(res) | r  :: dup | r r  :: dup | r r r  :: res = pop() | r r  :: push(flag[pop()]) | r f[r]  :: dup | r f[r] f[r]  :: jz 0x28 | r f[r] __flag_not_zero: [0A] :: ror | r (f[r] >>> 1) [0B] :: push 0x63 | r (f[r] >>> 1) 0x63 [0D] :: xor | r ((f[r] >>> 1)^0x63) [0E] :: push 0x98 | r ((f[r] >>> 1)^0x63) 0x98  :: add | r (((f[r] >>> 1)^0x63)+0x98)  :: not | r ~(((f[r] >>> 1)^0x63)+0x98)  :: dbg  :: push(res) | r ~(((f[r] >>> 1)^0x63)+0x98) r  :: push 0xa | r ~((f[r] >>> 1)^0x63)+0x98) r 0xa  :: add | r ~(((f[r] >>> 1)^0x63)+0x98) (r+0xa)  :: get res[top] | r ~(((f[r] >>> 1)^0x63)+0x98) res[(r+0xa)]  :: sub [1A] :: jz 0x20 __incorrect_char [1C] :: push 0x0 [1E] :: res = pop() __correct_char  :: push 0x1  :: add  :: dup  :: dup  :: xor  :: jz 0x2 __flag_zero:  :: ret
This program also loops over the flag until finding a null byte.
Here it checks each character
flag[i] against the corresponding byte
However the exact check is
~(((flag[i] >>> 1) ^ 0x63) + 0x98) == res[i+0xa]. (
>>> indicates rotate right here).
At this point, we can reverse the flag with the provided static buffer in the binary:
d = open('./VM', 'rb').read() check = d[0x20a0:0x20cc] check = [ord(x) for x in check] for i in range(len(check)): v = check[i] v = ~v v -= 0x98 v ^= 0x63 v &= 0xff v = (v << 1) + (v >> 7) # rotate left check[i] = chr(v & 0xff) print(''.join(check))