Discount VMProtect (465pt)

Reversing

Description: Seeing how easy new software is cracked I decided to use a Virtual Machine to protect my license check, because VMs can not be cracked, right? (Author: avlad171)

Files:

Solution:

TLDR: The binary reads a flag and runs two stack-based programs. The first program checks the length of the flag and the second validates it against a static string.

From the title of the challenge we know that this binary will implement some sort of VM. After opening main, we see:

The password is read into a global buffer flag_dat and we see two function that operate on some global data. Then it compares the value of a variable and tells us if our flag is correct.

Upon opening the function, we see some light obfuscation:

The return value gets mangled to 0x40087F which implements the VM.

Inside the VM function, we see a familiar switch table that implements our program opcodes:

After dynamic analysis, we can see that the program input is stored at [rbp-28h]. Looking at the first few instructions, it is clear that [rbp-4] must be some sort of program counter.

At this point, I started reversing the opcodes and wrote a simple disassembler:

def decode(prog, pc):
    '''disassemble a single opcode, returns (name, len)'''
    a = prog[pc]

    if a == 48:
        return 'ret', 1
    elif a == 49:
        return 'push(res[%d])' % prog[pc+1], 2
    elif a == 50:
        return 'dup', 1
    elif a == 51:
        return 'push(flag[pop()])', 1
    elif a == 52:
        return 'jz 0x%x' % prog[pc+1], 2
    elif a == 53:
        return 'ror', 1
    elif a == 54:
        return 'push 0x%x' % prog[pc+1], 2
    elif a == 55:
        return 'xor', 1
    elif a == 56:
        return 'add', 1
    elif a == 57:
        return 'sub', 1
    elif a == 97:
        return 'not', 1
    elif a == 98:
        return 'dbg', 1
    elif a == 99:
        return 'strcpy', 1
    elif a == 100:
        return 'res[%d] = pop()' % prog[pc+1], 2
    elif a == 101:
        return 'get res[top]', 1
    else:
        return 'unk', 1

def disasm(prog):
    pc = 0

    while pc < len(prog):
        d,sz = decode(prog, pc)
        print('[%02X] :: %s' % (pc, d))
        pc += sz

This is a pretty simple stack-based language. There are several math operations (ror, xor, add, sub, not) that operate on stack values. dup duplicates the top of the stack. dbg is an opcode that calls ptrace and potentially fails (if we are running in a debugger for example). Several opcodes operate on flag data and a separate global buffer that is shared between programs. The strcpy opcode, sets res[0] to 1 and copies a static buffer into res[0xa:...] (accessible to the VM programs).

Additionally, from main, we know that the value res[0] must be 1 after both programs are done for the flag to be correct.

At this point, I ran the disassembler on both the programs and reversed them by hand. I’ve annotated them below with the stack layout and potential labels:

Program 1

    __start:
[00] :: push(res[128])      | r
[02] :: strcpy

    __loop:
[03] :: dup                 | r r
[04] :: dup                 | r r r
[05] :: res[128] = pop()    | r r
[07] :: push(flag[pop()])   | r f[r]
[08] :: jz 0x12
[0A] :: push 0x1            | r 1
[0C] :: add                 | (r+1)
[0D] :: dup                 | (r+1) (r+1)
[0E] :: dup                 | (r+1) (r+1) (r+1)
[0F] :: sub                 | (r+1) (r+1)
[10] :: jz 0x3              | (r+1)

    __end_of_flag:
[12] :: push 0x23           | len 0x23
[14] :: sub                 | (len - 0x23)
[15] :: jz 0x1b

    __incorrect_length:
[17] :: push 0x0
[19] :: res[0] = pop()      | res[0] = 0

    __correct_length:
[1B] :: push 0x0
[1D] :: res[128] = pop()    | res[128] = 0
[1F] :: ret

This program simply loops over the flag and finds the first null byte. If the length is not 0x23, res[0] is set to 0 (indicating that our flag is incorrect).

Program 2

    __start:
[00] :: push(res[128])          | r
[02] :: dup                     | r r
[03] :: dup                     | r r r
[04] :: res[128] = pop()        | r r
[06] :: push(flag[pop()])       | r f[r]
[07] :: dup                     | r f[r] f[r]
[08] :: jz 0x28                 | r f[r]

    __flag_not_zero:
[0A] :: ror                     | r (f[r] >>> 1)
[0B] :: push 0x63               | r (f[r] >>> 1) 0x63
[0D] :: xor                     | r ((f[r] >>> 1)^0x63)
[0E] :: push 0x98               | r ((f[r] >>> 1)^0x63) 0x98
[10] :: add                     | r (((f[r] >>> 1)^0x63)+0x98)
[11] :: not                     | r ~(((f[r] >>> 1)^0x63)+0x98)
[12] :: dbg     
[13] :: push(res[128])          | r ~(((f[r] >>> 1)^0x63)+0x98) r
[15] :: push 0xa                | r ~((f[r] >>> 1)^0x63)+0x98) r 0xa
[17] :: add                     | r ~(((f[r] >>> 1)^0x63)+0x98) (r+0xa)
[18] :: get res[top]            | r ~(((f[r] >>> 1)^0x63)+0x98) res[(r+0xa)]
[19] :: sub
[1A] :: jz 0x20

    __incorrect_char
[1C] :: push 0x0
[1E] :: res[0] = pop()

    __correct_char
[20] :: push 0x1
[22] :: add
[23] :: dup
[24] :: dup
[25] :: xor
[26] :: jz 0x2

    __flag_zero:
[28] :: ret

This program also loops over the flag until finding a null byte. Here it checks each character flag[i] against the corresponding byte res[i+0xa]. However the exact check is ~(((flag[i] >>> 1) ^ 0x63) + 0x98) == res[i+0xa]. (>>> indicates rotate right here).

At this point, we can reverse the flag with the provided static buffer in the binary:

d = open('./VM', 'rb').read()

check = d[0x20a0:0x20cc]
check = [ord(x) for x in check]

for i in range(len(check)):
    v = check[i]

    v = ~v
    v -= 0x98
    v ^= 0x63
    v &= 0xff
    v = (v << 1) + (v >> 7) # rotate left

    check[i] = chr(v & 0xff)

print(''.join(check))

X-MAS{VMs_ar3_c00l_aNd_1nt3resting}