Decompiling Psychonauts Lua scripts

[Luis Correia]

Hi there,

i've begun working on a curious little endeavour to decompile the Lua scripts from the great Psychonauts game.

So far, i've figured out why standard decompilers just don't work: Psy uses a reduced Lua virtual machine, probably to cope with the little memory that the Playstation has. (either that or to reduce memory usage)

It uses Lua version 4.

After analysing the unpacked lua files (using Psychonauts Explorer) and compared them with some example files compiled with luac, (binary comparison of the Lua header, first 17 or so bytes of the file) i found the differences and by looking at the lua source code, there are hints about using a 'small virtual machine'.

Logically, I modified lua source and produced a new luacompiler.

The headers now matched! Good, I felt I was making a lot of progress by now!

When I tried to run the compiled lua files from Psy, lua didn't complain (but did also nothing).

My assumption at this point was that I had found out the reason behind noone has ever managed to produce a decompiler until now.

Onwards!

The task now is to find source of any decompiler for Lua 4. Found some, but most were for other versions, Lua 3 and Lua5. Others didn't provide source, which isn't useable.

I'm now basing my work off a decompiler that was made for Homeworld2 [http://forums.relicnews.com/showthread.php?t=140357].

Basic file parsing is done, working currently on the actual decompilation, which is giving me the griefs...

Since source is provided and no license is available, i think it is ok to work based on it. i've also sent an email to the author but no reply so far.

I'm going to keep updating this post with new discoveries and (who knows) maybe some test version of the decompiler

Cheers!

Luis Correia

[Luis Correia]

Hi all,

a bit more info about this

From LUA 4.0.1 lopcodes.h file, edited for clearness:

  We assume that instructions are unsigned numbers.
  We assume that instructions are unsigned numbers.
  All instructions have an opcode in the first 6 bits. Moreover,
  an instruction can have 0, 1, or 2 arguments. Instructions can
  have the following types:
  type 0: no arguments
  type 1: 1 unsigned argument in the higher bits (called `U')
  type 2: 1 signed argument in the higher bits          (`S')
  type 3: 1st unsigned argument in the higher bits      (`A')
          2nd unsigned argument in the middle bits      (`B')

  A signed argument is represented in excess K; that is, the number
  value is the unsigned value minus K. K is exactly the maximum value
  for that argument (so that -max is represented by 0, and +max is
  represented by 2*max), which is half the maximum for the corresponding
  unsigned argument.

  The size of each argument is defined in `llimits.h'. 
  The usual is an instruction with 32 bits, 
   U arguments with 26 bits (32-6), 
   A arguments with 17 bits (32-6-9), 
   B arguments with 9 bits, 
   
  For small installations, the instruction size can be 16, so 
   U has 10 bits, and 
   A has 5 bits and 
   B has 5 bits.

We are using a 'small instalation' so the second option applies.

This will result in a Lua instruction (OPCode and optional values) represented bitwise like this (hope I get this one right)

16 bit Lua instruction
MSB           LSB
11111111 11111111 - all bits
00000000 00111111 - OPCode
11111000 00000000 - A argument
00000111 11000000 - B argument
11111111 11000000 - U argument
*1111111 11000000 - S argument (* is the sign)

To decode the instructions properly some bit-shift magic has to happen. The code i'm using currently may or may not be accurate in respect of this issue.

Still a lot unsure of it.
Nevertheless, this explanation about bits is important because either DoubleFine has made some mumbojumbo with the opcodes, effectively making the scripts undecodable, or i'm doing something wrong (most likely option).

I'm publishing the source code i'm working at the moment. This is still far from finished and you'll need some sort of Visual Basic 6, I'm using the Portable version you can find easily by Googling a bit

Please send you comments in, via replies to this post, or private messeges. Either one is fine but please provide feedback.

I can't publish any of the game files for obvious reasons.
I do however provide a compiled script for fast testing (one of the test programs provided with Lua source code).

You can get the source code here

Have fun!

[Luis Correia]

Here's another update:

found a recording of what I think is the presentation made by Paul Du Bois (double fine productions) in the Lua Workshop 2005, where he describes "Psychonauts and Lua: a case study ".

in here most of my findings were answered positively, numbers are floats (or single), instructions are 16 bit and my worst fear was also confirmed... they've added and expanded some opcodes (OP_CALL and OP_CLOSURE, maybe some/all of the OP_JMP*).

so, unless I do get some help from someone willing to spend a lot of time into figuring this out, this project may well come to a halt.

thanks for listening

[Luis Correia]

Special Update!

i've managed to decompile some of the scripts. (i've had special help)

if there is enough interest (haven't seen any for a while), i'll give out the code that I have.

my code still b0rks with tons of stuff and the output Lua decompilation is unuseable as it is, there are a lot of constructs that are just plain wrong, others just appear as "();"

while being very funny, we can start to understand how Psychonauts works internally.

But a word of warning: it is way too much complex for us to take advantage and do a MOD, the internal code on the C++ side doesn't seem to support it.

p.s. if/when/ever the code is more finished, i'll humbly ask Double Fine for permission to publish it

Cheers,
Luis Correia

[Scummbuddy]

Hmm, with all your tough effort, you may be better off not asking for permission, but for forgiveness later.