![]() |
Some Lua details
As everyone knows, the Empire at War Lua files are some weird Lua variant.
Some while ago, I figured out what exactly made these lua files 'weird':
So, read a EaW Lua file (or, as I like to call them thanks to their signature: Lup files), change the signature, change the version and remove the extra integers and you're left with a perfectly valid Lua 5.0 file. Unfortunately, this is a long way from the Lua source code. I am currently aware of two decompilers: LuaDC and Luadec. Neither of which succesfully decompiled the files. LuaDC complained about invalid filenames and what not when I started it, and without the source I have no idea what's wrong with it (and I didn't give up easily). Luadec was able to read the files but it crashed on decompilation. As it turned out (yay for open-source), luadec assumes the information about a function's locals is present, whereas it is, in fact, not (the locals information is optional, and Petroglyph apparently decided to strip that. Can't blame them). After some l33t hacking :p in luadec I got to work around this by tricking it into thinking it had locals information. Now it actually produced Lua code from the most basic of Lua files. However, when decompiling the more complicated Lua files luadec is unable to make sense of them, leaving me less then impressed with this decompiler. I've even gone so far as to take a pot shot at writing a decompiler myself, but proper decompilation is hard, even for a 'simple' language such as Lua. Well, it is for me anyway. On the other hand, I can of course output Lua assembly from the lua files; that's easy to do. But then this has to be manually decompiled into normal Lua code, something I'm sure most modders can't, won't (or even shouldn't) do. Anyway, I just thought I'd put this information on the forums. Maybe someone who does know a thing or two about decompilation can use this info, or perhaps point me to, or adapt, another existing decompiler. Update 2006/05/05: decompiled lua files are now available: http://alpha1.dyns.net/eaw/ |
If you could figure out this one it will save us alot of trouble especialy with the AI.
Thank you for you findings so far. COntinue the good job. |
BTW I just forgot to ask you. What files do you edit for scripted action on the welcome screen ?
Thank you. |
Thats in the gameconstants.xml, its actually a replay of your most recent recorded game, a feature that was, unfortunatly, removed. So you get stuck with that same one.
|
Well Mike good job so far! This is the REAL editor I am looking for. Can you give us a version of the LuaDec you modified?
Quote:
I programmed way to much in assembler decades ago when computers were very 'dumb'. Even the decompiled asssembler code can be of great use to do basic modification of integer -say what the AI will use for a raid fleet total number or what it thinks it needs to attack your fleet. Then by going into the regular files, it can be hexed in. I did this for the DLL in Freelancer to get around some annoying behaviour to allow playing the game in single player mode without the storyline getting into the way. So if you can give out a copy, I think quite abit of simple modding of the scripts can be done to some extent using the assembler code as a guide. This well give the game a boost by making the AI smarter. |
Hey, the old demo is written slightly different than the final game release Lup. I manage to follow you advice for the old demo file Lua 5.0 and it decompiled quite a bit until it crashed! lol
I thought you may find this interesting. I only used LuaDec on BuildTurboLaser.lua of the old demo. I change the header from Lua(51) to Lua(50). No other change yet and it decompiled the bugger until it hit the 'end' or that integer thingy. The other thing it seems to have a problem with handling End of Line. If you can show me what you did with the LuaDec I think with this information; you will have a decompiler of some sort. BuildTurboLasers.lua output before crash: ---------------------------------------------- 1 GETGLOBAL 0 0 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{require} ---------------------------------------------- 2 LOADK 1 1 SET_CTR(Tpend) = 1 next bool: 0 locals(0): vpend(0): tpend(2): 0{require} 1{"pgevents"} ---------------------------------------------- 3 CALL 0 2 1 next bool: 0 locals(0): vpend(0): tpend(0): require("pgevents") ---------------------------------------------- 4 CLOSURE 0 0 ---------------------------------------------- 1 LOADK 0 1 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{"Build_Turbolasers"} () ---------------------------------------------- 2 SETGLOBAL 0 0 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(1): -1{Category="Build_Turbolasers"} tpend(0): () ---------------------------------------------- 3 LOADBOOL 0 1 0 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{true} () Category = "Build_Turbolasers" ---------------------------------------------- 4 SETGLOBAL 0 2 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(1): -1{IgnoreTarget=true} tpend(0): () Category = "Build_Turbolasers" ---------------------------------------------- 5 NEWTABLE 0 1 0 next bool: 0 locals(0): vpend(0): tpend(0): () Category = "Build_Turbolasers" IgnoreTarget = true ---------------------------------------------- 6 NEWTABLE 1 2 0 next bool: 0 locals(0): vpend(0): tpend(0): () Category = "Build_Turbolasers" IgnoreTarget = true ---------------------------------------------- 7 LOADK 2 4 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 2{"StructureForce"} () Category = "Build_Turbolasers" IgnoreTarget = true ---------------------------------------------- 8 LOADK 3 5 SET_CTR(Tpend) = 1 next bool: 0 locals(0): vpend(0): tpend(2): 2{"StructureForce"} 3{"Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"} () Category = "Build_Turbolasers" IgnoreTarget = true ---------------------------------------------- 9 SETLIST 1 1 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 1{{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}} () Category = "Build_Turbolasers" IgnoreTarget = true ---------------------------------------------- 10 SETLIST 0 0 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{{{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}}} () Category = "Build_Turbolasers" IgnoreTarget = true ---------------------------------------------- 11 SETGLOBAL 0 3 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(1): -1{TaskForce={{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}}} tpend(0): () Category = "Build_Turbolasers" IgnoreTarget = true ---------------------------------------------- 12 RETURN 0 1 0 next bool: 0 locals(0): vpend(0): tpend(0): () Category = "Build_Turbolasers" IgnoreTarget = true TaskForce = {{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}} SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{function() Category = "Build_Turbolasers" IgnoreTarget = true TaskForce = {{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}} end } require("pgevents") ---------------------------------------------- 5 SETGLOBAL 0 2 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(1): -1{Definitions=function() Category = "Build_Turbolasers" IgnoreTarget = true TaskForce = {{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}} end } tpend(0): require("pgevents") ---------------------------------------------- 6 CLOSURE 0 1 ---------------------------------------------- 1 GETGLOBAL 0 0 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{Sleep} () ---------------------------------------------- 2 LOADK 1 1 SET_CTR(Tpend) = 1 next bool: 0 locals(0): vpend(0): tpend(2): 0{Sleep} 1{1} () ---------------------------------------------- 3 CALL 0 2 1 next bool: 0 locals(0): vpend(0): tpend(0): () Sleep(1) ---------------------------------------------- 4 GETGLOBAL 0 2 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{StructureForce} () Sleep(1) ---------------------------------------------- 5 GETTABLE 0 0 253 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{StructureForce.Set_As_Goal_System_Removable} () Sleep(1) ---------------------------------------------- 6 LOADBOOL 1 0 0 SET_CTR(Tpend) = 1 next bool: 0 locals(0): vpend(0): tpend(2): 0{StructureForce.Set_As_Goal_System_Removable} 1{false} () Sleep(1) ---------------------------------------------- 7 CALL 0 2 1 next bool: 0 locals(0): vpend(0): tpend(0): () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) ---------------------------------------------- 8 GETGLOBAL 0 4 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{AssembleForce} () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) ---------------------------------------------- 9 GETGLOBAL 1 2 SET_CTR(Tpend) = 1 next bool: 0 locals(0): vpend(0): tpend(2): 0{AssembleForce} 1{StructureForce} () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) ---------------------------------------------- 10 CALL 0 2 1 next bool: 0 locals(0): vpend(0): tpend(0): () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) AssembleForce(StructureForce) ---------------------------------------------- 11 GETGLOBAL 0 2 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{StructureForce} () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) AssembleForce(StructureForce) ---------------------------------------------- 12 GETTABLE 0 0 255 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{StructureForce.Set_Plan_Result} () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) AssembleForce(StructureForce) ---------------------------------------------- 13 LOADBOOL 1 1 0 SET_CTR(Tpend) = 1 next bool: 0 locals(0): vpend(0): tpend(2): 0{StructureForce.Set_Plan_Result} 1{true} () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) AssembleForce(StructureForce) ---------------------------------------------- 14 CALL 0 2 1 next bool: 0 locals(0): vpend(0): tpend(0): () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) AssembleForce(StructureForce) StructureForce.Set_Plan_Result(true) ---------------------------------------------- 15 GETGLOBAL 0 6 SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{ScriptExit} () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) AssembleForce(StructureForce) StructureForce.Set_Plan_Result(true) ---------------------------------------------- 16 CALL 0 1 1 next bool: 0 locals(0): vpend(0): tpend(0): () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) AssembleForce(StructureForce) StructureForce.Set_Plan_Result(true) ScriptExit() ---------------------------------------------- 17 RETURN 0 1 0 next bool: 0 locals(0): vpend(0): tpend(0): () Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) AssembleForce(StructureForce) StructureForce.Set_Plan_Result(true) ScriptExit() SET_CTR(Tpend) = 0 next bool: 0 locals(0): vpend(0): tpend(1): 0{function() Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) AssembleForce(StructureForce) StructureForce.Set_Plan_Result(true) ScriptExit() end } require("pgevents") Definitions = function() Category = "Build_Turbolasers" IgnoreTarget = true TaskForce = {{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = |
I've written a small (command-line) tool that will convert Lup files to Lua files and back: lup2lua (and its source).
Second, here are my (small) modifications to luadec: luadec-0.6-mod. I only changed two things in the original file, and those changes are marked by my name, so you should find them easily. Running the resulting luadec over BuildTurboLasers.lua results in: C:\>move BUILDTURBOLASERS.LUA buildturbolasers.lup C:\>lup2lua buildturbolasers.lup buildturbolasers.lua C:\>luadec buildturbolasers.lua require("pgevents") Definitions = function() Category = "Build_Turbolasers" IgnoreTarget = true TaskForce = {{"StructureForce", "E_Galactic_Turbolaser_Tower_Defenses | R_Galactic_Turbolaser_Tower_Defenses = 1"}} end StructureForce_Thread = function() Sleep(1) StructureForce.Set_As_Goal_System_Removable(false) AssembleForce(StructureForce) StructureForce.Set_Plan_Result(true) ScriptExit() end StructureForce_Production_Failed = function(L0, L1) DebugMessage("%s -- Abandonning plan owing to production failure.", tostring(Script)) ScriptExit() end Yet, running it on larger, more complex lua files will crash the decompiler as usual. I've tried to find out why, but I don't understand the laudec code that well and it seems like more than just a superficial problem. Anyway, I'll create a lua disassembler over the next days, if that'll help you guys. |
Sure, please do. I do not think we will get a LUA editor for now. With a disassembler, we can (the hard way) change any small aspect of the game that many modders may like.
I for one would like to change the way the DeathStar Battle is played out. If it means to manually recreating the script from assembler -so be it. It would only be a few hours work to type up the text script. I assume the EXE will read the expanded form of it. Thanks for all your hard work Mike! |
Quote:
I've written several decompilers over the years, so I'll see what I can figure out about the crash with luadec. Thanks for the effort you've put into this! What an awesome place to start! |
Good work Mike! It would be great to have a community made LUA deassembler until/if Petro releases their own.
|
LoW's Coder Alisama is busy doing Model File Comparisons So We may have a Filefront Community Model Converter and Bone Editor
|
I figured out the luadec crash: It's because of the lack of local information.
I coded around it by using _TEMP_ for all local variable names. At least that got me able to decompile everything I've tried it on so far, but make no mistake, it's a kluge (for example, if there are 5 locals, all 5 are named _TEMP_ - not good!). Email me if you want a copy of the modified source (no place to put it online...). I will work on a proper solution to the local variable issue next. I just have to do some heavy code reading to understand luadec parse structure... |
Quote:
I have no clue of LUA coding, but in C, C++, Java and such this is doable. Either with matrices of strings or pointers. |
Quote:
So, now, all local variables have distinct names of L0, L1, L2, etc. The best we can do without the local variable information (i.e. not knowing the actual local variable names). The code is quite readable, and I have successfully decompiled all the lua code in config.meg now. There is one other strange bug in luadec: When a function is declared, the line of code generated is like "name = function()" instead of "function name()". It's annoying, but the code is still readable. I'll look into eventually... I can make a zip of all the decompiled sources if anybody is interested. As well as the fixed luadec and code if interested. |
If you could do us the favor and make a zip of the decompiled scripts and modified luadec it would be great.
It could give us a head start on using Lua. |
I would suggest to work closely with Mike, so that you break Lua till the end. So that we can even compile models too.
Yet, in the meanwhile, you can upload what's you've done. Just mind to post this to PFF forums too, there are people interested for this there too! |
Quote:
From my perspective, luadec is more complicated than it needs to be (it appears as though it actually attempts to walk the code by pseudo-executing it, which is, uh, not the way I'd do it...). As I mentioned before, I've written several deompilers so I am quite familiar with how to get one up and running quickly, and correctness is the most important thing for a decompiler IMO, and luadec still bails on some simple valid structures (which I just noticed the failure messages in a few of the files I decompiled - although 13 failures out of 260 isn't _that_ horrible for our purposes). |
Ok, I've found what I consider the fundamental issue with luadec: It's not decompiling the binary code directly, but using Lua to load and walk the code. As a result, the internals are quite ugly, as a lot of Lua specific internal states, upvalues, etc. all have to be tracked.
The odd thing is that none of that is needed. The object format is very straight forward, and the mapping from the 35 opcodes to the source is almost direct. Even without the debugging info (function names, local variable names), recreating the original source should be doable. Anyway, I've been studying chunkspy as well (a disassembler for Lua object code writen in Lua - only 2100 lines, with lots of comments), and I think I am going to work on creating something from scratch over the weekend. I'll post the progress. This is so much more fun than working on work! :-) |
Squeegee, mind that decompliling is one step, next step will be re-compiliing our modded lua files. As for opcodes, you should pay very attention so that we dont mess with micromemory (virtual i guess?), but you already know this ;).
If you can do both of these, then you should be given the "golden EaW metal" and ill be the first one to contact admins for this :D! As soon as we can get our hands dirty with lua scripting.........our way is open ;). Keep up the good work! |
Allright, this is it!
I have finished two tools that:
Yes, that's right. Lua sources are available. Along with instructions on how to create custom Lua files for Empire at War. Get it all here: http://alpha1.dyns.net/eaw/ Lacking a good decompiler, I've manually decompile everything luadec crashed on, and I still have to do the remaining story scripts, but I decided to release the Luas I've decompiled so far so you guys have something to work on :) Of course, keep in mind the source is uncommented and uses a bit vague function arguments, which is inherent to any decompilation. Perhaps I will one day rename them to something meaningful, based on what they do, once I have time. Anyway, as usual, if you have questions or remarks (both the good and the bad kind), post here. |
Quote:
Quote:
Quote:
Unlike luadec, this loads the binary object directly, wihtout Lua's involvement. The code is only 500 lines of c++ so far. I should have the source generation working reliably within another couple of days. I've named it luad (I know, how original...). Once this is done, along with your lup2lua tool, and luac, we should be able to decompile, modify, and recompile the sources! If you want, we can combine all of the tools into one, so it can be done easier for the non-tech types out there. Great job! |
MUST BE STICKIED I-M-M-E-D-I-A-T-E-L-L-Y !!!
Also Mike, do not forget to post this over PFF forums too. The people will go mad! And someone give those guys a golden medal :D. Mike, i suggest you work with squeege so that we get 100% bug-free and 100% commented lua files. Check your pm also ;). [edit] I posted this for you in PFF, just in case you're not registered ;). |
Quote:
And we wouldn't want 100% commented lua files. That would leave no room for the actual code ;) Quote:
Actually, the biggest challenge now is to create documentation. Establish an API. Find out how the Luas interface with the game and how far that interface can be stretched. And to squeegee: I'm looking forward to your decompiler. It would certainly be nice to bundle these tools. Out of curiosity, how robust will your decompiler be? Will it depend heavily on known constructs by the compiler? Or is it able to generate equivalent source code from other, but functionally equivalent code as well? For instance, the compiler compiles while loops with the test at the end, but what if the test were moved to the beginning? And what if the assembly was more optimized in terms of register (re)usage? The standard lua compiler is pretty brain-dead from what I can tell. But I do agree, the lua object format is ridiculously simple, that's why luadec never impressed me that much, with all its bloat. |
Quote:
Quote:
The lua compiler (tokenizer...) doesn't need to optimize much, as the vm assembly is so simple, and the vm itself can optimize stuff out. If you read some of the vm asm specific stuff out there, they point these things out, as well as other quirks. Most of those doc's are dated in the last 6 to 9 months, so they didn't exist when luadec was written. My "plan" is to be able to decompile any lua source. Once that works perfectly, then I'll derive a version specific to EAW/Lup, and we can combine the stuff into a single executable. Maybe even provide a simple GUI so non-techies can use it (of course, maybe they shouldn't be modifying the code anyway... ;-). Quote:
What I have is entirely stack based, and thus has no limits on depth of functions, loops, etc. (well, other than the memory of the machine it's running on). By definition, it can only re-generate what the object code contains. That is, if the compile moves things around (which I haven't seen a case of yet, btw), then it will regenerate it that way, as there is no way to know different. Functionally, though, it'd be the same as the original, and the object code will be the same. Looking forward to getting this done; It's been a lot of fun so far! BTW I can send the code to anyone who wants it at any time. It'll be public domain or BSD/MIT styled licensed when I get done with it. Also, Mike, if you want the version of luadec that has fixed most of the crashes, let me know. I have a site I can put it on, I just have to put some work into the web site first (the domain is for email, but the http hosting is included but currently unused). |
A big thank you to you guys! :) You know your stuff very well (this is way above my head). It will be good to have even more versions to work with, in case something came up for redunancy.
As for me, the 'technically challenged', I can play with the AI game mechanics and see what happens! lol |
this sounds very good. i have almost no knowledge of LUA or anything of that sort, but im gonna look into it later this weekend when i have time.
way to go mike! |
If I read through all you posted, I could learn somthing, but my god you guys know your stuff, Keep up the good work ;)
|
Using Mike's new cool LUA Tools I was able to make a first Enhanced_AI_RAID LUA Module!
I wanted more of a challenging RAID ability for the Rebels. Now for the first iteration the AI will use up to 6 units plus the normal heros and commandos, and have a faster retreat time. Hit and Run tactics really is almost human like and to be feared! :) Later you can change what the AI considers for target priority and task force mix. It really does allow you change the AI behavior to be smarter! I am (have to) re-install EAW and making sure it is fully functional under patch 1.4. Once released, modders should try out the LUA scripting to make changes to the AI behaviour for the better after seeing what a HUGE difference it can make. |
Just to let you know, the patch 1.4 gives you the decompiled source LUA to use!
I am testing it to see if the game likes it if it is modified. |
Wow. I will re-decode the meg's and check it out. Thanks for the heads up!
|
The modified Lua will work in game. You can also add extra lines to it like C language. I was 'copy paste' command blocks to add extra stuff IE a Huge Rebel Invasion Fleet when it decides to "Crush.lua"...and it is awesome.
The RIAD_AI_BOOST works fine and is ready to go. I tesed it. Now fear a Rebel raid for sure! lol |
Quote:
|
I'm not sure if Slocket was correct by saying patch 1.4. It could be from the map editor as well.
Anyway, look in GameData\Mods\Source\Data\Scripts. |
Quote:
Thanks for the clarification Mike! |
Map Editor = Mod Tools. Including a exporter for ALO format :P
|
Quote:
So much for needing a decompiler. I am still working on it, but work has been getting in the way (if only I worked for google...:-). With mike's lup converter and the source, we should have everything needed to make changes to the ai now. |
We don't need a converter, I tried replacing the lua files in gamedata\data\scripts folder with the ones that came with the map editor, and the game still worked.
|
I am trying to decompile the new Forces of Corruption LUA with Mike's tools. Been awhile and his site for instructions is gone.
I think it is the same as EAW. Now where did I place his LuaDec 0.6 program. Any else have a copy? EDIT: Found the files, just have to look at the URL home. The download section was moved. http://alpha1.dyndns.info/eaw/LuaFiles I was able to decompile into some kind of assember code... |
likely pg smartened up
|
have any new tools, mike?
|
| All times are GMT -4. The time now is 10:23 PM. |
Powered by vBulletin®
Copyright ©2000 - 2013, Jelsoft Enterprises Ltd.
LFNetwork, LLC ©2002-2011 - All rights reserved.