lfnetwork.com mark read register faq members calendar

Thread: Some Lua details
Thread Tools Display Modes
Post a new thread. Add a reply to this thread. Indicate all threads in this forum as read. Subscribe to this forum. RSS feed: this forum RSS feed: all forums
Old 04-18-2006, 10:08 AM   #1
Mike.nl
Rookie
 
Mike.nl's Avatar
 
Join Date: Jan 2006
Location: Netherlands
Posts: 52
Some Lua details

As everyone knows, the Empire at War Lua files are some weird Lua variant.
Some while ago, I figured out what exactly made these lua files 'weird':
  1. The signature is "\033Lup" instead of "\033Lua"
  2. The indicated version is 0x51 instead of 0x50 (yes, the files are actually in the 5.0 format)
  3. Every function contains an additional integer, after the "lineDefined" integer. I don't know what this integer is for, but it doesn't seem to serve any real purpose when reading the files.

So, read a EaW Lua file (or, as I like to call them thanks to their signature: Lup files), change the signature, change the version and remove the extra integers and you're left with a perfectly valid Lua 5.0 file.

Unfortunately, this is a long way from the Lua source code. I am currently aware of two decompilers: LuaDC and Luadec. Neither of which succesfully decompiled the files.

LuaDC complained about invalid filenames and what not when I started it, and without the source I have no idea what's wrong with it (and I didn't give up easily).
Luadec was able to read the files but it crashed on decompilation. As it turned out (yay for open-source), luadec assumes the information about a function's locals is present, whereas it is, in fact, not (the locals information is optional, and Petroglyph apparently decided to strip that. Can't blame them).

After some l33t hacking in luadec I got to work around this by tricking it into thinking it had locals information. Now it actually produced Lua code from the most basic of Lua files. However, when decompiling the more complicated Lua files luadec is unable to make sense of them, leaving me less then impressed with this decompiler.

I've even gone so far as to take a pot shot at writing a decompiler myself, but proper decompilation is hard, even for a 'simple' language such as Lua. Well, it is for me anyway.

On the other hand, I can of course output Lua assembly from the lua files; that's easy to do. But then this has to be manually decompiled into normal Lua code, something I'm sure most modders can't, won't (or even shouldn't) do.

Anyway, I just thought I'd put this information on the forums. Maybe someone who does know a thing or two about decompilation can use this info, or perhaps point me to, or adapt, another existing decompiler.


Update 2006/05/05: decompiled lua files are now available: http://alpha1.dyns.net/eaw/

Last edited by Mike.nl; 05-05-2006 at 04:16 PM.
Mike.nl is offline   you may: quote & reply,
Old 04-18-2006, 10:38 AM   #2
Orao
Junior Member
 
Orao's Avatar
 
Join Date: Jan 2006
Posts: 266
If you could figure out this one it will save us alot of trouble especialy with the AI.

Thank you for you findings so far. COntinue the good job.



EaW: Total Realism
mod....
TR mod team status: TR Advisor, programmer, map maker, Multimedia developer
Orao is offline   you may: quote & reply,
Old 04-18-2006, 11:02 AM   #3
Orao
Junior Member
 
Orao's Avatar
 
Join Date: Jan 2006
Posts: 266
BTW I just forgot to ask you. What files do you edit for scripted action on the welcome screen ?

Thank you.



EaW: Total Realism
mod....
TR mod team status: TR Advisor, programmer, map maker, Multimedia developer
Orao is offline   you may: quote & reply,
Old 04-18-2006, 01:26 PM   #4
Admiral Sith
Junior Member
 
Admiral Sith's Avatar
 
Join Date: Aug 2005
Posts: 375
Thats in the gameconstants.xml, its actually a replay of your most recent recorded game, a feature that was, unfortunatly, removed. So you get stuck with that same one.



"It may be that the old astrologers had the truth exactly reversed, when they believed that the stars controlled the destinies of men. The time may come when men control the destinies of stars." -Arthur C. Clarke

"Ideals are like stars: you will not succeed in touching them with your hands, but like the seafaring man on the ocean desert of waters, you choose them as your guides, and following them, you reach your destiny." -Carl Schurz
Admiral Sith is offline   you may: quote & reply,
Old 04-19-2006, 03:30 AM   #5
Slocket
Junior Member
 
Join Date: Jan 2006
Posts: 309
Well Mike good job so far! This is the REAL editor I am looking for. Can you give us a version of the LuaDec you modified?

Quote:
On the other hand, I can of course output Lua assembly from the lua files; that's easy to do. But then this has to be manually decompiled into normal Lua code, something I'm sure most modders can't, won't (or even shouldn't) do.

I programmed way to much in assembler decades ago when computers were very 'dumb'. Even the decompiled asssembler code can be of great use to do basic modification of integer -say what the AI will use for a raid fleet total number or what it thinks it needs to attack your fleet.

Then by going into the regular files, it can be hexed in. I did this for the DLL in Freelancer to get around some annoying behaviour to allow playing the game in single player mode without the storyline getting into the way.

So if you can give out a copy, I think quite abit of simple modding of the scripts can be done to some extent using the assembler code as a guide.

This well give the game a boost by making the AI smarter.
Slocket is offline   you may: quote & reply,
Old 04-19-2006, 04:28 AM   #6
Slocket
Junior Member
 
Join Date: Jan 2006
Posts: 309
Hey, the old demo is written slightly different than the final game release Lup. I manage to follow you advice for the old demo file Lua 5.0 and it decompiled quite a bit until it crashed! lol

I thought you may find this interesting. I only used LuaDec on BuildTurboLaser.lua of the old demo. I change the header from Lua(51) to Lua(50). No other change yet and it decompiled the bugger until it hit the 'end' or that integer thingy. The other thing it seems to have a problem with handling End of Line.

If you can show me what you did with the LuaDec I think with this information; you will have a decompiler of some sort.

BuildTurboLasers.lua output before crash:

----------------------------------------------
1 GETGLOBAL 0 0
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{require}

----------------------------------------------
2 LOADK 1 1
SET_CTR(Tpend) = 1

next bool: 0
locals(0):
vpend(0):
tpend(2): 0{require} 1{"pgevents"}

----------------------------------------------
3 CALL 0 2 1

next bool: 0
locals(0):
vpend(0):
tpend(0):
require("pgevents")

----------------------------------------------
4 CLOSURE 0 0
----------------------------------------------
1 LOADK 0 1
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{"Build_Turbolasers"}
()

----------------------------------------------
2 SETGLOBAL 0 0
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(1): -1{Category="Build_Turbolasers"}
tpend(0):
()

----------------------------------------------
3 LOADBOOL 0 1 0
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{true}
()
Category = "Build_Turbolasers"

----------------------------------------------
4 SETGLOBAL 0 2
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(1): -1{IgnoreTarget=true}
tpend(0):
()
Category = "Build_Turbolasers"

----------------------------------------------
5 NEWTABLE 0 1 0

next bool: 0
locals(0):
vpend(0):
tpend(0):
()
Category = "Build_Turbolasers"
IgnoreTarget = true

----------------------------------------------
6 NEWTABLE 1 2 0

next bool: 0
locals(0):
vpend(0):
tpend(0):
()
Category = "Build_Turbolasers"
IgnoreTarget = true

----------------------------------------------
7 LOADK 2 4
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 2{"StructureForce"}
()
Category = "Build_Turbolasers"
IgnoreTarget = true

----------------------------------------------
8 LOADK 3 5
SET_CTR(Tpend) = 1

next bool: 0
locals(0):
vpend(0):
tpend(2): 2{"StructureForce"} 3{"Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}
()
Category = "Build_Turbolasers"
IgnoreTarget = true

----------------------------------------------
9 SETLIST 1 1
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 1{{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}}
()
Category = "Build_Turbolasers"
IgnoreTarget = true

----------------------------------------------
10 SETLIST 0 0
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{{{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}}}
()
Category = "Build_Turbolasers"
IgnoreTarget = true

----------------------------------------------
11 SETGLOBAL 0 3
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(1): -1{TaskForce={{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}}}
tpend(0):
()
Category = "Build_Turbolasers"
IgnoreTarget = true

----------------------------------------------
12 RETURN 0 1 0

next bool: 0
locals(0):
vpend(0):
tpend(0):
()
Category = "Build_Turbolasers"
IgnoreTarget = true
TaskForce = {{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}}

SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{function()
Category = "Build_Turbolasers"
IgnoreTarget = true
TaskForce = {{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}}
end
}
require("pgevents")

----------------------------------------------
5 SETGLOBAL 0 2
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(1): -1{Definitions=function()
Category = "Build_Turbolasers"
IgnoreTarget = true
TaskForce = {{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses = 1"}}
end
}
tpend(0):
require("pgevents")

----------------------------------------------
6 CLOSURE 0 1
----------------------------------------------
1 GETGLOBAL 0 0
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{Sleep}
()

----------------------------------------------
2 LOADK 1 1
SET_CTR(Tpend) = 1

next bool: 0
locals(0):
vpend(0):
tpend(2): 0{Sleep} 1{1}
()

----------------------------------------------
3 CALL 0 2 1

next bool: 0
locals(0):
vpend(0):
tpend(0):
()
Sleep(1)

----------------------------------------------
4 GETGLOBAL 0 2
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{StructureForce}
()
Sleep(1)

----------------------------------------------
5 GETTABLE 0 0 253
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{StructureForce.Set_As_Goal_System_Removable}
()
Sleep(1)

----------------------------------------------
6 LOADBOOL 1 0 0
SET_CTR(Tpend) = 1

next bool: 0
locals(0):
vpend(0):
tpend(2): 0{StructureForce.Set_As_Goal_System_Removable} 1{false}
()
Sleep(1)

----------------------------------------------
7 CALL 0 2 1

next bool: 0
locals(0):
vpend(0):
tpend(0):
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)

----------------------------------------------
8 GETGLOBAL 0 4
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{AssembleForce}
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)

----------------------------------------------
9 GETGLOBAL 1 2
SET_CTR(Tpend) = 1

next bool: 0
locals(0):
vpend(0):
tpend(2): 0{AssembleForce} 1{StructureForce}
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)

----------------------------------------------
10 CALL 0 2 1

next bool: 0
locals(0):
vpend(0):
tpend(0):
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)
AssembleForce(StructureForce)

----------------------------------------------
11 GETGLOBAL 0 2
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{StructureForce}
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)
AssembleForce(StructureForce)

----------------------------------------------
12 GETTABLE 0 0 255
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{StructureForce.Set_Plan_Result}
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)
AssembleForce(StructureForce)

----------------------------------------------
13 LOADBOOL 1 1 0
SET_CTR(Tpend) = 1

next bool: 0
locals(0):
vpend(0):
tpend(2): 0{StructureForce.Set_Plan_Result} 1{true}
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)
AssembleForce(StructureForce)

----------------------------------------------
14 CALL 0 2 1

next bool: 0
locals(0):
vpend(0):
tpend(0):
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)
AssembleForce(StructureForce)
StructureForce.Set_Plan_Result(true)

----------------------------------------------
15 GETGLOBAL 0 6
SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{ScriptExit}
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)
AssembleForce(StructureForce)
StructureForce.Set_Plan_Result(true)

----------------------------------------------
16 CALL 0 1 1

next bool: 0
locals(0):
vpend(0):
tpend(0):
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)
AssembleForce(StructureForce)
StructureForce.Set_Plan_Result(true)
ScriptExit()

----------------------------------------------
17 RETURN 0 1 0

next bool: 0
locals(0):
vpend(0):
tpend(0):
()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)
AssembleForce(StructureForce)
StructureForce.Set_Plan_Result(true)
ScriptExit()

SET_CTR(Tpend) = 0

next bool: 0
locals(0):
vpend(0):
tpend(1): 0{function()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)
AssembleForce(StructureForce)
StructureForce.Set_Plan_Result(true)
ScriptExit()
end
}
require("pgevents")
Definitions = function()
Category = "Build_Turbolasers"
IgnoreTarget = true
TaskForce = {{"StructureForce", "Q_Galactic_Turbolaser_Tower_Defenses | X_Galactic_Turbolaser_Tower_Defenses =

Last edited by Slocket; 04-19-2006 at 05:17 AM.
Slocket is offline   you may: quote & reply,
Old 04-19-2006, 07:17 AM   #7
Mike.nl
Rookie
 
Mike.nl's Avatar
 
Join Date: Jan 2006
Location: Netherlands
Posts: 52
I've written a small (command-line) tool that will convert Lup files to Lua files and back: lup2lua (and its source).

Second, here are my (small) modifications to luadec: luadec-0.6-mod.
I only changed two things in the original file, and those changes are marked by my name, so you should find them easily. Running the resulting luadec over BuildTurboLasers.lua results in:

C:\>move BUILDTURBOLASERS.LUA buildturbolasers.lup
C:\>lup2lua buildturbolasers.lup buildturbolasers.lua
C:\>luadec buildturbolasers.lua

require("pgevents")
Definitions = function()
Category = "Build_Turbolasers"
IgnoreTarget = true
TaskForce = {{"StructureForce", "E_Galactic_Turbolaser_Tower_Defenses | R_Galactic_Turbolaser_Tower_Defenses = 1"}}
end

StructureForce_Thread = function()
Sleep(1)
StructureForce.Set_As_Goal_System_Removable(false)
AssembleForce(StructureForce)
StructureForce.Set_Plan_Result(true)
ScriptExit()
end

StructureForce_Production_Failed = function(L0, L1)
DebugMessage("%s -- Abandonning plan owing to production failure.", tostring(Script))
ScriptExit()
end

Yet, running it on larger, more complex lua files will crash the decompiler as usual. I've tried to find out why, but I don't understand the laudec code that well and it seems like more than just a superficial problem.

Anyway, I'll create a lua disassembler over the next days, if that'll help you guys.
Mike.nl is offline   you may: quote & reply,
Old 04-25-2006, 10:03 PM   #8
Slocket
Junior Member
 
Join Date: Jan 2006
Posts: 309
Sure, please do. I do not think we will get a LUA editor for now. With a disassembler, we can (the hard way) change any small aspect of the game that many modders may like.

I for one would like to change the way the DeathStar Battle is played out. If it means to manually recreating the script from assembler -so be it. It would only be a few hours work to type up the text script. I assume the EXE will read the expanded form of it.

Thanks for all your hard work Mike!
Slocket is offline   you may: quote & reply,
Old 05-01-2006, 06:53 PM   #9
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
Quote:
Originally Posted by Mike.nl
Anyway, I'll create a lua disassembler over the next days, if that'll help you guys.
I had just used winhex to update the sig bytes before I found out about the extra integer's and thus luadec's failure to decompile. Decided to google at that point, and glad I did!

I've written several decompilers over the years, so I'll see what I can figure out about the crash with luadec.

Thanks for the effort you've put into this! What an awesome place to start!
squeegee is offline   you may: quote & reply,
Old 05-01-2006, 07:13 PM   #10
popcorn2008
Little Green Friend
 
popcorn2008's Avatar
 
Join Date: Aug 2005
Location: Ypsilanti, MI
Posts: 1,239
Good work Mike! It would be great to have a community made LUA deassembler until/if Petro releases their own.


> Lucas Forums Moderator
> Petroglyph Forums Moderator
> Unofficial Petroglyph Fan Forums Global Moderator
popcorn2008 is offline   you may: quote & reply,
Old 05-01-2006, 07:44 PM   #11
Foshjedi2004
Junior Member
 
Foshjedi2004's Avatar
 
Join Date: Aug 2005
Location: Buckinghamshire, England
Posts: 478
LoW's Coder Alisama is busy doing Model File Comparisons So We may have a Filefront Community Model Converter and Bone Editor



My Personal Opinions are not Endorsed by Any Websites, persons or places.
Check out my Gallery
[/color]
Foshjedi2004 is offline   you may: quote & reply,
Old 05-03-2006, 03:51 PM   #12
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
I figured out the luadec crash: It's because of the lack of local information.

I coded around it by using _TEMP_ for all local variable names. At least that got me able to decompile everything I've tried it on so far, but make no mistake, it's a kluge (for example, if there are 5 locals, all 5 are named _TEMP_ - not good!).

Email me if you want a copy of the modified source (no place to put it online...).

I will work on a proper solution to the local variable issue next. I just have to do some heavy code reading to understand luadec parse structure...
squeegee is offline   you may: quote & reply,
Old 05-03-2006, 04:49 PM   #13
Athanasios
Forumite
 
Athanasios's Avatar
 
Join Date: Oct 2005
Location: Thessaloniki, Greece
Posts: 561
Quote:
Originally Posted by squeegee
I figured out the luadec crash: It's because of the lack of local information.

I coded around it by using _TEMP_ for all local variable names. At least that got me able to decompile everything I've tried it on so far, but make no mistake, it's a kluge (for example, if there are 5 locals, all 5 are named _TEMP_ - not good!).

Email me if you want a copy of the modified source (no place to put it online...).

I will work on a proper solution to the local variable issue next. I just have to do some heavy code reading to understand luadec parse structure...
Can't you set a generator for random naming of local variables?

I have no clue of LUA coding, but in C, C++, Java and such this is doable. Either with matrices of strings or pointers.



EaW: Total Realism
mod....
TR mod team status: TR Leader, programmer, map maker, Multimedia & Website developer & administrator, Public Relations manager
Athanasios is offline   you may: quote & reply,
Old 05-03-2006, 05:15 PM   #14
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
Quote:
Originally Posted by Athanasios
Can't you set a generator for random naming of local variables?

I have no clue of LUA coding, but in C, C++, Java and such this is doable. Either with matrices of strings or pointers.
Actually, my original kluge was pretty stupid. As it turns out, all the crashes I've come across are related to the local vars. All I had to do was before using a local variable check for it being null, and if it was, call the function Mike created. The only other change was to make sure not to free the auto locals on function closure.

So, now, all local variables have distinct names of L0, L1, L2, etc. The best we can do without the local variable information (i.e. not knowing the actual local variable names). The code is quite readable, and I have successfully decompiled all the lua code in config.meg now.

There is one other strange bug in luadec: When a function is declared, the line of code generated is like "name = function()" instead of "function name()". It's annoying, but the code is still readable. I'll look into eventually...

I can make a zip of all the decompiled sources if anybody is interested. As well as the fixed luadec and code if interested.
squeegee is offline   you may: quote & reply,
Old 05-03-2006, 05:50 PM   #15
popcorn2008
Little Green Friend
 
popcorn2008's Avatar
 
Join Date: Aug 2005
Location: Ypsilanti, MI
Posts: 1,239
If you could do us the favor and make a zip of the decompiled scripts and modified luadec it would be great.

It could give us a head start on using Lua.


> Lucas Forums Moderator
> Petroglyph Forums Moderator
> Unofficial Petroglyph Fan Forums Global Moderator
popcorn2008 is offline   you may: quote & reply,
Old 05-03-2006, 05:55 PM   #16
Athanasios
Forumite
 
Athanasios's Avatar
 
Join Date: Oct 2005
Location: Thessaloniki, Greece
Posts: 561
I would suggest to work closely with Mike, so that you break Lua till the end. So that we can even compile models too.

Yet, in the meanwhile, you can upload what's you've done. Just mind to post this to PFF forums too, there are people interested for this there too!



EaW: Total Realism
mod....
TR mod team status: TR Leader, programmer, map maker, Multimedia & Website developer & administrator, Public Relations manager
Athanasios is offline   you may: quote & reply,
Old 05-03-2006, 06:18 PM   #17
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
Quote:
Originally Posted by Athanasios
I would suggest to work closely with Mike, so that you break Lua till the end. So that we can even compile models too.

Yet, in the meanwhile, you can upload what's you've done. Just mind to post this to PFF forums too, there are people interested for this there too!
Agreed, just waitng for a response. luadec has some internal problems, so I am looking into the actual structure of tokenized lua now to see if it's easier to start over or attempt to fix luadec.

From my perspective, luadec is more complicated than it needs to be (it appears as though it actually attempts to walk the code by pseudo-executing it, which is, uh, not the way I'd do it...).

As I mentioned before, I've written several deompilers so I am quite familiar with how to get one up and running quickly, and correctness is the most important thing for a decompiler IMO, and luadec still bails on some simple valid structures (which I just noticed the failure messages in a few of the files I decompiled - although 13 failures out of 260 isn't _that_ horrible for our purposes).
squeegee is offline   you may: quote & reply,
Old 05-04-2006, 04:40 AM   #18
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
Ok, I've found what I consider the fundamental issue with luadec: It's not decompiling the binary code directly, but using Lua to load and walk the code. As a result, the internals are quite ugly, as a lot of Lua specific internal states, upvalues, etc. all have to be tracked.

The odd thing is that none of that is needed. The object format is very straight forward, and the mapping from the 35 opcodes to the source is almost direct. Even without the debugging info (function names, local variable names), recreating the original source should be doable.

Anyway, I've been studying chunkspy as well (a disassembler for Lua object code writen in Lua - only 2100 lines, with lots of comments), and I think I am going to work on creating something from scratch over the weekend. I'll post the progress.

This is so much more fun than working on work! :-)
squeegee is offline   you may: quote & reply,
Old 05-04-2006, 09:37 AM   #19
Athanasios
Forumite
 
Athanasios's Avatar
 
Join Date: Oct 2005
Location: Thessaloniki, Greece
Posts: 561
Squeegee, mind that decompliling is one step, next step will be re-compiliing our modded lua files. As for opcodes, you should pay very attention so that we dont mess with micromemory (virtual i guess?), but you already know this .

If you can do both of these, then you should be given the "golden EaW metal" and ill be the first one to contact admins for this !

As soon as we can get our hands dirty with lua scripting.........our way is open .

Keep up the good work!



EaW: Total Realism
mod....
TR mod team status: TR Leader, programmer, map maker, Multimedia & Website developer & administrator, Public Relations manager
Athanasios is offline   you may: quote & reply,
Old 05-05-2006, 02:11 PM   #20
Mike.nl
Rookie
 
Mike.nl's Avatar
 
Join Date: Jan 2006
Location: Netherlands
Posts: 52
Allright, this is it!

I have finished two tools that:
  • Converts between the official Lua format and Petroglyph's Lua format.
  • Disassembles a Lua 5.0 object file.
I used these tools (among others) to decompile most Lua files (91%, lacking some story scripts for now).

Yes, that's right. Lua sources are available.
Along with instructions on how to create custom Lua files for Empire at War.
Get it all here: http://alpha1.dyns.net/eaw/

Lacking a good decompiler, I've manually decompile everything luadec crashed on, and I still have to do the remaining story scripts, but I decided to release the Luas I've decompiled so far so you guys have something to work on

Of course, keep in mind the source is uncommented and uses a bit vague function arguments, which is inherent to any decompilation. Perhaps I will one day rename them to something meaningful, based on what they do, once I have time.

Anyway, as usual, if you have questions or remarks (both the good and the bad kind), post here.
Mike.nl is offline   you may: quote & reply,
Old 05-05-2006, 02:39 PM   #21
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
Quote:
Originally Posted by Mike.nl
Allright, this is it!

I have finished two tools that:
  • Converts between the official Lua format and Petroglyph's Lua format.
  • Disassembles a Lua 5.0 object file.
I used these tools (among others) to decompile most Lua files (91%, lacking some story scripts for now).
Kewl.

Quote:
Originally Posted by Mike.nl
Yes, that's right. Lua sources are available.
Along with instructions on how to create custom Lua files for Empire at War.
Get it all here: http://alpha1.dyns.net/eaw/

Lacking a good decompiler, I've manually decompile everything luadec crashed on, and I still have to do the remaining story scripts, but I decided to release the Luas I've decompiled so far so you guys have something to work on
Ouch. Thanks for taking the time. I also have decompiled all but the 13 files luadec ended up bailing on (I fixed some of the crahes to get that far).

Quote:
Originally Posted by Mike.nl
Of course, keep in mind the source is uncommented and uses a bit vague function arguments, which is inherent to any decompilation. Perhaps I will one day rename them to something meaningful, based on what they do, once I have time.

Anyway, as usual, if you have questions or remarks (both the good and the bad kind), post here.
I am working on a good decompiler. It can decode the binary files for both debug and non-debug already (like what -list can do with luac), so today I am starting the source code generation.

Unlike luadec, this loads the binary object directly, wihtout Lua's involvement. The code is only 500 lines of c++ so far. I should have the source generation working reliably within another couple of days. I've named it luad (I know, how original...).

Once this is done, along with your lup2lua tool, and luac, we should be able to decompile, modify, and recompile the sources!

If you want, we can combine all of the tools into one, so it can be done easier for the non-tech types out there.

Great job!
squeegee is offline   you may: quote & reply,
Old 05-05-2006, 03:49 PM   #22
Athanasios
Forumite
 
Athanasios's Avatar
 
Join Date: Oct 2005
Location: Thessaloniki, Greece
Posts: 561
Thumbs up

MUST BE STICKIED I-M-M-E-D-I-A-T-E-L-L-Y !!!

Also Mike, do not forget to post this over PFF forums too. The people will go mad!

And someone give those guys a golden medal . Mike, i suggest you work with squeege so that we get 100% bug-free and 100% commented lua files.

Check your pm also .

[edit] I posted this for you in PFF, just in case you're not registered .



EaW: Total Realism
mod....
TR mod team status: TR Leader, programmer, map maker, Multimedia & Website developer & administrator, Public Relations manager

Last edited by Athanasios; 05-05-2006 at 04:08 PM.
Athanasios is offline   you may: quote & reply,
Old 05-05-2006, 04:32 PM   #23
Mike.nl
Rookie
 
Mike.nl's Avatar
 
Join Date: Jan 2006
Location: Netherlands
Posts: 52
Quote:
Originally Posted by Athanasios
Mike, i suggest you work with squeege so that we get 100% bug-free and 100% commented lua files.
I'm pretty sure the code is bug free. I recompiled my decompiled code and the resulting object files was an exact binary match with the original.

And we wouldn't want 100% commented lua files. That would leave no room for the actual code

Quote:
Originally Posted by Athanasios
I posted this for you in PFF, just in case you're not registered .
Yeah, I'm not, so thanks

Actually, the biggest challenge now is to create documentation. Establish an API. Find out how the Luas interface with the game and how far that interface can be stretched.


And to squeegee:
I'm looking forward to your decompiler. It would certainly be nice to bundle these tools.
Out of curiosity, how robust will your decompiler be? Will it depend heavily on known constructs by the compiler? Or is it able to generate equivalent source code from other, but functionally equivalent code as well?
For instance, the compiler compiles while loops with the test at the end, but what if the test were moved to the beginning? And what if the assembly was more optimized in terms of register (re)usage? The standard lua compiler is pretty brain-dead from what I can tell.

But I do agree, the lua object format is ridiculously simple, that's why luadec never impressed me that much, with all its bloat.
Mike.nl is offline   you may: quote & reply,
Old 05-05-2006, 04:58 PM   #24
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
Quote:
Originally Posted by Mike.nl
And we wouldn't want 100% commented lua files. That would leave no room for the actual code
Comments aren't stored in the object files, so it wouldn't matter. I'm sure you know this, just making sure everyone else does.


Quote:
Originally Posted by Mike.nl
Out of curiosity, how robust will your decompiler be? Will it depend heavily on known constructs by the compiler? Or is it able to generate equivalent source code from other, but functionally equivalent code as well?
For instance, the compiler compiles while loops with the test at the end, but what if the test were moved to the beginning? And what if the assembly was more optimized in terms of register (re)usage? The standard lua compiler is pretty brain-dead from what I can tell.
It will recreate the original source as close as possible; It will generate the exact same object, or it's not done (I mentioned before that IMO a decompiler's most important trait is accuracy).

The lua compiler (tokenizer...) doesn't need to optimize much, as the vm assembly is so simple, and the vm itself can optimize stuff out. If you read some of the vm asm specific stuff out there, they point these things out, as well as other quirks. Most of those doc's are dated in the last 6 to 9 months, so they didn't exist when luadec was written.

My "plan" is to be able to decompile any lua source. Once that works perfectly, then I'll derive a version specific to EAW/Lup, and we can combine the stuff into a single executable. Maybe even provide a simple GUI so non-techies can use it (of course, maybe they shouldn't be modifying the code anyway... ;-).

Quote:
Originally Posted by Mike.nl
But I do agree, the lua object format is ridiculously simple, that's why luadec never impressed me that much, with all its bloat.
Luadec's issue is the writer evidently hasn't written anything of the sort before, and attempted to use Lua's code to do some/most of the work.

What I have is entirely stack based, and thus has no limits on depth of functions, loops, etc. (well, other than the memory of the machine it's running on). By definition, it can only re-generate what the object code contains. That is, if the compile moves things around (which I haven't seen a case of yet, btw), then it will regenerate it that way, as there is no way to know different. Functionally, though, it'd be the same as the original, and the object code will be the same.

Looking forward to getting this done; It's been a lot of fun so far!

BTW I can send the code to anyone who wants it at any time. It'll be public domain or BSD/MIT styled licensed when I get done with it.

Also, Mike, if you want the version of luadec that has fixed most of the crashes, let me know. I have a site I can put it on, I just have to put some work into the web site first (the domain is for email, but the http hosting is included but currently unused).
squeegee is offline   you may: quote & reply,
Old 05-06-2006, 02:23 AM   #25
Slocket
Junior Member
 
Join Date: Jan 2006
Posts: 309
Thumbs up

A big thank you to you guys! You know your stuff very well (this is way above my head). It will be good to have even more versions to work with, in case something came up for redunancy.

As for me, the 'technically challenged', I can play with the AI game mechanics and see what happens! lol
Slocket is offline   you may: quote & reply,
Old 05-07-2006, 03:37 AM   #26
fortune faded13
Rookie
 
Join Date: Mar 2006
Posts: 78
this sounds very good. i have almost no knowledge of LUA or anything of that sort, but im gonna look into it later this weekend when i have time.
way to go mike!
fortune faded13 is offline   you may: quote & reply,
Old 05-16-2006, 01:36 AM   #27
GhOsT-Jedi
Rookie
 
GhOsT-Jedi's Avatar
 
Join Date: Oct 2005
Location: Past the outer rims of the universe training my sith armies.
Posts: 159
If I read through all you posted, I could learn somthing, but my god you guys know your stuff, Keep up the good work






GhOsT-Jedi is offline   you may: quote & reply,
Old 05-16-2006, 04:05 PM   #28
Slocket
Junior Member
 
Join Date: Jan 2006
Posts: 309
Using Mike's new cool LUA Tools I was able to make a first Enhanced_AI_RAID LUA Module!

I wanted more of a challenging RAID ability for the Rebels. Now for the first iteration the AI will use up to 6 units plus the normal heros and commandos, and have a faster retreat time. Hit and Run tactics really is almost human like and to be feared!

Later you can change what the AI considers for target priority and task force mix. It really does allow you change the AI behavior to be smarter!

I am (have to) re-install EAW and making sure it is fully functional under patch 1.4. Once released, modders should try out the LUA scripting to make changes to the AI behaviour for the better after seeing what a HUGE difference it can make.
Slocket is offline   you may: quote & reply,
Old 05-17-2006, 06:25 PM   #29
Slocket
Junior Member
 
Join Date: Jan 2006
Posts: 309
Just to let you know, the patch 1.4 gives you the decompiled source LUA to use!

I am testing it to see if the game likes it if it is modified.
Slocket is offline   you may: quote & reply,
Old 05-17-2006, 06:31 PM   #30
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
Wow. I will re-decode the meg's and check it out. Thanks for the heads up!
squeegee is offline   you may: quote & reply,
Old 05-18-2006, 02:13 PM   #31
Slocket
Junior Member
 
Join Date: Jan 2006
Posts: 309
The modified Lua will work in game. You can also add extra lines to it like C language. I was 'copy paste' command blocks to add extra stuff IE a Huge Rebel Invasion Fleet when it decides to "Crush.lua"...and it is awesome.

The RIAD_AI_BOOST works fine and is ready to go. I tesed it. Now fear a Rebel raid for sure! lol
Slocket is offline   you may: quote & reply,
Old 05-18-2006, 03:47 PM   #32
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
Quote:
Originally Posted by Slocket
Just to let you know, the patch 1.4 gives you the decompiled source LUA to use!

I am testing it to see if the game likes it if it is modified.
Really? Where? After I moved out the decompiled sources I created from luadec, and re-extracted the new meg files, the LUA files are still in the same luap format as before. Where did you find the source for them from 1.4?
squeegee is offline   you may: quote & reply,
Old 05-18-2006, 06:35 PM   #33
Mike.nl
Rookie
 
Mike.nl's Avatar
 
Join Date: Jan 2006
Location: Netherlands
Posts: 52
I'm not sure if Slocket was correct by saying patch 1.4. It could be from the map editor as well.

Anyway, look in GameData\Mods\Source\Data\Scripts.
Mike.nl is offline   you may: quote & reply,
Old 05-18-2006, 07:03 PM   #34
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
Quote:
Originally Posted by Mike.nl
I'm not sure if Slocket was correct by saying patch 1.4. It could be from the map editor as well.

Anyway, look in GameData\Mods\Source\Data\Scripts.
That directory structure doesn't exist with just the 1.4 update (specifically GameData/Mods), so it must be the map editor, which I haven't got around to yet. I'll go grab that next.

Thanks for the clarification Mike!
squeegee is offline   you may: quote & reply,
Old 05-19-2006, 05:14 PM   #35
Admiral Sith
Junior Member
 
Admiral Sith's Avatar
 
Join Date: Aug 2005
Posts: 375
Map Editor = Mod Tools. Including a exporter for ALO format :P



"It may be that the old astrologers had the truth exactly reversed, when they believed that the stars controlled the destinies of men. The time may come when men control the destinies of stars." -Arthur C. Clarke

"Ideals are like stars: you will not succeed in touching them with your hands, but like the seafaring man on the ocean desert of waters, you choose them as your guides, and following them, you reach your destiny." -Carl Schurz
Admiral Sith is offline   you may: quote & reply,
Old 05-19-2006, 05:18 PM   #36
squeegee
Rookie
 
Join Date: May 2006
Posts: 11
Quote:
Originally Posted by Admiral Sith
Map Editor = Mod Tools. Including a exporter for ALO format :P
Yeah, I knew that, I had let eaw dl and install the 1.4 update itself, so I didn't get the editor. I have it now, and all the scripts are in source form.

So much for needing a decompiler. I am still working on it, but work has been getting in the way (if only I worked for google...:-).

With mike's lup converter and the source, we should have everything needed to make changes to the ai now.
squeegee is offline   you may: quote & reply,
Old 05-20-2006, 07:51 AM   #37
Michael Plant
Rookie
 
Michael Plant's Avatar
 
Join Date: Feb 2004
Location: England
Posts: 44
We don't need a converter, I tried replacing the lua files in gamedata\data\scripts folder with the ones that came with the map editor, and the game still worked.


*Michael*
spoiler:
Hello!
Michael Plant is offline   you may: quote & reply,
Old 11-09-2006, 03:14 AM   #38
Slocket
Junior Member
 
Join Date: Jan 2006
Posts: 309
I am trying to decompile the new Forces of Corruption LUA with Mike's tools. Been awhile and his site for instructions is gone.

I think it is the same as EAW. Now where did I place his LuaDec 0.6 program. Any else have a copy?

EDIT: Found the files, just have to look at the URL home. The download section was moved.

http://alpha1.dyndns.info/eaw/LuaFiles

I was able to decompile into some kind of assember code...

Last edited by Slocket; 11-09-2006 at 03:27 AM.
Slocket is offline   you may: quote & reply,
Old 11-12-2006, 12:15 AM   #39
Kelathin
Rookie
 
Join Date: Sep 2006
Posts: 10
likely pg smartened up
Kelathin is offline   you may: quote & reply,
Old 05-07-2007, 06:28 PM   #40
CLONECOMMANDER501
Junior Member
 
Join Date: Aug 2006
Posts: 369
have any new tools, mike?


CLONECOMMANDER501 is offline   you may: quote & reply,
Post a new thread. Add a reply to this thread. Indicate all threads in this forum as read. Subscribe to this forum. RSS feed: this forum RSS feed: all forums
Go Back   LucasForums > Network > Empire At War > EaW General Discussion > Modding forum > Some Lua details

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 01:11 AM.

LFNetwork, LLC ©2002-2011 - All rights reserved.
Powered by vBulletin®
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.