Jade Empire modding (see first post for info summary)
 

Index - useful resource links:

JE Modding tools (so far):

• TalkEd v1.1.1a - TLK V4.0 editor for Jade Empire (for modifying all game text in dialog.tlk)
  
  
• ERF/RIM Edit v0.3.6a - ERF/MOD/RIM Editor for Jade Empire (with correct file type associations)
  
  
• JEFindRes v0.3a - Search tool to look for files with names matching the search pattern within all the game's BIF, MOD and RIM files (and the override folder for completeness). Allows extraction of found files and converting images from TXB format to TGA. (See this post for more info)
  
  
• Txb2Tga v0.1a - Converts TXB textures to a TGA and TXI file to allow viewing and modifying textures and images.
  
  
• JE DLGEditor v2.3.0a2 - Dialog editor for Jade Empire, alpha release (See this post for more info )
  
  
• Jade Empire Savegame Editor v0.6 - A basic savegame editor for Jade Empire (see this post for more info)
  
  
• SAC Utils - A save.sac ripper and mender in case you want to try manual GFF editing of your savegame.

Other Modding tools that work with JE:

• Scripts can be compiled with the KOTOR-adapted versions of nwnnsscomp using nwscript.nss from JE's override folder.
  
  
• 2DAs can be edited with any text editor if they're in 2DA V2.0 format. If they are in the 2DA v2.b binary format they can be edited with KotorTool or tk102's Excel Add-in or converted to 2DA V2.0 with my Convert2da utility.
  
  
• K-GFF v1.2.0 and above seems to work to edit the GFF files from Jade Empire. Make sure you don't use an older version since the new data type JE uses would not be supported then, resulting in data corruption when saving.

Modding info, tutorial and how-to posts:

• Making new gems, effects and techniques
• Modifying the combat system
• Model format information

Released JE Mods in this thread:

• stoffe's "White" Radiant Jen-Zi reskin
  
• sekan's "Three Brothers Gems"
  
• sekan's "Elephant Demon Style & Moon Light Weapon Style"

Original post:

Since I just got Jade Empire for PC I thought I'd try to update my RIM/ERF editor to handle the Jade Empire files to make modding it a little easier. The ERF and RIM file formats seems to be mostly the same, though the ResType <--> File extension lookup table seems to have changed, both changing the meaning of some existing ResTypes, and adding a bunch of new file types to it.

I've been looking through various RIM and MOD files so far trying to figure out what the different files are. So far I've only been able to figure out this fairly short list. Do anyone know what other file type <--> ResType associations there are, or what the new files are used for?

Edit: Found a list, so hopefully the table should be complete now. Still need to figure out the purpose of some of the files, and what format they are in though. (Some may be leftovers from Neverwinter Nights / Knights of the Old Republic and unused in Jade Empire.)

That's a good news.
I've done few things like editing some 2da's and make some scripts.
For that I have use 2DA Converter/Merger Tool, ERF/RIM Editor and KotorTool.
Maybe you have a look into .tlk files? All I want is to set description for few items. :)

I've made a new version of TalkEd that should handle the TLK V4.0 format used by Jade Empire. It seems to work to read and modify the English TLK file with it from my quick testing. Though please keep an eye out for bugs if you use it since I've only had time to do very limited testing. You can download it here for now. (This version only works with the Jade Empire TLK files, not the KOTOR ones.)

There is one data field in the TLK file I don't know the purpose of currently. It seems to be a 32 bit unsigned integer which is set to 0xFFFFFFFF (-1 I suppose) for the majority of the TLK entries, though a fair number of them (mostly dialog related strings) has some other value set here, and it seems each value (except 0xFFFFFFFF) used is unique and set for just that one entry.

TalkEd just lets you set this number directly (using hexadecimal notation) for the time being until someone can figure out what the purpose of this data is.

Edit: I suspect the unknown numeric value refers to names of the numbered BIP/sound files found in for example audio.bif, though I haven't been able to verify this yet.

stoffe, have you tried poking around in the .key/.bif files to fill in the gaps for erf extensions? I found

this thread on Bioware's forums that references a tool that recognizes most of the key/bif extensions for Jade Empire. Just a thought.

I haven't, but the problem I can see with that is that the KEY/BIF format doesn't store file extensions either, as far as I am aware. So if the lookup table for ResID <--> File type has changed for Jade Empire tools made for other games would still output data with the wrong file extension, or no valid extension at all, since it has no way of knowing what those ResID's are supposed to be.

That the data files seems to be an even more disorganized mess than KOTOR, with files or the same type or files belonging to the same area scattered all over the place, doesn't make it easier to find all the file types that are used and how they fit together :)

It was my hope that the tool referenced above (in a broken url link, now fixed) would shed the light on some unknown extensions. A long shot I'm sure, but just a different angle to attack from.

Poked around the BIF files a bit, and while they do suffer from the same problem of incorrect/unknown file extensions I managed to figure out a few more things (updated the list in the first post).

I've made a JE version of my ERF/RIM Editor that uses the above file extension/ResType table (minus the couple of unknowns in the list) so far. While the list isn't complete it's easier to spot which files aren't properly identified when it doesn't use the KOTOR file associations. :)

If anyone is interested you can download it from the link in the first post in this thread. If you notice any file types that are listed starting with a # followed by a number those are of unknown types. Please let me know what file it it and where (in what file) you found them in those cases. :)

That fiels is for sound ("0xFFFFFFFF" value = no sound). I test this by replacing values and it works. However the sound file must be loaded in that module or to be present in override because if I set a sound which is used in other module will do nothing. I haven't test append new entry but I'm sure is no problem with that. This tool is perfect for what I need to finish my mod.
Many thanks stoffe.
Edit: I just found dialog tool and I should test-it tonight.
I like how easy is to attach scripts for diferite events by editing ar_*.2da files.

Ok, thanks for the confirmation. I've updated TalkEd to handle the values as sound reference numbers and not display them in hexadecimal notation.

Haven't had much time to look at how things fit together yet since I've only had the game installed for roughly a day so far, but that may be very useful if they don't have any debug console in the game (haven't managed to find one) :)

Another update...
 

While poking around in the RIM and MOD files I found some more file types I managed to figure out with reasonably certainty what they should be. I've updated the list in the first post and uploaded another version of ERF/RIM Edit with the recent changes, see the link in the first post in this thread to download.

Any other file types I've not managed to find yet?

Thanks for all the effort stoffe! =)

Now I just need to get off my lazy behind and install the darn game (DDO is evil... just Evil!) ;)

Kind of on topic here, but if someone could find a way to open the PC & NPC textures (they're in txb form) I would be ever so grateful. :D

(If this doesn't fit in here, please excuse me. ;))

@stoffe: many thanks again for all this tools.
I have no idea how to use a debug console or what is that
... if this info good for anything.
However, I belive to be possible to send mesajes to log or display by #include debug.

@mjpb3: I also look for this and for a decomp. for mdl files

The TXB format has some similarities to the TPC format used for the KOTOR games. The header is different, but like TPC it contains compressed bitmaps and optionally TXI data at the end.

Don't know if I'm good enough at this kind of programming to be able to make a converter that converts TXB --> TGA directly, but I'll give it a try and see how it goes. :)


I was just looking for an in-game debug console (similar to the yellow "> " you can type commands at in KOTOR1, and the invisible one in TSL), but it doesn't seem like Jade Empire has one. Hopefully most of that can be handled via scripts instead though.

I built a TPC Perl mdoule based on eiz's tpc2tga source code. If you want any assistance let me know.

Thanks, that will probably be needed if this is going to go anywhere at all. :) I had a look at the tpc2tga source to get an idea of how to decode the DXT data and was immediately reminded why I strongly dislike C++ (couldn't even get that code to compile, so I'm doing it in Delphi instead).

Edit: Nevermind, I was just stupid. :)

Whatever you can come up with would be excellent for me. That's the main thing I ever wanted to mod in Jade Empire--the textures. Thanks for anything you can accomplish. :D

It is indeed a reversal of bytes (0xAABBCCDD becomes 0xDDCCBBAA for example).

This is so the $cpx variable in decode_dxt1 can be right shifted 2 bits at a time and disregarded as the new $px is calculated. The blended array is set up in such a way to make sense of the two bits (which are now reversed).

see:
I don't have the tpc2tga source in front of me at the moment so I'm just working off the tpc.pm, btw -- the Perl code was a direct (more or less) translation of the C++.

Edit: crossed out the part about the 2 bits being reversed -- they aren't reversed -- their position within their bytes are unaltered but the bytes themselves are reordered.

That reminds me why I'm not too fond of Perl either. :) I haven't done anything with Perl for over 6 years, so I didn't understand much of that code. (I prefer high-level "almost english" languages like Delphi, C#, PHP, so it feels like I'm in way over my head with this.)

The unpack() function takes a string and splits it by token into an array, like explode() in PHP? While pack() takes an array and spits out a token-separated string from it?

What does your reverse_bytes() function receive as parameter, a token-separated string (using "V" as separator) containing the three values a texel-struct holds?

If so you convert the string to an array, immediately explode the array into a string again (this time using "C4" as token separator instead), reverse-sort the string, turn it into an array again (wouldn't "C4" be "4C" when you reverse the string?), and immediately turns the array back into a string (this time separated with "V" again) which you return? Wouldn't that return an array of 3 values?

Huh? I don't follow what this accomplishes. :) It obviously works, but I still don't understand what it does. :confused:

(As you don't have the source: The C++ code takes a struct as function parameter containing two 16 bit unsigned ints and one 32 bit unsigned int value... does something with those three values which I have absolutely no idea what, and then spits out a single 32 bit unsigned integer as result.)

* * *

Another question ... this is called in a loop that loops up to the Y-size of the bitmap while writing the TGA (i is the incrementing value in the loop):
fwrite(file->pixels+pitch*(ysz-i-1), 4, pitch, fp)

file->pixels is an array of 32 bit unsigned integers (the RGBA data), pitch is the X-dimension of the image (in pixels), ysz is the Y-dimension of the image

What exactly is it writing? file->pixels is an array, but it doesn't supply any array index in that function call... Does the file->pixels+pitch*(ysz-i-1) add pitch*(ysz-i-1) to the memory pointer of the pixels array and then write the four bytes found in that location? Why do this instead of using an array index?

Weird lower-level languages I haven't used for years confuse me. Use of pointers always confuse me. :)

No, The pack and unpack functions are Perl's way forcing data to/from certain interpretations like an unsigned long (template: V) or a unsigned byte (template: C). This is needed because of Perl's weak typing mechanisms. pack takes data forces it into a datatype while unpack translates coerced data back into perl scalars.


reverse_bytes accepts a perl scalar (which itself was populated from an unpack function with template V). It then packs this value into an unsigned long again, unpacks it into 4 unsigned bytes, reverses the array of bytes, repacks the array of bytes, and then unpacks the result as unsigned long.

Anyway the more elegant C++ code appears as this:
Here you can see better the masking and data shifting involved. The perl code accomplishes the same thing but uses a temporary array of bytes that it reverses and repacks into a long then unpacks to get the numeric value. :xp:

* * *

Yes that is exactly what it is doing, byte by byte. I believe this syntax was used because in the tpc.cpp file, you can see file->pixels+pitch*(ysz-i-1) is used to assign the values byte-by-byte. I think eiz just used copy/paste of the code as a mnuemonic so he could easily recognize what exactly he looping on.

The fwrite function is writing row by row starting at the end of the array. Think of the array as the image mirrored vertically. (Left-right is maintained but top-bottom is reversed.) I'm sorry for confusing you by saying byte-by-byte earlier.

Here's the declaration for fwrite We are using size=4, and nitems=pitch (the width in pixels of the iamge), so each fwrite is writing 4*pitch bytes. The fwrite function returns the number of items (not bytes) which should be equal to pitch in our case. The array's memory address begins at file->pixels, so at the beginning of the loop to write the last row from the array the offset would be (pitch*(ysz-1)). The next to last row would begin at (pitch*(ysz-2)).

HTH and sorry for not having the concentration earlier. :xp:

Ah, stupid me... I was thinking to something more complex... I saw a console_font file somewhere.

Great news about textures and I must thank you all for efforts.

But still something does not seem right with how I'm understanding this... In the above example each line of the image has 256 pixels... and each pixel is represented by a dword (4 byte integer) holding its RGBA value. In other words each line contains 4 * 256 = 1024 bytes, which you read from the array and write to the file.

But each iteration of the loop still only seems to back off pitch (=256) bytes. Now can it back up one row if it only backs up 25% of the data that is being read/written? Shouldn't it back up pitch*4 to get to the next line of data? Does the memory pointer voodoo deal with array elements and not byte positions in memory? I.e. it backs up pitch (=256) elements and not bytes?

Edit again: Yes! :emodanc: That seems to have done the trick. Seems like the utility works to convert at least those TXB files to TGA I have tested it on. :)

\o/ Yay for stoffe! Did you answer your own question then? The *4 part seems like it should be there to me too... but then again I'm not a C++ guru. I would've expected the use of brackets here to denote element iteration. (Don't like having to think at this level with my programming either, stoffe.) I remember this is the area I fumbled through with trial and error also until I got something recognizable. :D Good job.

More or less, I figured out how to replicate the behavior reasonably using array indexes instead. :)

The quick converter utility seems to work reasonably well on the few TXB textures I've tried to convert with it. I've uploaded it for now in case someone else wants to help test it and determine how well it works: Txb2Tga.

Use at your own risk. It seems to work OK, but I've only tested it on a handful of different TXB files. If you encounter any file it can't convert properly please let me know which file that was.

* * *

On more general notes it seems the NWNNSSCOMP for KOTOR (at least the variant tk102 did) works fine to compile scripts for Jade Empire if you use the nwscript.nss file found in the override folder and fix the couple of syntax errors it contains. :emodanc:

Jade Empire also seems to use a mix of the traditional plaintext 2DA V2.0 format, and the same binary 2DA V2.b format used with KOTOR. So the binary files can be edited with KotorTool, or converted to text using tk102's excel plugin or my quick converter utility.

Yes yes yes!!!! Thank you stoffe and tk102! It works! I was able to convert Scholar Ling's outfit and head texture. Just want I needed!

:bow: :bow: :bow:

Thanks for all your hard work, nowhere else is even close to this far for texture editing yet. But how do you get these new textures into the game now? Do you need to change them back to txb's? If so, how?

I assume the .tga files are readable as-is in the override directory. Bioware has been pretty good about that in the past, just as they do with v2.b and v2.0 2da files.

Still can't get it to work. I converted Radiant Jen Zi's body and edited it, then saved it as TGA image named P_Li01 which is the same as the txb file. I also tried naming it P_Li01.txb which I knew wouldn't work and it didn't. I deleted the actual txb file thinking it might just be loading that one first, and that didn't work either. Her body came up black, so I replaced it. Any Ideas?

It should work to put both the TGA and TXI file in the override folder directly. Make sure there isn't a corresponding TXB file located in the override folder as well, since it will take precedence over the TGA's. Many of the player appearance textures are located in the override folder as well as in players.mod (where those in override will be used since it has higher priority than the BIF/RIM/MOD files).

I did a quick test and recolored the shirt of my character green, moved p_blossom01.txb out of the override folder and put p_blossom01.tga and p_blossom01.txi in the override folder, and she turned up with a green shirt in-game.

(I.e. you shouldn't need to remove the TXB file from within the data files for the TGA to be used, just make sure one doesn't exist in the override folder.)


Edit: I did a quick reskin of the character you mentioned as well, and it showed up fine in-game:
http://img518.imageshack.us/img518/7...iantmb9.th.jpg

Thanks, I got it working now. I don't know why it didn't work the last time I got rid of the TXB though. Thanks again.

GFF editing observations
 

Seems like Jade Empire has added yet another data type to the GFF format (still without changing the version number :roleyess:). Type 18 appears to be a StrRef type, similar to ExoLocString in use but only containing a dialog.tlk reference and not any localized substring data. Be warned that these fields may get messed up and stripped if using any GFF editors for NWN or KOTOR to edit Jade Empire's GFF files.

Aside from that it seems JE uses the Rotation and Vector field types added in KOTOR as well, which would cause trouble when using GFF editors made for NWN.

So Field 16 and 17 exist-- that's good. Have you been able to tell if CResRef fields include up to 32 chars (like NWN2) or just 16 chars (like KotOR)? If it helps, I could modify either KGFF or NWN2GFF to include field 18. Send me a PM if you like. :)

As far as I can tell Jade Empire uses 16 character ResRefs. It seems to be the same format used for KOTOR except for (at least) one new data type that looks like it holds a StrRef from a quick look at it. It would be very useful if you could make a JE-compatible version of k-gff. :)

Is there any current information about decompiling and replacing models at this time?

KGFF v1.1.9 now supports StrRef fields (Field Type: 18). I anticipate if/when KotOR3 gets made, there is a good likelihood it will use this field type as well.

@Adelphus: I don't believe cchargin or Taina have devoted time to modifying their modeling tooling for Jade Empire at this point.

Nice work on KGFF tk, and thanks for the info.

I've uploaded another minor update to the ERF/RIM Editor, adding some more file associations to the table. See the link in the first post in this thread to download it.

stoffe, just a couple questions/comments about your TXB2TGA converter...

Not that it doesn't work properly or anything--and I can use it just as it is--but, I was curious as to why I get so much dithering when converting the TXBs to TGAs.

Also, is there any way for the converter to increase the output file size from 256x256 to the more commonly used 512x512? It just seems as if the smaller file size could be causing the dithering, but I could be completely wrong about that and the dithering is caused by something else entirely.

Anyway, thank you so much for this converter! Whether or not you can do anything else with it, it still works fine just as it is and I am enjoying using it a lot! ;)