Monkey Island Special Edition - File Formats

[Murray the Chao]

Any updates on the music?

[Guy.brush]

Regarding the Nvidia Photoshop DDS plugin: Alpha works fine. you need to save as DXT5 with interpolated alpha (or whatever the fancy alpha variant is called)
also you need to generate an alpha channel, but for experienced Photoshop users this is easily done.

[Farlander1991]

Hi guys. Great work, very useful stuff!

Ditto to ronzbig1. Does anyone know how to change subtitles without making voices to disappear?

And, by the way, when changing speech.info, though Guybrush's phrases do change, the same phrase in the selection menu during dialog doesn't.

[jott]

Here are my latest findings:

typedef struct CostumeHeader {
  unsigned int identifier; // number corresponds to filename
  unsigned int relativePositionToObjectName; 
  
  unsigned int numberOfTextures; // Number of dxt files referred
  unsigned int realtiveOffsetToTextureIndex; // relative to current position
  
  unsigned int numberOfStates; // Like InitLeft or WalkLeft
  unsigned int relativeOffsetToStateIndex; 
  
  unsigned int unk1;  // 0 most of the time, sometimes 1,2,3
  
  unsigned int numberOfSpriteGroups; // 1 most of the time, in the rage of 1-11
  unsigned int relativeOffsetToSpriteGroups;
  
  unsigned int unk2; // 0 most of the time, sometimes 1,2,3,4
  
  unsigned int numberOfPointsOnPath; // some sort of 2D positions
  unsigned int relativeOffsetToPath;
  
  unsigned int num3_1; // 0 most of the time, sometimes 1
  unsigned int num3_2; // always 0

  float version[3]; // always 0.5 1.0 1.0 
  unsigned int reserved[3]; // always 0 0 0
} CostumeHeader;

typedef struct TextureIndex {
  unsigned int identifier;
  unsigned int relativeOffsetToFilename;
} TextureIndex;

typedef struct StateIndex {
  unsigned int relativeOffsetToStateName;
  unsigned int identifier;
  unsigned int numberOfStateData;
  unsigned int relativeOffsetToStateData;
} StateIndex;

typedef struct SpriteGroup {
  unsigned int identifier;
  unsigned int unk1;
  unsigned int numberOfSprites;
  unsigned int relativeOffsetToSprite;
} SpriteGroup;

typedef struct PathPoint {
   char unk1;
   char unk2;
   char unk3;
   char unk4;
   float x;
   float y;
} PathPoint;   

typedef struct Sprite {
   unsigned int textureNumber;
   unsigned int textureposx;
   unsigned int textureposy;
   unsigned int width;
   unsigned int height;
   float screenposx;
   float screenposy;
   unsigned int unk2;
   unsigned int unk3;
   unsigned int unk4;
} Sprite;


typedef struct StateData {
   unsigned int spriteGroup;
   unsigned int sizeOfFrames; // ???
   unsigned int numberOfFrames;
   unsigned int relativeOffsetToFrames;
} StateData;

typedef struct Frame {
   unsigned int spriteIndex;
   unsigned int unk1;
   unsigned int unk2;
} Frame;

I think it is nearly complete now. Some work has to be done for the StateData and Frames but I guess some bruteforcing could bring some more insights. The other "XML" files should have a similar structure.

I guess the main question is on how to proceed...
One way could be to transform the structure to a real XML file so it can be modified with a text editor - the other would probably be some kind of visual editor.... well I don't have *that* much time unfortunately..... any help is needed!

[Guy.brush]

So the work to "decode" the content into a real .XML that's readable by e.g. Dreamweaver is not the problem? The problem is to get the edited .XML back into the format used inside the .pak?
hmm...

[bgbennyboy]

I've only been away for a day and the progress has been rapid. Good job everyone!

I'm going to look into the WMA's now. Hopefully it'll just be a case of adding the correct header to them.

Another possibility is that they might be headerless xWMA

[bgbennyboy]

Giving up for now. I've been trying to manually build a wma file using the raw data, just to test that the data really is wma.

Unfortunately this hasn't yielded any results, I haven't been able to build a correct wma header that will play back any audio. I've been assuming that the audio is 44100 stereo 16bit wma and after failing with calculating the track length from the data, I tried using the track length from the corresponding wav files in the old music soundbank.

[Benjy]

Hi everyone,

Wow, it's been years since I had come here. Anyway, this might seem to be a dumb question, but what is the engine behind MI: SE? A newer SCUMM?

[bgbennyboy]

Having another crack at the audio.
I'm now assuming that its headerless xWMA. I'm currently trying to use xWMA encode to produce a valid xWMA file from a wav, then chopping out the data, correcting the wav size headers, inserting the data from one of the MI WMA's and seeing if xWMA will decode it back to pcm wav. The trick is in finding the correct bitrate for the input and output audio.

[jott]

Yes I can confirm that this have to be xWMA files, given the same 20 byte pattern in the beginning occurs when converting a WAV with xWMAEncode.exe with -b 96000.

Now we only need to create the proper header....

[Tomas]

I can confirm this too. I'm actually able to play the files now!

Just took a long wavefile, converted it xwma at 96 kHz using xwmaencode. Simply copied the header and pasted first in one of the extracted files from the wavebanks then converted it back to .WAV using xwmaencode. It gives an error because the header size is off but it still converts the file and I can play it successfully. Generating a correct header should be no problem!

EDIT: So, for the record, the WAVEFORMATEX chunk I ended up with for the working file was:

66 6D 74 20 12 00 00 00 61 01 02 00 44 AC 00 00 E0 2E 00 00 6B 11 10 00 00 00

So the problem boils down to generating the dpds (decoded packet cumulative data size array). Hopefully there is some obvious way to generate it from the actual data somehow.

[bgbennyboy]

Excellent! 96khz it is!
For those who cant do this themselves here is a track ripped in this way (converted to mp3 to save space).

[bgbennyboy]

We can easily put together a batch file to rip and do the conversion. However, a more optimal solution would be to get a correct WMA header to add to the raw xWMA's. As I understand it xWMA's are pretty much the same as a normal WMA and should work as part of a WMA container.

I'm going to keep trying to do this, if I can get a container then I can easily make an app to read the soundbanks and put the data in a RIFF/WMA container. Its a shame that the WMA format is such a pain in the arse¬

[jott]

Here is a modified (i.e. hacked together) version of unxwb that produces "xwm" files that can be converted with the xWMAEncode.exe tool back to WAV (part of the DirectX SDK).

Source and binary (updated):
http://helicoid.de/scumm/unxwb-mi.zip

So for example run

unxwb.exe MusicNew.xwb

and then something like

for /r . %i IN (*.xwm) do @xWMAEncode.exe %i %~ni.wav

to convert all the files to wav. (Given both tools are in the PATH or in the current directory...)

If you have LAME installed you could finally do

for /r . %i IN (*.wav) do @lame.exe --preset extreme %i %~ni.mp3

to produce mp3 files.

[michallo_w]

So... Can you upload it?

[jott]

Originally Posted by michallo_w

So... Can you upload it?

Upload what? I most certainly will not upload the soundtrack as everyone owning MISE can extract the files.
As for the xWMAEncode.exe - Microsoft is not very friendly regarding redistribution of their tools. A real shame. So you have to get the whole DX SDK (you could extract the EXE file if you wish instead of installing it though).

[daltysmilth]

So what about extracting the Tales Of Monkey Island music?

[bgbennyboy]

xWMAEncode can be found here.

daltysmith:
I'll have Telltale Music Extractor updated to support TOMI Music soon.

[rumperdumper]

Originally Posted by jott

Here is a modified (i.e. hacked together) version of unxwb that produces "xwm" files that can be converted with the xWMAEncode.exe tool back to WAV (part of the DirectX SDK).

Source and binary (updated):
http://helicoid.de/scumm/unxwb-mi.zip

So for example run

unxwb.exe MusicOriginal.xwb

and then something like

for /r . %i IN (*.xwm) do @xWMAEncode.exe %i %~ni.wav

to convert all the files to wav. (Given both tools are in the PATH or in the current directory...)

If you have LAME installed you could finally do

for /r . %i IN (*.wav) do @lame.exe --preset extreme %i %~ni.mp3

to produce mp3 files.

works just fine for me this way... thanks ^^

now the only thing i need are the right names for the right file... would be nice if anybody could make a list or so... i'm just too dump to name them propably... until now i'm only pretty sure that 00000016 is actually the theme music... could someone check that please

and thanks again for mentaning how it may could be possible to get the new music playable

[deadca7]

Just had a listen to them all, these are the ones I know without playing the game and also the BEST songs !

00000008 = SCUMM Bar
00000009 = Ghost ship
00000012 = Stan's Previously Used Ships

[jott]

I just modified the tool to use the "description" field as the file name.

Same URL:
http://helicoid.de/scumm/unxwb-mi.zip

This way the music score is still not named very descriptive ("track18b" is the theme music for example, oh and you can guess what "silence" is) but the speech and sound effect names are MUCH better this way.

So the list should probably be made for the "trackNN" names.

EDIT: Actually the track order is basically the same as the original (Enhanced CD) version, so there should be a list out there already ;-)

[spiralout]

Thanks for the instructions. Worked like a charm. But remember to do MusicNew.xwb instead of MusicOriginal.xwb for the new music. (Why is the file so much smaller?)

[jott]

Originally Posted by spiralout

Why is the file so much smaller?

Because the "Original" contains some uncompressed wav files.
On top of this, it contains some ambient sounds which are stored in an extra container with the new version.

(Edited my example above using the "New" version to prevent some confusions.)

Can you get this in-game somehow: Guybrush_UnknownFilename_07?

[LogicDeLuxe]

Originally Posted by jott

Can you get this in-game somehow: Guybrush_UnknownFilename_07?

There are quite a lot samples which likely aren't used in the game. The keelhaul-description is in there, too (which is a dialog option, but Guybrush refuses to say this). And a whole bunch of different prices for Monkey Island, including XBOX points, while the game only seems to use the original "59.95 + tax" one. Many lines include alternative takes, which the game seems to ignore as well.
I guess, any sample not referred in speech.info is not used.

[The Sporkman]

Can someone create a simple, step-by-step, dumbed-down explanation of how to extract the music files for someone who hasn't the slightest idea how to use the command prompt?

Edit: I've manged to extract all the .xwm files with unxwb.exe, I but I still can't figure out to convert them to waves or mp3s with xWMAencode.exe. Sorry to be such a pain... I'm pretty incompetent at this sort of thing...

Edit again: Never mind. I figured everything out now.

[Farlander1991]

Hey again, guys.

I've managed to edit speech.info correctly, without removing the voice overs, using HEX editor (don't know why it didn't work earlier). But, speech.info doesn't edit any text in the original SMI version OR the dialouge choices in the new version (I mean, like, if we change Guybrush's line, the choice line will still be the same, though Guybrush will say other things). Editing original text files kind of helped, but voice overs disappear for any edited in the original monkey game line. Does anybody know how to fix this?

[bgbennyboy]

Original dialog text is inside the script files in the old resource files (monkey.001).

[Farlander1991]

Yes, I already said that I found that out. The question is not "where", but "how to change the old resource files without making the character voice overs to disappear in the new version".

[Jeroen_JRP]

While bg is working on a more optimal solution I made a batch file for the audio extraction/converting process.

It will extract every .xwb file into subdirectories and then convert every .xwm to .wav wich ends up with the folowing directory structure:

--Main (where batchfile and xwb files are)
   --MusicNew (example)
      --Original (containing xwm and wav files extracted from xwb)
      --Converted_WAV (containing the wav files converted from the xwm files)

All you need to do to get is running is placing the unxwb.exe and xWMAEncode.exe with the batch file (miseAudioProcessing.cmd),
Then placing the xwb files you want to extract in the same directory and then run the batch file.

You can download the batch file here: http://hulauncher.jrpweb.nl/files/mi...Processing.cmd

[Burarum]

hello all! I've watched this thread intensively in these past days, and I must say I'm astonished by how well and quick you guys work :-) I just have this confession: I love the monkey island music, and even more the special edition music. I have great respect for people who know exactly how to rip and extract music from different types of files but, alas, I have no knowledge of how to do so WHATSOEVER. So I would apprieciate (and bring you monkey luv) if some kind sould might take the time to write a down to earth, step-by-step instruction of how to extract the music. I would really apprieciate this

Thanks in advance, and good job everyone!

[Jeroen_JRP]

Originally Posted by Burarum

So I would apprieciate (and bring you monkey luv) if some kind sould might take the time to write a down to earth, step-by-step instruction of how to extract the music. I would really apprieciate this

Thanks in advance, and good job everyone!

1) First you will need the folowing files:
unwxb.exe from jott (you only need the executable file found in the .zip archive)

Originally Posted by jott

http://helicoid.de/scumm/unxwb-mi.zip

xWMAEncode.exe (you only need the executable file found in the .zip archive)

Originally Posted by bgbennyboy

xWMAEncode can be found here.

And my batch program to make it easier
http://hulauncher.jrpweb.nl/files/mi...Processing.cmd or http://hulauncher.jrpweb.nl/files/mi...Processing.zip

2) Place these files together in the same folder somewhere on your computer.
3) Assuming you want the new music, copy the MusicNew.xwb file from the audio folder of Monkey Island: SE (default: C:\Program Files\Steam\steamapps\common\the secret of monkey island special edition\audio) to the folder where you placed the downloaded files in step 2
4) Run (double-click) on miseAudioProcessing.cmd and wait till it says "Done"
5) The folder from step 2 now contains a subfolder named "MusicNew", open that folder.
6) You will find a subfolder there named "Converted_WAV", open that folder.
7) Here is all the music in .wav format, ready to be played!

[jott]

Originally Posted by Farlander1991

Yes, I already said that I found that out. The question is not "where", but "how to change the old resource files without making the character voice overs to disappear in the new version".

My (unconfirmed) guess is, that the first int fields in the speech.info are some sort of CRC/hash that are used as a lookup for the original strings.

@John_Doe:
I played around a bit with the CRC (the one from Monkey1.pak to test it) but I can't get the right value. Could you share on how you can compute the CRC exactly? (I tried to & 0xDF the strings but this does not give the right solution with a CRC32).

[spencersage]

Okay Jeroen_JRP, I've almost got this figured out... but for some reason Windows downloaded your batch program as a text file... .cmd.txt and I can't get it to recognize it or run it any other way. What do I do to get it to run it as a CMD? Sorry for my uselessness.

[Krasas]

Originally Posted by jott

My (unconfirmed) guess is, that the first int fields in the speech.info are some sort of CRC/hash that are used as a lookup for the original strings.

From what I could gather, the header before each "group of cues" entry in the speech.info file has the following fields.

• 4 bytes that are some kind of CRC or otherwise uniquely indexing a spoken line (in english). This means that the cues: "Guybrush Threepwood.", "Guybrush THREEPWOOD", "Guybrush Threepwood!" will all have the same value for this field, even though they are spoken from different characters, and have differences in pronounciation. I really have trouble understanding the reason for this field, but I bet that this is what causes problems with fansubs.
• 2 bytes that index the room where the cue is spoken
• 2 bytes that are some kind of index for the "interaction" where the cue is spoken (e.g. same value if guybrush interacts with the same object, same value throughout one conversation)
• 2 bytes that somehow index the cue within the conversation or the room -I am not sure.
• 2 bytes that incementally index a "subcue" (if any) within a single cue. Subcues are seperated by the escape sequence \255\003 in the original text. (value 00 for the first subcue, 01 for the second etc.)
• 2 bytes: that have the value 01 if the line is spoken by Guybrush, and 00 else.
• 2 bytes: of unknown purpose. They almost always seem to be 00, but there are cases where they have other value.

[elTee]

Originally Posted by spencersage

Okay Jeroen_JRP, I've almost got this figured out... but for some reason Windows downloaded your batch program as a text file... .cmd.txt and I can't get it to recognize it or run it any other way. What do I do to get it to run it as a CMD? Sorry for my uselessness.

If you click on 'tools' in any windows explorer folder, then 'folder options' there should be an option to hide/reveal filename extensions. If you make them visible you can just remove the '.txt' from the end of the filename and it'll work fine.

[Jeroen_JRP]

Originally Posted by elTee

If you click on 'tools' in any windows explorer folder, then 'folder options' there should be an option to hide/reveal filename extensions. If you make them visible you can just remove the '.txt' from the end of the filename and it'll work fine.

Exactly, and in case that doesn't work I uploaded a .zip containing the .cmd file:
http://hulauncher.jrpweb.nl/files/mi...Processing.zip

[koun]

I have created a batch file to rename the tracks of the new Soundtrack:

ren "track18b.mp3" "01. Opening Themes.mp3"
ren "track18c.mp3" "02. Introduction.mp3"
ren "track7.mp3" "03. Chapter Screen.mp3"
ren "track9.mp3" "04. The Scumm Bar.mp3"
ren "track13.mp3" "05. LeChuck's Theme.mp3"
ren "track3.mp3" "06. Following The Shop Keeper.mp3"
ren "track10.mp3" "07. Ghost Ship Shuffle.mp3"
ren "track6.mp3" "08. Meleé Island.mp3"
ren "track2.mp3" "09. The Voodoo Shop.mp3"
ren "track8.mp3" "10. Guybrush and Elaine.mp3"
ren "track4.mp3" "11. Meleé Forest.mp3"
ren "track19.mp3" "12. Stan's Previously Used Ships.mp3"
ren "track12.mp3" "13. The Fettucini Brothers.mp3"
ren "track16.mp3" "14. The Journey.mp3"
ren "track15.mp3" "15. Monkey Island.mp3"
ren "track22.mp3" "16. The Cannibal Village.mp3"
ren "track14.mp3" "17. LeChuck's Theme (Alternate).mp3"
ren "track5.mp3" "18. Organ Prelude.mp3"
ren "track17.mp3" "19. Closing Themes.mp3"
ren "track18a.mp3" "20. Theme (alternative version).mp3"
ren "track20.mp3" "21. Unknown.mp3"
ren "track10a.mp3" "22. Ghost Ship Shuffle (alternative version).mp3"

It is based on the following soundtrack: http://soundtracks.mixnmojo.com/mi1.htm
Feel free to post corrections of any kind

[ramy]

Originally Posted by Jeroen_JRP

2) Place these files together in the same folder somewhere on your computer.

For the script to work, be sure the folder that you place these in from the c:\ don't have any spaces in the folder name. For example, having your folder in C:\Documents and Settings\etc...won't work.

I was having this problem and it took me a while to figure out why I couldn't get the script to work.

[Jeroen_JRP]

Originally Posted by ramy

For the script to work, be sure the folder that you place these in from the c:\ don't have any spaces in the folder name. For example, having your folder in C:\Documents and Settings\etc...won't work.

I was having this problem and it took me a while to figure out why I couldn't get the script to work.

Ow, I'm sorry about that, forgot to check it. Didn't have any trouble myself becouse Windows Vista and 7 use far less spaces in their folders.
Thanks for the heads up!

[Rum Rogers]

Originally Posted by Farlander1991

Hey again, guys.

I've managed to edit speech.info correctly, without removing the voice overs, using HEX editor (don't know why it didn't work earlier). But, speech.info doesn't edit any text in the original SMI version OR the dialouge choices in the new version (I mean, like, if we change Guybrush's line, the choice line will still be the same, though Guybrush will say other things). Editing original text files kind of helped, but voice overs disappear for any edited in the original monkey game line. Does anybody know how to fix this?

Hi everybody!
How'd you do that? I tried editing it with HEX editor, but the game crashes as soon as the line I edit gets said by the character.