Which format are you using? Ogg, FLAC or mp3?
And how did you create the mappings? All manually or did you use the information from speech.info to generate a basic layout?
It would be good to know how many exceptions there are. I'm all for an automated process which would ease redistribution. It's not a real problem to join multiple wav files for example given some previous definition (or even a heuristic that matches multiple lines).