Mono waveform should be fine. I've put regular WAV files into Override and played them through ammunitiontypes.2da. I've never tried adding a sound effect (I'm assuming you're trying it with PlaySound) because I doubt it can be done easily (well, maybe not doubt so much as I didn't want to bother trying it at the time). So basically I can't help...

except to say I wouldn't waste time messing with the format when so many other things could go wrong; try replacing a sound effect and see if it plays fine, then you'll know if it's the format.