The player uses one of LEC's own dll's to do the decoding, if you wanted to make a proper smush player you'd be much better using the scummvm source I think.
You can get a decent capture I think, I know I managed it when I did the
GF Trailer. I *think* I used hypercam and I *think* I used its recommended codec (as mentioned in the help file) and then converted the video later. You need to try and use a codec that captures quickly without being as hefty as raw avi.
I dont know if Grim and Residual would play the videos if they arent compressed, or if it would make much of a difference, you could always give it a go. My guess is that its the capture program/codec thats the bottleneck, not the game.