You'd actually be better of using Residual
. The documentation on the website is perfectly clear, but basically you simply have to install Residual, create a folder in a convenient place on your hard drive to copy every file on the CDs ending with .lab plus the grim.tab file. You also would need to download to that same directory the Grim Fandango patch (available on the Residual website). You don't have to install it. You don't need to. Just put the patch executable in the same folder with the files copied from the CD. Then you run Residual and add the game. The interface is similar to ScummVM, so if you've used ScummVM, you've already got the basics of Residual. But you should read the documentation on the Residual website anyway.
There are other settings, like built-in antialiasing, but I'd focus on just the basic setup before bothering with tweaks.
Residual is still at an early stage of development, but it's gotten far enough that the game is completely playable with the only major bugs being graphical glitches.
As for what you've already installed, you could try fiddling with compatibility settings, but I think it's simpler to go with Residual. It's also already better than the original in one big way in that Residual fixes a major bug (that stops an important piece of dialog from ever happening) that even Tim Schafer didn't know existed until Residual found and fixed it.