Which OPT files are you running? If you're trying to use the straight XWA Upgrade Project OPTs, they won't really work b/c they're meant for X-Wing Alliance, a later game. Some of the XWAUP OPTs have been converted down for XvT/BoP, however...look for the XvT Upgrade on
the "Astilleros" site (English translation available on the site).
Other than that, try uninstalling, do a clean install, then enable 3D acceleration and play without mods. If you find what you described still happens, I'd be willing to wager that it's a driver problem.