The VegaStrike engine is open source and pretty nice.
http://vegastrike.sourceforge.net/
Then again, Beyond the Red Line is built on a modified Freespace 2 engine, and it looks pretty nifty.
http://www.beyondtheredline.net/
That being said, while I'd enjoy a mega-pretty X-Wing title built on open source, I'd be quite happy with an open source engine that can support the existing bevy of X-Wing Alliance models. You can always replace those later with nicer models and nicer textures, but I assume you'll be at a playable state quicker if you can use all the existing models the fan community has made.