After you set the new resolution, did you hit apply?
Otherwise you can always try setting it in console with r_mode and then doing a vid_restart
These are what I believe each r_mode # corrisponds to in actual resolution.
Code:
r_mode "0" 320x240
r_mode "1" 400x300
r_mode "2" 512x386
r_mode "3" 640x480
r_mode "4" 800x600
r_mode "6" 1024x768
r_mode "7" 1152x864
r_mode "8" 1280x1024
r_mode "9" 1600x1200
You could also set r_mode -1 and then use r_customewidth and r_customheight to set your resolution, followed by a vid_restart to apply it.