I have found a solution that seems to work for me. Ymmv.
1. Make sure hdmi_force_hotplug=1 is
commented in /boot/config.txt
The following doesn't seem to have any effect is hdmi_force_hotplug=1 is set. The vnc resolution is always 1184x624 if it's set.
2. Use fbset prior to starting X to change the resolution via the geometry (-g) command.
*NOTE: This must be done before X is started for the first time. Even if you shut down X, run fbset and the restart X, it does not seem to have any effect. A reboot is required to change the resolution after X has been started for the first time. I'm not sure why.
ie.
for 1280x720, 16 bit color:
for 1920x1080 (1080p), 16 bit color:
3. Start X
Alternatively, if you have X set to start at boot, you'll need to insert a script into init.d containing the fbset command and create a symlink in your default runlevel rc.d folder (probably rc2.d) starting with S with a lower number than the script that starts X so it runs first. I personally have X set to disable at boot and start it manually via ssh, so I haven't tried implementing this. The most proper way would probably be to create the init script and use update-rc.d to create the symlinks, there are guides online on how to accomplish that.
Hope this helps. Best of luck.