Rate-SSIM optimization for video coding