Rate-distortion optimized transform for intra-frame coding