Artifact reduction of compressed video via three-dimensional adaptive estimation of transform coefficients