Adaptive boundary dependent transform optimization for HEVC