Rate-distortion modeling for efficient H.264/AVC encoding

The rate-distortion (R-D) optimization technique plays an important role in optimizing video encoders. Modeling the rate and distortion functions accurately in acceptable complexity helps to make the optimization more practical. In this paper, we propose a bit-rate estimation function and a distortion measure by modeling the transform coefficients with spatial-domain variance. Furthermore, with quantization-based thresholding to determine the number, the absolute sum, and the squared sum of transform coefficients, the simplified transform-domain R-D measurement is introduced. The proposed algorithms can reduce the computation complexity for the R-D optimized mode-decision by using the new cost function evolving from the simplified transform-domain R-D model. Based on the proposed estimations, a rate-control scheme in the macroblock layer is also proposed to improve the coding efficiency.