Improved Rate Distortion Optimized Quantization for HEVC with Adaptive Thresholding

Coefficient-level rate distortion optimized quantization (RDOQ) is an efficient tool to improve rate-distortion performance with 6%-8% bit-rate saving. It has been widely adopted in video encoders such as JM, x264, HM and so on. However, software implementation of RDOQ suffers from high computation complexity due to intensive path search and from data dependency caused by context based entropy coding. In this paper, a fast RDOQ algorithm with accurate rate estimation between the two candidates quantized level for high efficient video coding (HEVC) is proposed to decrease the computation complexity and break the data dependency. Firstly, the preselection of candidate quantized levels is preformed according to hard decision quantization (HDQ) level and the coefficient position in transform unit (TU). The reduction of candidate quantized levels can significant decrease the computation complexity of RDOQ. Secondly, a model of delta bit-rate is constructed to estimate bit-rate change between the rest two candidate quantized levels. With this delta bit-rate model, the high computation complexity of bit-rate computation based on context adaptive binary arithmetic coding (CABAC) can be avoided, and coefficients can be parallel processed. Experiment results demonstrate that the proposed algorithm can decrease the encoding time of RDOQ by 54.36% in average with no more than 0.88% BD-rate loss.