A Novel Fast Inter-mode Decision Algorithm for H.264/AVC Based on Motion Estimation Residual

This document proposes a fast inter-mode decision method based on the residual of motion estimation for H.264/AVC. There are 7 different MB modes in inter-mode coding with a SKIP mode, as well as 9 modes of 4X4, 8X8 and 4 modes of 16X16 in infra-mode coding for P-slices coding, which brings drastic computational complexity to the encoder. To solve this problem, we suggest a novel fast inter-mode decision method using the variance and SAD of residual block. Our method computes the motion estimation residual block's variance and sum of absolute value (i.e. SAD) to decide the best mode of current MB, and computes residual sub-block's sum of absolute value to decide sub-MB's mode. Experimental results show that our method can save 30%-50% encoding time on PC platform without significant PSNR degradation or bitrate increasing.