An efficient fast mode decision algorithm for H.264/AVC intra prediction

Rate distortion optimization (RDO) based spatial intra prediction is an important new feature in H.264/AVC. It chooses the best mode for each MB by exhaustively examining all possible modes. Thus, drastically increased computation complexity is involved. This paper proposes an efficient fast intra mode decision algorithm. Prewitt operator is used to extract each block's two most dominant edge directions. Based on these two directions, only few modes are chosen as candidates. In this way, computational load caused by RDO can be reduced remarkably. Experimental results show that more than 70% computation reduction can be achieved, with little loss of visual quality. Moreover, only addition and comparison operations are used, which makes proposed algorithm friendly enough to be implemented in hardware.