MPEG-4 facial animation in video analysis and synthesis

MPEG-4 supports the definition, encoding, transmission, and animation of 3-D head and body models. These features can be used for a variety of different applications ranging from low bit-rate video coding to character and avatar animation. In this article, an entire system for the analysis of facial expressions from image sequences and their synthesis is presented. New methods for the estimation of MPEG-4 facial animation parameters as well as scene illumination are proposed. Experiments for different applications demonstrate the potential of using facial animation techniques in video analysis and synthesis. A model-based codec is presented that is able to encode head-and-shoulder video sequences at bit-rates of about 1 kbit/s. Besides the low bit-rate, many enhancements and scene modifications can be easily applied, like scene lighting changes or cloning of expressions for character animation. But also for the encoding of arbitrary sequences, 3-D knowledge can help to increase the coding efficiency. With our model-aided codec, bit-rate reductions of up to 45% at the same quality can be achieved in comparison to standard hybrid video codecs. (C) 2004 Wiley Periodicals, Inc.