Real-time MPEG-4 facial animation with 3D scalable meshes

In this paper the complete framework of a software package for real-time animation of 3D facial models according to the MPEG-4 specifications is described. The focus is mainly on the algorithmic solution that has been adopted for scaling the polygon mesh to a level of complexity suited to the specific hw/sw profile of the platform that hosts the decoder. The basic functionality of this system is its ability to deform virtual faces in response to MPEG-4 Facial Animation Parameters (FAP) according to "animation rules" whose computation is based on very simple "semantic" information previously associated with the model. The software architecture is modular and each module, implementing a specific functionality, can be easily plugged in or unplugged. In this paper, we begin with a general presentation of the system and then we present a brief description of its main modules: the face model, the mesh animation, the mesh calibration, the mesh simplification and the FAP decoding. While for the other modules a deeper analysis is left to referenced bibliography previously published by the authors, a comprehensive description is given here of the mesh simplification block together with experimental evidence of its efficiency. The algorithm we have adopted for mesh simplification is based on iterative edge contraction with the introduction of specific geometry constraints due to the particular nature of the 3D mesh representing an MPEG-4 animated face. In this paper it will be shown how rather complex 3D facial models, optionally textured, can be simplified in real-time in order to adapt the rendering performance to the hardware capabilities, whilst still maintaining compliance with the MPEG-4 specifications on facial animation. (C) 2002 Elsevier Science B.V. All rights reserved.