Morph Targets for 3D Facial Animation with Webcam Using Facemesh, Jeeliz-transfer APIs and Three.js']js

This paper presents a face control system for 3D avatar with webcam using the Facemesh API for face tracking and Jeeliz-transfer API for eye tracking. 3D avatar face animation is developed as a responsive web application. It starts with face detection and tracking through the webcam. Face coordinate data is normalized to a vertical face view where the distance between the eyes and the level of the eyes are the same for each video frame. Then new face coordinates are calculated in both 2D and 3D to study the change of specific coordinates such as mouth shape and face shape. In addition, the coordinates of face structures in the video platform have also been added to the 3D model platform. Specific coordinates are studied to analyze distance changes to be applied in 3D avatar manipulation. The 3D models are designed and created in multiple blend-shapes or basic character facial features. However, due to the limitations of web browser-based 3D morphing, which Three.js is used for morph target displays, it allows only eight combination shapes to be displayed at the same time. Therefore, the required blending geometry must be pre-assembled. Blend-shape factors are based on an analysis of the coordinates of each moving face in order to eliminate the limitations. Our 3D facial animations with a webcam generate results of high quality, real-time and online simulation. Therefore, our work is a fundamental technology that can be applied to animate other 3D characters in blend-shape format.