EnNeRFACE: improving the generalization of face reenactment with adaptive ensemble neural radiance fields

Face reenactment is a critical technology of digital face editing. Lately, the NeRFACE, a face reenactment method based on neural radiance fields, has been proposed, making the reconstruction accuracy of the training dataset much higher than the previous methods. However, face reenactment in realistic scenes often encounters poses and expressions that have not been seen before, which requires further improvement of the model’s generalization capability. Based on the idea of ensemble learning, we present EnNeRFACE as using the adaptive ensemble neural radiance fields architecture, which is mainly composed of a set of subgenerators and a controller. We divide the short video of human portraits into non-intersecting sub-datasets based on time correlation, thus enabling the trained subgenerators to have differentiated modeling capabilities. In response to different expression vectors, the generator dynamically adjusts the weights assigned to each generator so that the capabilities of the subgenerators are adequately exploited. Extensive experiments show that EnNeRFACE has more stable and superior performance in generalization (i.e., identity preservation, manipulation of expression and pose) than the state-of-the-art methods, demonstrating the effectiveness of our proposed adaptive ensemble structure.