ULME-GAN: a generative adversarial network for micro-expression sequence generation

Currently, the lack of micro-expression datasets is a significant obstacle to micro-expression research and hinders the development of micro-expression supervised data generation. To address this issue, we propose the unsupervised learning micro-expression sequences generative adversarial network (ULME-GAN) approach, which generates micro-expression sequences that can be controlled. By analyzing all action units (AUs) that appear in main micro-expression datasets, a novel method called action unit matrix and re-encoding (AUMR) is proposed to generate micro-expression sequences that appear more natural and seamless by smoothing the AU matrix extracted from the source video. Our experiments demonstrate that the ULME-GAN approach can generate micro-expression videos/images that maintain the input source video/image pattern better than other methods, such as the first order motion model and StyleGAN. Furthermore, the micro-expression recognition task demonstrates that the augmented dataset can lead to a significant improvement in the performance of micro-expression recognition models. Finally, ULME-GAN can generate videos/images with specific micro-expression patterns defined by an input AU matrix, making it suitable for various applications even when there is insufficient source video.