Lip2Speech: Lightweight Multi-Speaker Speech Reconstruction with Gabor Features

In environments characterised by noise or the absence of audio signals, visual cues,notably facial and lip movements, serve as valuable substitutes for missing or corrupted speechsignals. In these scenarios, speech reconstruction can potentially generate speech from visual data.Recent advancements in this domain have predominantly relied on end-to-end deep learning models,like Convolutional Neural Networks (CNN) or Generative Adversarial Networks (GAN). However,these models are encumbered by their intricate and opaque architectures, coupled with their lackof speaker independence. Consequently, achieving multi-speaker speech reconstruction withoutsupplementary information is challenging. This research introduces an innovative Gabor-basedspeech reconstruction system tailored for lightweight and efficient multi-speaker speech restoration.Using our Gabor feature extraction technique, we propose two novel models: GaborCNN2Speechand GaborFea2Speech. These models employ a rapid Gabor feature extraction method to derive lowdimensionalmouth region features, encompassing filtered Gabor mouth images and low-dimensionalGabor features as visual inputs. An encoded spectrogram serves as the audio target, and a LongShort-Term Memory (LSTM)-based model is harnessed to generate coherent speech output. Throughcomprehensive experiments conducted on the GRID corpus, our proposed Gabor-based modelshave showcased superior performance in sentence and vocabulary reconstruction when compared totraditional end-to-end CNN models. These models stand out for their lightweight design and rapidprocessing capabilities. Notably, the GaborFea2Speech model presented in this study achieves robustmulti-speaker speech reconstruction without necessitating supplementary information, therebymarking a significant milestone in the field of speech reconstruction.