Multi-stage temporal representation learning via global and local perspectives for real-time speech enhancement

Deep learning-based speech enhancement algorithms have been rapidly developed over the past few years. Although numerous approaches have been proposed, global and local information from speech features have not been thoroughly investigated. In this paper, we introduce a novel and highly effective speech enhancement network called Multi-stage Global-Local Network (MSGLN), which exploits both local and global information via temporal self-attention, temporal graph convolution, and 1D convolution. Local modeling blocks capture the fast changes in speech signals, while global modeling blocks learn long-term trends in noise or speech signals through factors such as pitch, tone, resonance, timbre, and rhythm. In addition, we propose a multi-stage temporal processing module as the bottleneck of a complex convolutional encoder-decoder structure to guide our network to learn different acoustic structures from different scales. Then a dual-path RNN postprocessing module is integrated to reconstruct the speech spectrum mask using a frequency-wise temporal refinement block followed by a frame-wise spectral refinement block. Experimental results demonstrate the superior performance of our proposed methodology compared to other state-of-the-arts on both real-time single- and multi-channel speech enhancement tasks.