Feature-Space Optimization-Inspired and Multi-Hypothesis Cross-Attention Reconstruction Neural Network for Video Compressive Sensing

The existing video compressive sensing reconstruction network usually uses the optical flow network to achieve pixel domain motion estimation and motion compensation. However, during the reconstruction process, the input of the optical flow network is the estimated frame with poor quality, resulting in inaccurate optical flow. The optical flow-based pixel domain alignment and fusion operation will cause noise accumulation, lead to obvious artificial effects in video reconstruction frames and affect the reconstruction quality. Based on the fact that multichannel information in the feature space has strong robustness to interference noise, this paper applied the idea of feature space optimization to the design of the video compressive sensing reconstruction neural network, and proposed a feature-space optimization-inspired and flow-guided multi-hypothesis cross-attention network (FOFMCNet). To avoid the image structure destruction caused by the noise in the optical flow when warping the image, the study designed multi-hypothesis motion estimation module guided by optical flow and the motion compensation module based on cross-attention to realize the motion estimation and motion compensation of inter-frame in feature space, so as to make full use of inter-frame correlation to assist non-key frame reconstruction. In order to strengthen the reuse of effective information in the process of feature optimization, improve the learning ability of the network and alleviate the problem of gradient explosion, this paper designed a feature-space optimization-inspired u-shape network (FOUNet) as a sub-network of FOFMCNet. Through the cascade of multiple FOUNets, the FOFMCNet realizes the optimization and reconstruction of non-key frames in the feature space. Experimental results show that the reconstruction results of the proposed algorithm are obviously better than those of the existing video compression sensing algorithms on the classical low-resolution dataset (UCF-101 and QCIF) and new high-resolution dataset (REDS4). © 2024 South China University of Technology. All rights reserved.