Real-time video super-resolution via motion convolution kernel estimation

The goal of video super-resolution (SR) is to generate a high-resolution (HR) video frame from multiple consecutive low-resolution (LR) frames. This task is challenging because it considers not only the spatial relationship but also the temporal one. Recently, lots of video SR methods emerged, especially the ones using convolutional neural networks (CNN). Most of these methods align reference frames with the predicted motion and send LR current and reference frames to a CNN to learn a mapping function to generate a corresponding HR frame. However, the performance of these methods is limited by inaccurately predicted motion and inappropriate CNN architecture. Therefore, we propose a novel real-time video SR method by addressing the above-mentioned problems from two aspects. First, we align each reference frame with a pair of 1D motion convolution kernels, which are predicted from our motion convolution kernel estimation network. Second, we improve the gated residual unit (GEU), which can combine input and output signals with trainable weights, and integrate GEU to our network architecture. Our proposed method is verified by benchmark datasets both qualitatively and quantitatively. Experimental results demonstrate that our method is able to provide a more than 1.0 dB gain in real-time manner compared to state-of-the-art methods. (C) 2019 Published by Elsevier B.V.