Non-stationary Multivariate Time Series Prediction with Selective Recurrent Neural Networks

Non-stationary multivariate time series (NSMTS) prediction is still a challenging issue nowadays. Methods based on deep learning, especially Long Short-Term Memory and Gated Recurrent Unit neural networks (LSTMs and GRUs) have achieved state-of-the-art results. However, the architecture of LSTM and GRU may contain some useless components that affect the training efficiency, thus it is possible that optional architecture exists. Recently, newly-introduced one gate Minimal Gated Unit neural networks (MGUs) have exhibited promising results in computer vision and some sequence analysis applications. In this paper, we first transplant the MGUs into NSMTS prediction and then evaluate the ability of LSTMs, GRUs and MGUs via experiments. During these trials, none of these neural networks can always dominate in performance over all the NSMTS. Therefore, we further propose a novel Selective Recurrent Neural Networks with Random Connectivity Gated Unit (SRCGUs) that train random connectivity LSTMs, GRUs and MGUs at a time. This model can not only reduce the number of parameters and save about 2/3 of time compared to the separate training but also adjust their importance weights dynamically to select a more appropriate neural network for prediction. Experimental results show that SRCGUs have better performance on the benchmarks used and flexibility. And to the best of our knowledge, this selective architecture has never been reported before.