Two constructive algorithms for improved time series processing with recurrent neural networks

Because of their universal approximation capabilities, recurrent neural networks are an attractive choice for building models of time series out of available data. Medium and long-term dependencies are easier to learn when the recurrent network contains time-delayed connections. We propose here two constructive algorithms which are able to choose the right locations and delays of such connections. To evaluate the capabilities of these algorithms we use both natural data and synthetic data having built-in time delays. We then compare the two algorithms in order to define their domain of interest. The results we obtain on several benchmarks show that by selectively adding a few time-delayed connections to recurrent networks, one is able to improve upon the results reported in the literature, while using significantly fewer parameters.