Echo state network with a non-convex penalty for nonlinear time series prediction

Echo state networks (ESNs) with large reservoirs have been widely used in nonlinear time series prediction. However, over-large reservoirs will lead to ill-conditioned solutions when the output weights of ESNs are calculated by solving a linear regression problem. To address this issue, we propose an improved ESN with a nonconvex penalty (NCP-ESN) for nonlinear time series prediction. The main idea of NCP-ESN is that an adjustable log penalty with nonconvex characteristics is introduced to the loss function for generating unbiased and sparse solutions when optimizing the output weights of the network. Meanwhile, a learning method with two-stage optimization is developed for the optimal output weights by combining the coordinate descent algorithm with the generalized inverse method. Finally, two simulation sequences and two real sequences are used to test the performance of the proposed NCP-ESN on time series prediction. Experimental results have shown the better performance of the proposed NCP-ESN compared with some regularized ESNs.