A New Learning Algorithm with General Loss for Neural Networks with Random Weights

Neural networks with random weights (NNRWs) which randomly assign weights to new hidden nodes, provide a new and promising stochastic approach for the research of neural networks, and have been proved to enjoy the universal approximation property. However, the design of existing learning algorithms for NNRWs is only based on the square loss function, which hinders the development of NNRWs. For strictly convex and strongly convex loss functions, it is unclear how to design learning algorithms, and what the convergence rates are. In this paper, we answer the qu estions affirmatively. First, we propose a new supervisory mechanism for constructing NNRWs, and prove the universal approximation property. Then we design a new learning algorithm and analyze the convergence rates of the algorithm, based on which the time complexities are also analyzed. To be specific, the proposed algorithm enjoys sublinear convergence for smooth and strictly convex loss functions, and linear convergence for smooth and strongly convex loss functions. Finally, experimental results on several real-world datasets verify the convergence rates of the algorithm with different loss functions.