Optimization of feedforward neural networks

This paper presents some novel approaches in the design of neural networks with one or two hidden layers trained by the backpropagation algorithm. First, hybrid neural networks that have different activation functions for different layers in fully connected feedforward neural networks are introduced. Second, a variant sigmoid function with three parameters is discussed. The parameters are the dynamic range, symmetry and slope of the function respectively, It is illustrated how these parameters influence the speed of backpropagation learning, and a parametric feedforward network with different parameter configurations in different layers is introduced. By regulating and modifying parameter configurations of the sigmoid function in different layers the error signal problem, oscillation problem and asymmetrical input problem can be reduced. Furthermore, hybrid optimization methods for the dynamic parameters are introduced: Genetic algorithms are used to optimize the initial parameter configuration. The dynamic parameters are adjusted on-line using gradient descent methods. Sequential adjustment algorithms are derived in order to avoid the moving target problem and to increase the stability of the gradient descent methods. The new schemes have advantages in both convergence speed and generalization capability. Experimental results on the two-spirals problem are provided. Copyright (C) 1996 Elsevier Science Ltd