ANALYSIS OF GRADIENT DESCENT LEARNING ALGORITHMS FOR MULTILAYER FEEDFORWARD NEURAL NETWORKS

We investigate certain dynamical properties of gradient-type learning algorithms as they apply to multilayer feedforward neural networks. These properties are more related to the multilayer structure of the net than to the particular threshold units at the nodes. The analysis explains the empirical observation that the weight sequence generated by backpropagation and related stochastic gradient algorithms exhibits a long term dependence on the initial choice of weights, and also a continued growth and/or drift even long after the outputs have converged. The analysis is carried out in two steps. First a simplified deterministic algorithm is derived using a describing function-type approach. Next, an analysis of the simplified algorithm is performed by considering an associated ordinary differential equation (ODE). Some numerical examples are given to illustrate the analysis. There has been almost no analysis of the dynamical behavior of backpropagation and related algorithms for the training of multilayer nets; this paper represents a first step in that direction.