Effect of batch learning in multilayer neural networks

This paper discusses batch gradient descent learning in multilayer networks with a large number of statistical training data. We emphasize on the difference between regular cases, where the prepared model has the same size as the true function, and overrealizable cases, where the model has surplus hidden units to realize the true function. First, experimental study on multilayer perceptrons and linear neural networks (LNN) shows that batch learning induces strong overtraining on both models in overrealizable cases, which means the degrade of generalization error by surplus units can be alleviated. We theoretically analyze the dynamics in LNN, and show that this overtraining is caused by shrinkage of the parameters corresponding to surplus units.