The effect of initial weight, learning rate and regularization on generalization performance and efficiency

The goal of this paper is to study the factors that affect the generalization performance and efficiency for neural network learning. First, this paper will investigate the effect of initial weight ranges, learning rate, and regularization coefficient on generalization performance and learning speed. Based on this, we will propose a hybrid method that simultaneously considers these three factors, and dynamically tune the learning rate and regularization coefficient. Then we will present the results of some experimental comparison among these kinds of methods in several different problems. Finally, we will draw conclusions and make plan for future work.