A Learning Algorithm with a Gradient Normalization and a Learning Rate Adaptation for the Mini-batch Type Learning

The development of a high-performance optimization algorithm to solve the learning problem of the neural networks is strongly demanded with the advance of the deep learning. The learning algorithms with gradient normalization mechanisms have been investigated, and their effectiveness has been shown. In the learning algorithms, the adaptation of the learning rate is very important issue. The learning algorithms of the neural networks are classified into the batch learning and the mini-batch learning. In the learning with vast training data, the mini-batch type learning is often used due to the limitation of memory size and the computational cost. The mini-batch type learning algorithms with gradient normalization mechanisms have been investigated. However, the adaptation of the learning rate in the mini-batch type learning algorithm with the gradient normalization has not been investigated well. This study proposes to introduce a new learning rate adaptation mechanism based on sign variation of gradient to a mini-batch type learning algorithm with the gradient normalization. The effectiveness of the proposed algorithm is verified through applications to a learning problem of the multi-layered neural networks and a learning problem of the convolutional neural networks.