Natural Gradient Descent of Complex-Valued Neural Networks Invariant under Rotations

The natural gradient descent is an optimization method for real-valued neural networks that was proposed from the viewpoint of information geometry. Here, we present an extension of the natural gradient descent to complex-valued neural networks. Our idea is to use the Hermitian extension of the Fisher information matrix. Moreover, we generalize the projected natural gradient (PRONG), which is a fast natural gradient descent algorithm, to complex-valued neural networks. We also consider the advantage of complex-valued neural networks over real-valued neural networks. A useful property of complex numbers in the complex plane is that the rotation is simply expressed by the multiplication. By focusing on this property, we construct the output function of complex-valued neural networks, which is invariant even if the input is changed to its rotated value. Then, our complex-valued neural network can learn rotated data without data augmentation. Finally, through simulation of online character recognition, we demonstrate the effectiveness of the proposed approach.