CovNN: A Covariance Neural Network Extended from CNN

Convolutional neural networks (CNNs) show commendable performance in computer vision, approaching high accuracy in a broad number of application domains. However, the training process of feature kernels in CNNs is easily affected by illumination intensity and feature interaction, which leads to over-fitting. In this paper, we propose a covariance neural network (CovNN), which replaces the original convolutional operation with our covariance algorithm, to make the learned kernels more robust to different illumination conditions and irrelevant features. This covariance layer uses the 3D covariance between all the input feature maps and the corresponding group of kernels by sliding window method, and regularizes them without additional parameters. Moreover, the covariance layer can be seamlessly transplanted to a variety of neural network architectures extended from CNNs (e.g., ResNet, Faster R-CNN). We evaluate the proposed CovNN on several popular datasets for image recognition (MNIST, Fashion-MNIST, CIFAR 10 and AR) and classification of organs (Abdominal Ultrasound Dataset). Experimental results demonstrate that CovNN achieves significant improvements over the state-of-the-art on most of them.