Compressing Fully Connected Layers using Kronecker Tensor Decomposition

In recent years, deep neural networks have made great achievements in many fields such as data science and image processing. Models of deep neural networks require pretty large memory storage resources with high computational complexity. In particular, the great number of parameters of fully connected layers prevents the application of deep neural networks in mobile devices. It is shown redundancy and low rank structure inside weight matrices of fully connected layers, thus methods of tensor decomposition can be applied to reduce the number of parameters. In this paper, a novel method for decomposition, Kronecker Tensor Decomposition with single pattern and multiple patterns is used to decompose the weight matrices of fully connected layers by using a series of sub-tensors to represent the original tensor with less amount of parameters. Experimental results showed that the proposed scheme can achieve a high compression ratio without significant accuracy loss.