Tensor Ring Restricted Boltzmann Machines

Restricted Boltzmann Machines are important and useful generative models which learn a probability distribution from a set of vector inputs. Despite their success in a number of applications, standard RBMs designed for vectorized inputs are incapable of dealing with high-order data, since vectorization of high-order data may cause both modes collapsing and explosive parameter growth. To address this issue, we formulate a new tensor-input RBM model, which employs the tensor-ring (TR) decomposition structure to naturally represent the high-order relationship between the visual layer and the hidden layer. For convenience, we name the proposed model as TR-RBM. In particular, the tensor ring decomposition enjoys many good properties, such as the rank stableness, leading to better generalization performance compared with other low-rank decomposition methods. Moreover, TR-RBM can also reduce the complexity of RBM by reshaping of both visible and hidden layers into the tensor forms, leading a significant drop of parameter size. Experimental results in comparison with the classical RBMs and the Matrix-Product-Operator RBM have shown the promising performance of the proposed method in the tasks of feature extraction and denoising.