Integrated Optimization in Training Process for Binary Neural Network

Deep Neural Networks (DNNs) have recently become larger and deeper to keep up with more complex applications, resulting in high power and memory consumption. Due to simplicity in computation and storage, Binary Neural Networks (BNNs) have been one of the potential approaches to overcome these challenges. Previous works proposed many techniques to mitigate the accuracy degradation because of less bit-width representation. However, each technique follows different optimization directions, while the combination can gain better results. In addition, the padding value which is an essential factor directly affecting the accuracy and inference implementation has not been touched on in the state-of-the-art solutions. In this paper, based on the previous works, an integrated approach is applied in the training process for BNNs to improve accuracy and training stability. In particular, to increase the probability of changing weights' sign, the ReCU function proposed in related work is used to transform full-precision weight to binary weight, while to make the gradient mismatch of the sign function closer to the real one, the training-aware approximation function is used to replace the sign function. Besides, to make the BNNs compatible with post-XNOR implementation, the padding value for convolution is proposed to change to minus one from the default zero. The integrated method is implemented on the Cifar-10 dataset with VGG-small model shows that the training process is more stable with higher accuracy, compared to the baseline, while the model architecture and training algorithm are preserved.