CHE: Channel-Wise Homomorphic Encryption for Ciphertext Inference in Convolutional Neural Network

Privacy-preserving deep learning (PPDL), which leverages Homomorphic Encryption (HE), has attracted attention as a promising approach to ensure the privacy of deep learning applications' data. While recent studies have developed and evaluated the HE-based PPDL algorithms, the achieved performances, such as accuracy and latency, need improvement to make the applications practical. This work aims to improve the performance of the image classification of HE-based PPDL by combining two approaches - Channel-wise Homomorphic Encryption (CHE) and Batch Normalization (BN) with coefficient merging. Although these are commonly used schemes, their detailed algorithms and formulations have not been clearly described. The main contribution of the current study is to provide complete and reproducible descriptions of these schemes. We evaluate our CHE and BN implementation by targeting the Cheon-Kim-Kim-Song scheme as an HE scheme and Convolution Neural Network (CNN) as a machine learning scheme while using the MNIST and CIFAR-10 as the datasets. In addition, we compare the results with the five state-of-the-art neural network architectures. Our experiments demonstrate that the CHE can serve as a tool for empirically achieving shorter latency (the shortest 7.76 seconds) and higher accuracy (the highest 99.32%) compared with the previous studies that aimed to establish the classification of the encrypted MNIST data with CNN. Our approach can aid in designing a more robust and flexible PPDL.