Ensemble Method for Privacy-Preserving Logistic Regression Based on Homomorphic Encryption

Homomorphic encryption (HE) is one of promising cryptographic candidates resolving privacy issues in machine learning on sensitive data such as biomedical data and financial data. However, HE-based solutions commonly suffer from relatively high computational costs due to a large number of iterations in the optimization algorithms such as gradient descent (GD) for the learning phase. In this paper, we propose a new method called ensemble GD for logistic regression, a commonly used machine learning technique for binary classification. Our ensemble method reduces the number of iterations of GD, which results in substantial improvement on the performance of logistic regression based on HE in terms of speed and memory. The convergence of ensemble GD based on HE is guaranteed by our theoretical analysis on the erroneous variant of ensemble GD. We implemented ensemble GD for the logistic regression based on an approximate HE scheme HEAAN on MNIST data set and Credit data set from UCI machine learning repository. Compared to the standard GD for logistic regression, our ensemble method requires only about 60% number of iterations, which results in 60-70% reduction on the running time of total learning procedure in encrypted state, and 30-40% reduction on the storage of encrypted data set.