Differentially private federated learning with local momentum updates and gradients filtering

Differential Privacy (DP) is applied in Federated Learning (FL) for defending against various privacy attacks. Existing methods based on Gaussian mechanism require the operations of clipping and adding noise, leading to significant accuracy degradation. In this paper, we propose a novel FL scheme named DPFL-LMG to provide user-level DP guarantee while maintaining a high model accuracy. Our main idea is to mitigate the negative effects of the clipping on the model convergence by decreasing the L-2 norm of local updates and the cross-client update variance. Specifically, our method includes two techniques, Local Momentum Updates (LMU) and Gradients Filtering (GF). LMU combines local updates of different rounds in a momentum way. It can significantly decrease the cross-client update variance by weakening the gradient noise in local updates caused by stochastic gradient descent (SGD) algorithm. GF estimates the gradient noise in each element of local updates by observing the element-wise variance. Elements with large noise are considered unnecessary and are zeroed out for the reduction of local update norms. We theoretically analyze the privacy guarantee and the convergence of our method. Experiments demonstrate that DPFL-LMG can effectively mitigate the accuracy degradation caused by clipping and outperform previous DPFL methods in the accuracy.