Communication-efficient and Utility-Aware Adaptive Gaussian Differential Privacy for Personalized Federated Learning

In recent years, there has been an increasing focus on the privacy protection in the field of federated learning (FL). This widespread attention is mainly due to the fact that communication parameters (or gradients) during the process of collaborative learning among the central server and various participants can cause the significant risk of the privacy leakage. In other words, the communication process in the FL system poses a potential threat of exposing the sensitive data belonging to local participants, which has raised heightened concerns among researchers and practitioners. Furthermore, in addition to the challenge of the privacy protection in FL, a series of other unavoidable factors such as the frequent gradients exchange, the heterogeneous data distribution among local participants, and limited resources available on the local hardware need to be simultaneously taken into consideration. These factors obviously add difficulties to the challenge of the privacy protection in FL. In order to effectively address four critical issues of data privacy, model utility, communication efficiency, and non-independently and identically distributed data among local participants in a unified manner, this paper proposes a novel Communication-efficient and Utility-aware Adaptive Gaussian Differential Privacy for Personalized FL method, called CUAG-PFL. Specifically, a dynamic layer-compression scheme for model gradients in the FL system is proposed. This scheme aims to improve the communication efficiency as much as possible and reduce the loss of the model utility caused by compression and reconstruction through dynamically customizing the compression rate for each layer of communication gradients, and then constructing the corresponding deterministic binary measurement matrix based on the compression rate. This designed deterministic binary measurement matrix can effectively remove the redundant information of model gradients that needs to be uploaded to the central server. Subsequently, the adaptive Gaussian differential privacy operation is performed on compressed model gradients of local participants. This operation involves optimizing the main privacy-related parameters such as the clipping threshold, the sensitivity, and the noise scale. By optimizing these parameters at the same time, this operation ensures that the privacy of the local data is preserved, while allowing each model of the corresponding local participant to have the satisfactory performance. In addition, the rigorous privacy analysis of the proposed CUAG-PFL is presented in this paper. In order to validate the superiority of the proposed CUAG-PFL in four critical aspects of data privacy, model utility, communication efficiency, and personalization, a large number of experimental simulations, comparisons, and analyses are conducted on two classic real-world federated datasets, i.e., CIFAR-10 and CIFAR-100. All experimental results and analyses show that the proposed CUAG-PFL can simultaneously improve the privacy of local sensitive data, the communication efficiency and the model utility, as well as address the problem of non-independently and identically distributed data among local participants in the FL system. In particular, it is worth emphasizing that even when the privacy budget is only 0.92 and the amount of the upstream communication is reduced by 68.6%, the loss of the model performance caused by both the privacy protection and the communication gradients compression is just 1.66% for the proposed CUAG-PFL. © 2024 Science Press. All rights reserved.