Empirical Evaluation of Federated Learning with Local Privacy for Real-World Application

As Machine Learning-based applications become increasingly pervasive, a growing concern is how to balance the need for large, representative data sets with the need to respect user data privacy. The increased compute and connectivity capabilities of edge devices (e.g. phones, PCs) presents us with new avenues for achieving this balance, including a promising approach known as federated learning with local privacy. However, today we have gaps in practical knowledge about applicability, trade-offs, and benefits for large-scale real-world implementation. In this paper, using large-scale data from a real-world Windows Update ML-driven application (as well as the publicly available CIFAR-10 data set to enhance reproducibility), we report empirical evaluations of four practical considerations: heterogeneity in device availability that may cause bias, resiliency of federated learning with local differential privacy, benefits of time-varying adaptive configurations, and data transmission/storage savings based on the Pareto principle. We discuss the implications of these findings for practitioners and researchers.