Projection-free Online Empirical Risk Minimization with Privacy-preserving and Privacy Expiration

Streaming machine learning and data mining problems are prevalent in real world applications, where individual data are collected and revealed consecutively. These problems can often be modeled and solved under Constrained Online Convex Optimization (COCO) algorithmic framework. The ever-growing amount of sensitive individual data is posing greater challenge to the contradictory goals of privacy protection and reasonable model usability. Despite its extensive studies via projected/proximal gradient based methods, its projection-free counterpart has not been well-explored. Inspired by the better per-iteration computational efficiency and privacy-utility tradeoff under non-private/non-online settings of the projection-free algorithms, we propose the projection-free COCO with differential privacy guarantee, a de facto standard for privacy preserving. We rigorously analyze its utility in terms of regret rate, which shows that, even without the expensive projection/proximal operators, it still matches the differentially privacy COCO with projection/proximal operations. To the best of our knowledge, it is the first projection-free differentially private COCO, and thus broadens the applicability of COCO with privacy guarantee. Furthermore, since protecting the privacy of all incoming samples will lead to inferior regret rate compared to the nonprivate optimal, we propose a relaxed privacy guarantee which trades the privacy of remote samples for better utility. To achieve this, we adopt a window tree mechanism for maintaining a private gradient summation, which is then used to construct an approximation function for updating the new response variable at each timestamp. It improves the regret bound to O(ln T) with respect to the sequence length T, matching the nonprivate optimal regret.