Learning Task-Specific Initialization for Effective Federated Continual Fine-Tuning of Foundation Model Adapters

As large models demonstrate their power across a wide range of applications, the federated learning (FL) community has also begun to seek solutions for leveraging these large models in a communication- and computation-efficient manner. In light of this, fine-tuning of lightweight adapters has emerged as a promising solution for adopting large models in FL. Another real-world challenge concerns with non-static data streams encountered by local clients, requiring continuous adapter fine-tuning to accommodate new tasks. In this work, we propose a method for effective continual adapter fine-tuning in FL (FedCAF), aimed at enhancing a client's local learning on new tasks. Specifically, FedCAF employs both cross-task and cross-client knowledge transfer to generate an informed, task-specific initialization. By learning a set of attentive weights to combine past task models from all clients, FedCAF produces task-specific initializations that effectively enable better and faster task learning. On the large-scale cross-domain dataset DomainNet, we show that FedCAF significantly outperforms several competitive personalized and continual learning baselines under both class-incremental and domain-incremental settings.