Enhancing Recommendation Capabilities Using Multi-Head Attention-Based Federated Knowledge Distillation

被引:1
|
作者
Wu, Aming [1 ]
Kwon, Young-Woo [1 ]
机构
[1] Kyungpook Natl Univ, Sch Comp Sci & Engn, Daegu 41566, South Korea
基金
新加坡国家研究基金会;
关键词
Data models; Adaptive learning; Data privacy; Servers; Privacy; Computational modeling; Solid modeling; Federated learning; multi-head attention; Wasserstein distance; adaptive learning rate; CHALLENGES;
D O I
10.1109/ACCESS.2023.3271678
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
As the internet and mobile computing have advanced, recommendation algorithms are used to manage large amounts of data. However, traditional recommendation systems usually require collecting user data on a central server, which may expose user privacy. Furthermore, data and models from different organizations may be proprietary and cannot be shared directly, leading to data isolation. To address these challenges, we propose a method that combines federated learning (FL) with the recommendation system using a federated knowledge distillation algorithm based on a multi-head attention mechanism. In the proposed approach, knowledge distillation is introduced on the basis of FL to induce the training of the local network and facilitate knowledge transfer. Further, to address the non-independent identical distribution of training samples in FL, Wasserstein distance and regularization terms are incorporated into the objective function of federated knowledge distillation to reduce the distribution difference between server and client networks. A multi-head attention mechanism is used to enhance user encoding information. A combined adaptive learning rate is adopted to further improve the convergence. Compared to the benchmark model, this approach demonstrates significant improvements, with accuracy enhanced up to 10%, model training time shortened by approximately 45%, and average error and NDCG values decreased by 10%.
引用
收藏
页码:45850 / 45861
页数:12
相关论文
共 50 条
  • [1] MAFD: A Federated Distillation Approach with Multi-head Attention for Recommendation Tasks
    Wu, Aming
    Kwon, Young-Woo
    [J]. 38TH ANNUAL ACM SYMPOSIUM ON APPLIED COMPUTING, SAC 2023, 2023, : 1221 - 1224
  • [2] Combining Multi-Head Attention and Sparse Multi-Head Attention Networks for Session-Based Recommendation
    Zhao, Zhiwei
    Wang, Xiaoye
    Xiao, Yingyuan
    [J]. 2023 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS, IJCNN, 2023,
  • [3] Multi-Head Attention-Based Spectrum Sensing for Radio
    Devarakonda, B. V. Ravisankar
    Nandanavam, Venkateswararao
    [J]. INTERNATIONAL JOURNAL OF ELECTRICAL AND COMPUTER ENGINEERING SYSTEMS, 2023, 14 (02) : 135 - 143
  • [4] Personalized federated learning based on multi-head attention algorithm
    Jiang, Shanshan
    Lu, Meixia
    Hu, Kai
    Wu, Jiasheng
    Li, Yaogen
    Weng, Liguo
    Xia, Min
    Lin, Haifeng
    [J]. INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS, 2023, 14 (11) : 3783 - 3798
  • [5] Personalized federated learning based on multi-head attention algorithm
    Shanshan Jiang
    Meixia Lu
    Kai Hu
    Jiasheng Wu
    Yaogen Li
    Liguo Weng
    Min Xia
    Haifeng Lin
    [J]. International Journal of Machine Learning and Cybernetics, 2023, 14 : 3783 - 3798
  • [6] Intelligent Bearing Fault Diagnosis Using Multi-Head Attention-Based CNN
    Wang, Hui
    Xu, Jiawen
    Yan, Ruqiang
    Sun, Chuang
    Chen, Xuefeng
    [J]. PROCEEDINGS OF THE 8TH INTERNATIONAL CONFERENCE ON THROUGH-LIFE ENGINEERING SERVICES (TESCONF 2019), 2020, 49 : 112 - 118
  • [7] Internal defects inspection of arc magnets using multi-head attention-based CNN
    Li, Qiang
    Huang, Qinyuan
    Yang, Tian
    Zhou, Ying
    Yang, Kun
    Song, Hong
    [J]. MEASUREMENT, 2022, 202
  • [8] Sequential Recommendation Using Deep Reinforcement Learning and Multi-Head Attention
    Sultan, Raneem
    Abu-Elkheir, Mervat
    [J]. 2022 56TH ANNUAL CONFERENCE ON INFORMATION SCIENCES AND SYSTEMS (CISS), 2022, : 258 - 262
  • [9] Federated learning based multi-head attention framework for medical image classification
    Firdaus, Naima
    Raza, Zahid
    [J]. CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE, 2024,
  • [10] A personalized federated learning method based on the residual multi-head attention mechanism
    Li, Zhaobin
    Zhong, Zixuan
    Zuo, Peiliang
    Zhao, Hong
    [J]. JOURNAL OF KING SAUD UNIVERSITY-COMPUTER AND INFORMATION SCIENCES, 2024, 36 (04)