Named Entity Recognition Method Based on Multi-Teacher Collaborative Cyclical Knowledge Distillation

Named Entity Recognition (NER) is a crucial task in Natural Language Processing (NLP), with applications ranging from information retrieval to biomedical research. Large pre-trained language models, like BERT, have significantly improved NER performance, but they require substantial computational resources. Knowledge distillation, a method where a smaller "student" model learns from a larger "teacher" model, can compress models while retaining their effectiveness. This paper introduces Multi-Teacher Collaborative Cyclical Knowledge Distillation (MTCCKD), a novel approach inspired by collaborative learning. MTCCKD addresses the "curse of competence gap" by using multiple teachers of varying expertise. In each iteration, the student assesses its performance and decides whether to change teachers. This collection of teachers collaboratively works to enhance the student model. MTCCKD effectively compresses knowledge while maintaining or even improving NER performance, improving efficiency, adaptability, and robustness. Empirical validation on publicly available NER datasets demonstrates that MTCCKD outperforms state-of-the-art models, achieving a 22-fold model compression while preserving 96% of the teacher model's performance. This method offers a promising solution for practical NER tasks in resource-constrained environments.