Harnessing Knowledge Distillation for Enhanced Text-to-Text Translation in Low-Resource Languages

Text-to-text translation is crucial for effective communication and understanding across different languages. In this paper, we present a deep learning-based approach for text-to-text translation. Our method leverages knowledge distillation from a high-performing teacher model, specifically the BART model, to train a smaller and more efficient student model, the mBART model. For that, we minimize the cross-entropy between the model distribution and a learned teacher distribution rather than the observed data, to achieve effective knowledge distillation. Our approach mitigates catastrophic forgetting, especially in low-resource languages, by utilizing the complementary knowledge provided by the teacher model. Extensive experimentation and evaluation demonstrate that our model outperforms state-of-the-art methods, achieving superior BLEU scores on benchmark datasets for French-to-Russian, English-to-Dutch, and Russian-to-Vietnamese translations. An ablation study further shows that the combination of fine-tuning and knowledge distillation enhances the student model's ability to capture linguistic nuances and produce more accurate translations.