Leveraging Adversarial Training to Facilitate Grammatical Error Correction

Grammatical error correction (GEC) task aims to detect and correct grammatical errors in sentences. Recently, the pre-trained language model has provided a strong baseline for GEC and achieved excellent results by fine-tuning on a small amount of annotated data. However, due to the lack of large-scale erroneous-corrected parallel datasets, these models tend to suffer from the problem of overfitting. Previous researchers have proposed a variety of data augmentation methods to generate more training data and enlarge the dataset, but these methods either rely on rules to generate grammatical errors and are not automated, or produce errors that do not match human writing errors. The pre-trained model only improves significantly after task-specific data fine-tuning; otherwise, the highly noisy data can impair the performance of the pre-trained model. To address this issue, we propose a method to enhance the robustness of the model based on adversarial training. This approach constructs the adversarial samples and treats them as the augmented data. Unlike previous methods that introduce token-level noise, our method introduces embedding-level noise and can obtain extra samples that are close to human writing errors. Besides, we employ the adversarial consistency constraint to reduce the gap between the adversarial sample and the original sample. The experimental results demonstrate that our method can further boost the performance of the pre-trained model on GEC task.