Improving AMR-to-text Generation with Multi-task Pre-training

被引：0
|
作者：
Xu D.-Q. [1 ]
Li J.-H. [1 ]
Zhu M.-H. [2 ]
Zhou G.-D. [1 ]
机构：
[1] School of Computer Science and Technology, Soochow University, Suzhou
[2] Tencent News, Tencent Technology (Beijing) Co. Ltd., Beijing
来源：
Ruan Jian Xue Bao/Journal of Software | 2021年 / 32卷 / 10期
基金：
中国国家自然科学基金; 国家重点研发计划;
关键词：
Abstract meaning representation (AMR); AMR-to-text generation; Multi-task pre-training; Sequence-to-sequence;
D O I：
10.13328/j.cnki.jos.006207
中图分类号：
学科分类号：
摘要：
Given an AMR (abstract meaning representation) graph, AMR-to-text generation aims to generate text with the same meaning. Related studies show that the performance of AMR-to-text severely suffers from the size of the manually annotated dataset. To alleviate the dependence on manually annotated dataset, this study proposes a novel multi-task pre-training for AMR-to-text generation. In particular, based on a large-scale automatic AMR dataset, three relevant pre-training tasks are defined, i.e., AMR denoising auto-encoder, sentence denoising auto-encoder, and AMR-to-text generation itself. In addition, to fine-tune the pre-training models, the vanilla fine-tuning method is further extended to multi-task learning fine-tuning, which enables the final model to maintain performance on both AMR-to-text and pre-training tasks. With the automatic dataset of 0.39M sentences, detailed experimentation on two AMR benchmarks shows that the proposed pre-training approach significantly improves the performance of AMR-to-text generation, with the improvement of 12.27 BLEU on AMR2.0 and 7.57 on AMR3.0, respectively. This greatly advances the state-of-the-art performance with 40.30 BLEU on AMR2.0 and 38.97 on AMR 3.0, respectively. To the best knowledge, this is the best result achieved so far on AMR 2.0 while AMR-to- text generation performance on AMR 3.0 is firstly reported. © Copyright 2021, Institute of Software, the Chinese Academy of Sciences. All rights reserved.
引用
收藏
页码：3036 / 3050
页数：14
相关论文
共 45 条
  • [1] Banarescu L, Bonial C, Cai S, Et al., Abstract meaning representation for sembanking, Proc. of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pp. 178-186, (2013)
  • [2] Tamchyna A, Quirk C, Galley M., A discriminative model for semantics-to-string translation, Proc. of the 1st Workshop on Semantics-driven Statistical Machine Translation, pp. 30-36, (2015)
  • [3] Mitra A, Baral C., Addressing a question answering challenge by combining statistical methods with inductive rule learning and reasoning, Proc. of the 30th AAAI Conf. on Artificial Intelligence, pp. 2779-2785, (2016)
  • [4] Li X, Nguyen T H, Cao K, Et al., Improving event detection with abstract meaning representation, Proc. of the 1st Workshop on Computing News Storylines, pp. 11-15, (2015)
  • [5] Xu K, Wu L, Wang Z, Et al., Graph2seq: Graph to sequence learning with attention-based neural networks, (2018)
  • [6] Flanigan J, Dyer C, Smith NA, Et al., Generation from abstract meaning representation using tree transducers, Proc. of the 2016 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 731-739, (2016)
  • [7] Song L, Peng X, Zhang Y, Et al., AMR-to-text generation with synchronous node replacement grammar, Proc. of the 55th Annual Meeting of the Association for Computational Linguistics, 2, pp. 7-13, (2017)
  • [8] Pourdamghani N, Knight K, Hermjakob U., Generating English from abstract meaning representations, Proc. of the 9th Int'l Natural Language Generation Conf, 21, (2016)
  • [9] Ferreira TC, Calixto I, Wubben S, Et al., Linguistic realisation as machine translation: Comparing different MT models for AMR-to-text generation, Proc. of the 10th Int'l Conf. on Natural Language Generation, pp. 1-10, (2017)
  • [10] Zhu J, Li J, Zhu M, Et al., Modeling graph structure in transformer for better AMR-to-text generation, Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing and the 9th Int'l Joint Conf. on Natural Language Processing, pp. 5462-5471, (2019)
12345