A Test Suite for the Evaluation of Portuguese-English Machine Translation

被引：0
|
作者：
Avelino, Mariana [1 ]
Macketanz, Vivien [2 ]
Avramidis, Eleftherios [2 ]
Moller, Sebastian [1 ,2 ]
机构：
[1] Tech Univ Berlin, Berlin, Germany
[2] German Res Ctr Artificial Intelligence DFKI, Berlin, Germany
来源：
COMPUTATIONAL PROCESSING OF THE PORTUGUESE LANGUAGE, PROPOR 2022 | 2022年 / 13208卷
关键词：
Machine translation; Evaluation; Portuguese; Test suite;
D O I：
10.1007/978-3-030-98305-5_2
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
This paper describes the development of the first test suite for the language direction Portuguese-English. Designed for fine-grained linguistic analysis, the test suite comprises 330 test sentences for 66 linguistic phenomena and 14 linguistic categories. Eight different MT systems were compared using quantitative and qualitative methods via the test suite: DeepL, Google Sheets, Google Translator, Microsoft Translator, Reverso, Systran, Yandex and an internally built NMT system trained over 30 h on 2,5M sentences. It was found that ambiguity, named entity & terminology and verb valency are the categories where MT systems struggle most. Negation, pronouns, subordination, verb tense/aspect/mood and false friends are the categories where MT systems perform best.
引用
收藏
页码：15 / 25
页数：11
相关论文
共 50 条
  • [1] Linguistic evaluation of German-English Machine Translation using a Test Suite
    Avramidis, Eleftherios
    Macketanz, Vivien
    Strohriegel, Ursula
    Uszkoreit, Hans
    [J]. FOURTH CONFERENCE ON MACHINE TRANSLATION (WMT 2019), 2019, : 445 - 454
  • [2] A DICTIONARY OF VERBAL COLLIGATIONS - ENGLISH-PORTUGUESE PORTUGUESE-ENGLISH
    TAGNIN, SEO
    [J]. AMERICAN TRANSLATORS ASSOCIATION CONFERENCE 1989, 1989, : 343 - 346
  • [3] INVESTIGATING THE COGNITIVE PROCESSING OF PROFESSIONAL PORTUGUESE-ENGLISH TRANSLATORS IN DIRECT AND INVERSE TRANSLATION
    Ferreira, Aline Alves
    [J]. CADERNOS DE TRADUCAO, 2012, 29 (01): : 73 - 92
  • [4] A PORTUGUESE-ENGLISH DICTIONARY - TAYLOR,JL
    KURTZ, LP
    [J]. MODERN LANGUAGE JOURNAL, 1959, 43 (01): : 54 - 54
  • [5] Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite
    Jwalapuram, Prathyusha
    Joty, Shafiq
    Temnikova, Irina
    Nakov, Preslav
    [J]. 2019 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING AND THE 9TH INTERNATIONAL JOINT CONFERENCE ON NATURAL LANGUAGE PROCESSING (EMNLP-IJCNLP 2019): PROCEEDINGS OF THE CONFERENCE, 2019, : 2964 - 2975
  • [6] Introducing Compara, the Portuguese-English parallel corpus
    Frankenberg-Garcia, Ana
    Santos, Diana
    [J]. Corpora in Translator Education, 2003, : 71 - 87
  • [7] Textual markers of the scientific article in Portuguese-English comparison - a study from a translation perspective
    Bocorny Finatto, Maria Jose
    [J]. DEBATE TERMINOLOGICO, 2005, (01):
  • [8] Vocabulary acquisition and the use of dictionaries portuguese-english
    da Silva Polizer, Alex Sandro
    de Oliveira Marques, Giseli Sampaio
    Santos Zacarias, Regiani Aparecida
    [J]. ENTREPALAVRAS, 2022, 11 : 195 - 214
  • [9] Portuguese-English word alignment: some experiments
    Santos, Diana
    Simoes, Alberto
    [J]. SIXTH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION, LREC 2008, 2008, : 2988 - 2995
  • [10] Portuguese-English experiments using latent semantic indexing
    Orengo, VM
    Huyck, C
    [J]. ADVANCES IN CROSS-LANGUAGE INFORMATION RETRIEVAL, 2003, 2785 : 147 - 154
12345