Perceptual Quality Dimensions of Machine-Generated Text with a Focus on Machine Translation

被引：0
|
作者：
Macketanz, Vivien [1 ]
Naderi, Babak [2 ]
Schmidt, Steven [2 ]
Moeller, Sebastian [2 ]
机构：
[1] German Res Ctr AI, Kaiserslautern, Germany
[2] TU Berlin, Qual & Usabil Lab, Berlin, Germany
来源：
PROCEEDINGS OF THE 2ND WORKSHOP ON HUMAN EVALUATION OF NLP SYSTEMS (HUMEVAL 2022) | 2022年
关键词：
D O I：
暂无
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
The quality of machine-generated text is a complex construct consisting of various aspects and dimensions. We present a study that aims to uncover relevant perceptual quality dimensions for one type of machine-generated text, that is, Machine Translation. We conducted a crowd-sourcing survey in the style of a Semantic Differential to collect attribute ratings for German MT outputs. An Exploratory Factor Analysis revealed the underlying perceptual dimensions. As a result, we extracted four factors that operate as relevant dimensions for the Quality of Experience of MT outputs: precision, complexity, grammaticality, and transparency.
引用
收藏
页码：24 / 31
页数：8
相关论文
共 50 条
  • [1] MAGE: Machine-generated Text Detection in the Wild
    Li, Yafu
    Li, Qintong
    Cui, Leyang
    Bi, Wei
    Wang, Zhilin
    Wang, Longyue
    Yang, Linyi
    Shi, Shuming
    Zhang, Yue
    PROCEEDINGS OF THE 62ND ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, VOL 1: LONG PAPERS, 2024, : 36 - 53
  • [2] A Comparative Study on the Translation Quality between Human and Machine-Generated Subtitles
    Du, Jiaying
    Lu, Jiabi
    2024 6TH INTERNATIONAL CONFERENCE ON NATURAL LANGUAGE PROCESSING, ICNLP 2024, 2024, : 62 - 66
  • [3] Detection of Machine-Generated Text: Literature Survey
    University of Arkansas at Little Rock, United States
    arXiv,
  • [4] IMGTB: A Framework for Machine-Generated Text Detection Benchmarking
    Spiegel, Michal
    Macko, Dominik
    PROCEEDINGS OF THE 62ND ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, VOL 3: SYSTEM DEMONSTRATIONS, 2024, : 172 - 179
  • [5] On the Zero-Shot Generalization of Machine-Generated Text Detectors
    Pu, Xiao
    Zhang, Jingyu
    Han, Xiaochuang
    Tsvetkov, Yulia
    He, Tianxing
    FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS - EMNLP 2023, 2023, : 4799 - 4808
  • [6] RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text
    Dugan, Liam
    Ippolito, Daphne
    Kirubarajan, Arun
    Callison-Burch, Chris
    PROCEEDINGS OF THE 2020 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING: SYSTEM DEMONSTRATIONS, 2020, : 189 - 196
  • [7] On the Evaluation of Machine-Generated Reports
    Mayfield, James
    Yang, Eugene
    Lawrie, Dawn
    MacAvaney, Sean
    McNamee, Paul
    Oard, Douglas W.
    Soldaini, Luca
    Soboroff, Ian
    Weller, Orion
    Kayi, Efsun
    Sanders, Kate
    Mason, Marc
    Hibbler, Noah
    PROCEEDINGS OF THE 47TH INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL, SIGIR 2024, 2024, : 1904 - 1915
  • [8] Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
    Crothers, Evan N.
    Japkowicz, Nathalie
    Viktor, Herna L.
    IEEE ACCESS, 2023, 11 : 70977 - 71002
  • [9] RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
    Dugan, Liam
    Hwang, Alyssa
    Trhlik, Filip
    Ludan, Josh Magnus
    Zhu, Andrew
    Xu, Hainiu
    Ippolito, Daphne
    Callison-Burch, Chris
    PROCEEDINGS OF THE 62ND ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, VOL 1: LONG PAPERS, 2024, : 12463 - 12492
  • [10] Intelligent Granulation of Machine-generated Data
    Slezak, Dominik
    Kowalski, Marcin
    PROCEEDINGS OF THE 2013 JOINT IFSA WORLD CONGRESS AND NAFIPS ANNUAL MEETING (IFSA/NAFIPS), 2013, : 68 - 73
12345