AMBICOREF: Evaluating Human and Model Sensitivity to Ambiguous Coreference

被引：0

作者：

Yuan, Yuewei ^{[1
]}

Malaviya, Chaitanya ^{[1
]}

Yatskar, Mark ^{[1
]}

机构：

[1] Univ Penn, Philadelphia, PA 19104 USA

来源：

17TH CONFERENCE OF THE EUROPEAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, EACL 2023 | 2023年

关键词：

D O I：

暂无

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Given a sentence "Abby told Brittney that she upset Courtney", one would struggle to understand who "she" refers to, and ask for clarification. However, if the word "upset" were replaced with "hugged", "she" unambiguously refers to Abby. We study if modern co-reference resolution models are sensitive to such pronominal ambiguity. To this end, we construct AMBICOREF, a diagnostic corpus of minimal sentence pairs with ambiguous and unambiguous referents. Our examples generalize psycholinguistic studies of human perception of ambiguity around particular arrangements of verbs and their arguments. Analysis shows that (1) humans are less sure of referents in ambiguous AmbiCoref examples than unambiguous ones, and (2) most coreference models show little difference in output between ambiguous and unambiguous pairs. We release AMBICOREF as a diagnostic corpus for testing whether models treat ambiguity similarly to humans.(1)

引用

页码：1023 / 1030

页数：8

共 50 条

[11] Decision biases in evaluating ambiguous information
Brown, CL
Chernev, A
ADVANCES IN CONSUMER RESEARCH, VOL XXIV, 1997, 24 : 173 - 174
[12] OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres
Zhu, Yilun
Pradhan, Sameer
Zeldes, Amir
ACL-IJCNLP 2021: THE 59TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS AND THE 11TH INTERNATIONAL JOINT CONFERENCE ON NATURAL LANGUAGE PROCESSING, VOL 2, 2021, : 461 - 467
[13] Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective
Poradal, Ian
Olteanu, Alexandra
Suleman, Kaheer
Trischler, Adam
Cheung, Jackie Chi Kit
FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS: ACL 2024, 2024, : 15380 - 15395
[14] Evaluating the Impact of a Hierarchical Discourse Representation on Entity Coreference Resolution Performance
Khosla, Sopan
Fiacco, James
Rose, Carolyn
2021 CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS: HUMAN LANGUAGE TECHNOLOGIES (NAACL-HLT 2021), 2021, : 1645 - 1651
[15] An ELECTRA-Based Model for Neural Coreference Resolution
Gargiulo, Francesco
Minutolo, Aniello
Guarasci, Raffaele
Damiano, Emanuele
De Pietro, Giuseppe
Fujita, Hamido
Esposito, Massimo
IEEE ACCESS, 2022, 10 : 75144 - 75157
[16] When less is more: How excluding experimentally ambiguous observations may enhance the sensitivity of a model
Ghosh, Jayeeta
Lawless, Michael
Clark, Robert
ABSTRACTS OF PAPERS OF THE AMERICAN CHEMICAL SOCIETY, 2015, 250
[17] TECHNIQUE FOR EVALUATING MODEL SENSITIVITY TO VARIATIONS IN GLOBAL TRANSPORT COEFFICIENTS
PETERSEN, T
DWYER, H
BREWER, J
BULLETIN OF THE AMERICAN METEOROLOGICAL SOCIETY, 1973, 54 (10) : 1125 - 1125
[18] Evaluating the Sensitivity of Agricultural Model Performance to Different Climate Inputs
Glotter, Michael J.
Moyer, Elisabeth J.
Ruane, Alex C.
Elliott, Joshua W.
JOURNAL OF APPLIED METEOROLOGY AND CLIMATOLOGY, 2016, 55 (03) : 579 - 594
[19] Contrast Sensitivity Model of the Human Eye
Roka, Andras
Galambos, Peter
Baranyi, Peter
ISCII 2009: 4TH INTERNATIONAL SYMPOSIUM ON COMPUTATIONAL INTELLIGENCE AND INTELLIGENT INFORMATICS, PROCEEDINGS, 2009, : 93 - 99
[20] Sensitivity Analysis of a Human Finger Model
Allouch, S.
Younes, R.
Laforet, J.
Boudaoud, S.
Khalil, M.
2017 FOURTH INTERNATIONAL CONFERENCE ON ADVANCES IN BIOMEDICAL ENGINEERING (ICABME), 2017, : 57 - 60

← 1 2 3 4 5 →