Reinforcement of BERT with Dependency-Parsing Based Attention Mask

被引：0

作者：

Mechouma, Toufik ^{[1
]}

Biskri, Ismail ^{[2
]}

Meunier, Jean Guy ^{[1
]}

机构：

[1] Univ Quebec Montreal, Montreal, PQ, Canada

[2] Univ Quebec Trois Rivieres, Trois Rivieres, PQ, Canada

来源：

ADVANCES IN COMPUTATIONAL COLLECTIVE INTELLIGENCE, ICCCI 2022 | 2022年 / 1653卷

关键词：

Bert; Transformers; Attention mechanisms; Dependency parsing;

D O I：

10.1007/978-3-031-16210-7_9

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Dot-Product based attention mechanism is among recent attention mechanisms. It showed an outstanding performance with BERT. In this paper, we propose a dependency-parsing mask to reinforce the padding mask, at the multi-head attention units. Padding mask, is already used to filter padding positions. The proposed mask, aims to improve BERT attention filter. The conducted experiments, show that BERT performs better with the proposed mask.

引用

页码：112 / 122

页数：11

共 50 条

[31] Generating Triples Based on Dependency Parsing for Contradiction Detection
Li, Luyang
Qin, Bing
Liu, Ting
SOCIAL MEDIA PROCESSING, SMP 2015, 2015, 568 : 200 - 208
[32] Transition-Based Dependency Parsing Exploiting Supertags
Ouchi, Hiroki
Duh, Kevin
Shindo, Hiroyuki
Matsumoto, Yuji
IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2016, 24 (11) : 2059 - 2068
[33] Dependency-based Hybrid Trees for Semantic Parsing
Jie, Zhanming
Lu, Wei
2018 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (EMNLP 2018), 2018, : 2431 - 2441
[34] Mining Product Reviews Based on Shallow Dependency Parsing
Zhang, Qi
Ogihara, Mitsunori
Wu, Yuanbin
Johnson, Joseph
Li, Tao
Huang, Xuanjing
PROCEEDINGS 32ND ANNUAL INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL, 2009, : 726 - 727
[35] Transition based neural network dependency parsing of Tibetan
Duo, Jiecairang
Hua, Quecairang
Huan, Keyou
Cai, Rangdangzhi
2020 2ND INTERNATIONAL CONFERENCE ON COMPUTER SCIENCE COMMUNICATION AND NETWORK SECURITY (CSCNS2020), 2021, 336
[36] Headed-Span-Based Projective Dependency Parsing
Yang, Songlin
Tu, Kewei
PROCEEDINGS OF THE 60TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2022), VOL 1: (LONG PAPERS), 2022, : 2188 - 2200
[37] Chinese Dependency Parsing Based on An Improved Model of MST
Qi, Sun
Yang, Xiang
Xiao, Tu
PROCEEDINGS OF THE 2ND INTERNATIONAL CONFERENCE ON ELECTRONICS, NETWORK AND COMPUTER ENGINEERING (ICENCE 2016), 2016, 67 : 9 - 14
[38] Multilingual dependency-based syntactic and semantic parsing
Che, Wanxiang
Li, Zhenghua
Li, Yongqiang
Guo, Yuhang
Qin, Bing
Liu, Ting
CoNLL- 2009: Shared Task - Proceedings of the Thirteenth Conference on Computational Natural Language Learning, CoNLL: Shared Task, 2009, : 49 - 54
[39] Transition-based dependency parsing with topological fields
de Kok, Daniel
Hinrichs, Erhard
PROCEEDINGS OF THE 54TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2016), VOL 2, 2016, : 1 - 7
[40] Dependency Parsing Based Chinese Open Relation Extraction
Xu, Jing
Gan, Liang
Deng, Lu
Wang, Jing
Yan, Zhou
PROCEEDINGS OF 2015 4TH INTERNATIONAL CONFERENCE ON COMPUTER SCIENCE AND NETWORK TECHNOLOGY (ICCSNT 2015), 2015, : 552 - 556

← 1 2 3 4 5 →