Reinforcement of BERT with Dependency-Parsing Based Attention Mask

被引：0

作者：

Mechouma, Toufik ^{[1
]}

Biskri, Ismail ^{[2
]}

Meunier, Jean Guy ^{[1
]}

机构：

[1] Univ Quebec Montreal, Montreal, PQ, Canada

[2] Univ Quebec Trois Rivieres, Trois Rivieres, PQ, Canada

来源：

ADVANCES IN COMPUTATIONAL COLLECTIVE INTELLIGENCE, ICCCI 2022 | 2022年 / 1653卷

关键词：

Bert; Transformers; Attention mechanisms; Dependency parsing;

D O I：

10.1007/978-3-031-16210-7_9

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Dot-Product based attention mechanism is among recent attention mechanisms. It showed an outstanding performance with BERT. In this paper, we propose a dependency-parsing mask to reinforce the padding mask, at the multi-head attention units. Padding mask, is already used to filter padding positions. The proposed mask, aims to improve BERT attention filter. The conducted experiments, show that BERT performs better with the proposed mask.

引用

页码：112 / 122

页数：11

共 50 条

[41] STRESS DISTRIBUTION BASED ON DEPENDENCY PARSING OF CHINESE DISCOURSE
Wang, Yaru
Li, Aijun
Jia, Yuan
2014 17TH ORIENTAL CHAPTER OF THE INTERNATIONAL COMMITTEE FOR THE CO-ORDINATION AND STANDARDIZATION OF SPEECH DATABASES AND ASSESSMENT TECHNIQUES (COCOSDA), 2014,
[42] Relationship classification based on dependency parsing and the pretraining model
Yin, Baosheng
Sun, Yifei
SOFT COMPUTING, 2022, 26 (17) : 8575 - 8583
[43] Transition-Based Parsing for Deep Dependency Structures
Zhang, Xun
Du, Yantao
Sun, Weiwei
Wan, Xiaojun
COMPUTATIONAL LINGUISTICS, 2016, 42 (03) : 353 - 389
[44] Research on Dependency Parsing based on Optimized Neural Networks
Tang, Zhong
Liu, Shanshan
5TH INTERNATIONAL CONFERENCE ON ALGORITHMS, COMPUTING AND SYSTEMS, ICACS 2021, 2021, : 33 - 38
[45] Undirected Dependency Structures for Transition-Based Parsing
Gomez-Rodriguez, Carlos
Fernandez-Gonzalez, Daniel
PROCESAMIENTO DEL LENGUAJE NATURAL, 2012, (48): : 43 - 50
[46] Sarcasm detection based on BERT and attention mechanism
Meng, Jiana
Zhu, Yanlin
Sun, Shichang
Zhao, Dandan
MULTIMEDIA TOOLS AND APPLICATIONS, 2024, 83 (10) : 29159 - 29178
[47] Sarcasm detection based on BERT and attention mechanism
Jiana Meng
Yanlin Zhu
Shichang Sun
Dandan Zhao
Multimedia Tools and Applications, 2024, 83 : 29159 - 29178
[48] Leverage External Knowledge and Self-attention for Chinese Semantic Dependency Graph Parsing
Liu, Dianqing
Zhang, Lanqiu
Shao, Yanqiu
Sun, Junzhao
INTELLIGENT AUTOMATION AND SOFT COMPUTING, 2021, 28 (02): : 447 - 458
[49] Dependency to Semantics: Structure Transformation and Syntax-Guided Attention for Neural Semantic Parsing
Wu, Shan
Chen, Bo
Han, Xianpei
Sun, Le
KNOWLEDGE GRAPH AND SEMANTIC COMPUTING: KNOWLEDGE GRAPH EMPOWERS NEW INFRASTRUCTURE CONSTRUCTION, 2021, 1466 : 119 - 133
[50] Dependency-based BERT for Chinese Event Argument Extraction
Li, Daiyi
Yan, Li
Ma, Zongmin
ACM TRANSACTIONS ON ASIAN AND LOW-RESOURCE LANGUAGE INFORMATION PROCESSING, 2023, 22 (12)

← 1 2 3 4 5 →