Reinforcement of BERT with Dependency-Parsing Based Attention Mask

被引：0

作者：

Mechouma, Toufik ^{[1
]}

Biskri, Ismail ^{[2
]}

Meunier, Jean Guy ^{[1
]}

机构：

[1] Univ Quebec Montreal, Montreal, PQ, Canada

[2] Univ Quebec Trois Rivieres, Trois Rivieres, PQ, Canada

来源：

ADVANCES IN COMPUTATIONAL COLLECTIVE INTELLIGENCE, ICCCI 2022 | 2022年 / 1653卷

关键词：

Bert; Transformers; Attention mechanisms; Dependency parsing;

D O I：

10.1007/978-3-031-16210-7_9

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Dot-Product based attention mechanism is among recent attention mechanisms. It showed an outstanding performance with BERT. In this paper, we propose a dependency-parsing mask to reinforce the padding mask, at the multi-head attention units. Padding mask, is already used to filter padding positions. The proposed mask, aims to improve BERT attention filter. The conducted experiments, show that BERT performs better with the proposed mask.

引用

页码：112 / 122

页数：11

共 50 条

[21] Vietnamese Span-based Constituency Parsing with BERT Embedding
Phan, Thi-Phuong-Uyen
Huynh, Ngoc-Thanh-Tung
Truong, Hung-Thinh
PROCEEDINGS OF 2019 11TH INTERNATIONAL CONFERENCE ON KNOWLEDGE AND SYSTEMS ENGINEERING (KSE 2019), 2019, : 293 - 299
[22] Knowledge enhancement BERT based on domain dictionary mask
Cao, Xianglin
Xiao, Hong
Jiang, Wenchao
JOURNAL OF HIGH SPEED NETWORKS, 2023, 29 (02) : 121 - 128
[23] Improving Graph-Based Dependency Parsing Models With Dependency Language Models
Zhang, Min
Chen, Wenliang
Duan, Xiangyu
Zhang, Rong
IEEE TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2013, 21 (11): : 2313 - 2323
[24] A Burmese Dependency Parsing Method Based on Transfer Learning
Mao, Cunli
Man, Zhibo
Yu, Zhengtao
Wang, Zhenhan
Gao, Shengxiang
Zhang, Yafei
2020 INTERNATIONAL CONFERENCE ON ASIAN LANGUAGE PROCESSING (IALP 2020), 2020, : 92 - 97
[25] Query expansion based on global dependency parsing analysis
Xiang, Qinyuan
Li, Weijiang
Deng, Hui
Wang, Feng
INFORMATION SCIENCE AND MANAGEMENT ENGINEERING, VOLS 1-3, 2014, 46 : 1515 - 1521
[26] Relationship classification based on dependency parsing and the pretraining model
Baosheng Yin
Yifei Sun
Soft Computing, 2022, 26 : 8575 - 8583
[27] Blog sentiment orientation analysis based on dependency parsing
Feng, Shi
Fu, Yongchen
Yang, Feng
Wang, Daling
Zhang, Yifei
Jisuanji Yanjiu yu Fazhan/Computer Research and Development, 2012, 49 (11): : 2395 - 2406
[28] The Chinese Open Relation Extraction Based on Dependency Parsing
Wang Yuzhao
Yang Yunfei
Zhao Ruixue
PROCEEDINGS OF THE 2017 5TH INTERNATIONAL CONFERENCE ON FRONTIERS OF MANUFACTURING SCIENCE AND MEASURING TECHNOLOGY (FMSMT 2017), 2017, 130 : 1212 - 1216
[29] Nucleus Composition in Transition-based Dependency Parsing
Nivre, Joakim
Basirat, Ali
Duerich, Luise
Moss, Adam
COMPUTATIONAL LINGUISTICS, 2022, 48 (04) : 849 - 886
[30] Graph-based Dependency Parsing with Bidirectional LSTM
Wang, Wenhui
Chang, Baobao
PROCEEDINGS OF THE 54TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, VOL 1, 2016, : 2306 - 2315

← 1 2 3 4 5 →