Reinforcement of BERT with Dependency-Parsing Based Attention Mask

被引：0

作者：

Mechouma, Toufik ^{[1
]}

Biskri, Ismail ^{[2
]}

Meunier, Jean Guy ^{[1
]}

机构：

[1] Univ Quebec Montreal, Montreal, PQ, Canada

[2] Univ Quebec Trois Rivieres, Trois Rivieres, PQ, Canada

来源：

ADVANCES IN COMPUTATIONAL COLLECTIVE INTELLIGENCE, ICCCI 2022 | 2022年 / 1653卷

关键词：

Bert; Transformers; Attention mechanisms; Dependency parsing;

D O I：

10.1007/978-3-031-16210-7_9

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Dot-Product based attention mechanism is among recent attention mechanisms. It showed an outstanding performance with BERT. In this paper, we propose a dependency-parsing mask to reinforce the padding mask, at the multi-head attention units. Padding mask, is already used to filter padding positions. The proposed mask, aims to improve BERT attention filter. The conducted experiments, show that BERT performs better with the proposed mask.

引用

页码：112 / 122

页数：11

共 50 条

[1] Dependency parsing of biomedical text with BERT
Jenna Kanerva
Filip Ginter
Sampo Pyysalo
BMC Bioinformatics, 21
[2] Dependency parsing of biomedical text with BERT
Kanerva, Jenna
Ginter, Filip
Pyysalo, Sampo
BMC BIOINFORMATICS, 2020, 21 (Suppl 23)
[3] BERT-Based Sequence Labelling Approach for Dependency Parsing in Tamil
Kumar, C. S. Ayush
Das Maharana, Advaith
Krishnan, Srinath Murali
Premjith, B.
Soman, K. P.
PROCEEDINGS OF THE SECOND WORKSHOP ON SPEECH AND LANGUAGE TECHNOLOGIES FOR DRAVIDIAN LANGUAGES (DRAVIDIANLANGTECH 2022), 2022, : 1 - 8
[4] Learning the Optimal use of Dependency-parsing Information for Finding Translations with Comparable Corpora
Department of Computer Science, University of Tokyo, Japan
不详
Proc. Annu. Meet. Assoc. Comput Linguist., (10-18):
[5] D-BERT: Incorporating dependency-based attention into BERT for relation extraction
Huang, Yuan
Li, Zhixing
Deng, Wei
Wang, Guoyin
Lin, Zhimin
CAAI TRANSACTIONS ON INTELLIGENCE TECHNOLOGY, 2021, 6 (04) : 417 - 425
[6] MABERT: Mask-Attention-Based BERT for Chinese Event Extraction
Ding, Ling
Chen, Xiaojun
Wei, Jian
Xiang, Yang
ACM TRANSACTIONS ON ASIAN AND LOW-RESOURCE LANGUAGE INFORMATION PROCESSING, 2023, 22 (07)
[7] RNN-Based Sequence-Preserved Attention for Dependency Parsing
Zhou, Yi
Zhou, Junying
Liu, Lu
Feng, Jiangtao
Peng, Haoyuan
Zheng, Xiaoqing
THIRTY-SECOND AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE / THIRTIETH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE / EIGHTH AAAI SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2018, : 5738 - 5745
[8] Attention-Based Belief or Disbelief Feature Extraction for Dependency Parsing
Peng, Haoyuan
Liu, Lu
Zhou, Yi
Zhou, Junying
Zheng, Xiaoqing
THIRTY-SECOND AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE / THIRTIETH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE / EIGHTH AAAI SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2018, : 5382 - 5389
[9] Dependency Parsing with Backtracking using Deep Reinforcement Learning
Dary, Franck
Petit, Maxime
Nasr, Alexis
TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 2022, 10 : 888 - 903
[10] SHORT-TEXT QUESTION CLASSIFICATION BASED ON DEPENDENCY PARSING AND ATTENTION MECHANISM
Fang, An
PROCEEDINGS OF 2019 INTERNATIONAL CONFERENCE ON MACHINE LEARNING AND CYBERNETICS (ICMLC), 2019, : 498 - 503

← 1 2 3 4 5 →