共 50 条
- [1] Does It Pay to Pay Attention?[J]. REVIEW OF FINANCIAL STUDIES, 2018, 31 (12): : 4595 - 4649Gargano, AntonioRossi, Alberto G.
Pay attention to the hidden semanteme
被引：1
|作者：
Tang, Huanling
[1
,2
,3
]
Liu, Xiaoyan
[1
]
Wang, Yulin
[1
]
Dou, Quansheng
[1
,2
,3
]
Lu, Mingyu
[4
]
机构：
[1] Shandong Technol & Business Univ, Sch Comp Sci & Technol, Yantai 264005, Shandong, Peoples R China
[2] Coinnovat Ctr Shandong Coll & Univ Future Intellig, Yantai 264005, Shandong, Peoples R China
[3] Shandong Technol & Business Univ, Key Lab Intelligent Informat Proc Univ Shandong, Yantai 264005, Shandong, Peoples R China
[4] Dalian Maritime Univ, Informat Sci & Technol Coll, Dalian 116026, Liaoning, Peoples R China
基金：
中国国家自然科学基金;
关键词：
Feature representation;
Attention mechanism;
Deep learning;
Modeling lightly;
Natural language processing;
D O I：
10.1016/j.ins.2023.119076
中图分类号：
TP [自动化技术、计算机技术];
学科分类号：
0812 ;
摘要：
With the capability of modeling lighter, MLP-based models like the pNLP-Mixer and the HyperMixer demonstrate the potential for diverse tasks in NLP. However, these linguistic models are not optimized for the regularity of textual hierarchical abstraction. Here, this paper proposes the hidden bias attention (HBA), a novel attention mechanism that is lighter than self-attention and focuses on extracting hidden (topic) semanteme. Additionally, this paper introduces a series of lightweight deep learning architectures, HBA-Mixer based on HBA and MHBA-Mixers based on multi-head HBA, which both outperforms pNLP-Mixer and HyperMixer in accuracy with fewer parameters on 3 tasks, including text classification, natural language inference, and sentiment analysis. Compared with large pre-trained models, MHBA-Mixers achieve over 90% of their accuracy with one-thousandth of the parameters.
引用
收藏
页数：12
相关论文