A Comparison of Transformer-Based Language Models on NLP Benchmarks

被引：1

作者：

Greco, Candida Maria ^{[1
]}

Tagarelli, Andrea ^{[1
]}

Zumpano, Ester ^{[1
]}

机构：

[1] Univ Calabria, DIMES, Arcavacata Di Rende, CS, Italy

来源：

NATURAL LANGUAGE PROCESSING AND INFORMATION SYSTEMS (NLDB 2022) | 2022年 / 13286卷

关键词：

BERTology; Deep learning; Benchmarks;

D O I：

10.1007/978-3-031-08473-7_45

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Since the advent of BERT, Transformer-based language models (TLMs) have shown outstanding effectiveness in several NLP tasks. In this paper, we aim at bringing order to the landscape of TLMs and their performance on important benchmarks for NLP. Our analysis sheds light on the advantages that some TLMs take over the others, but also unveils issues in making a complete and fair comparison in some situations.

引用

页码：490 / 501

页数：12

共 50 条

[1] End-to-End Transformer-Based Models in Textual-Based NLP
Rahali, Abir
Akhloufi, Moulay A.
[J]. AI, 2023, 4 (01) : 54 - 110
[2] Performance Evaluation of Transformer-based NLP Models on Fake News Detection Datasets
Babu, Raveen Narendra
Lung, Chung-Horng
Zaman, Marzia
[J]. 2023 IEEE 47TH ANNUAL COMPUTERS, SOFTWARE, AND APPLICATIONS CONFERENCE, COMPSAC, 2023, : 316 - 321
[3] Ouroboros: On Accelerating Training of Transformer-Based Language Models
Yang, Qian
Huo, Zhouyuan
Wang, Wenlin
Huang, Heng
Carin, Lawrence
[J]. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019), 2019, 32
[4] Transformer-Based Language Models for Software Vulnerability Detection
Thapa, Chandra
Jang, Seung Ick
Ahmed, Muhammad Ejaz
Camtepe, Seyit
Pieprzyk, Josef
Nepal, Surya
[J]. PROCEEDINGS OF THE 38TH ANNUAL COMPUTER SECURITY APPLICATIONS CONFERENCE, ACSAC 2022, 2022, : 481 - 496
[5] RadBERT: Adapting Transformer-based Language Models to Radiology
Yan, An
McAuley, Julian
Lu, Xing
Du, Jiang
Chang, Eric Y.
Gentili, Amilcare
Hsu, Chun-Nan
[J]. RADIOLOGY-ARTIFICIAL INTELLIGENCE, 2022, 4 (04)
[6] Applications of transformer-based language models in bioinformatics: a survey
Zhang, Shuang
Fan, Rui
Liu, Yuti
Chen, Shuang
Liu, Qiao
Zeng, Wanwen
[J]. NEURO-ONCOLOGY ADVANCES, 2023, 5 (01)
[7] TAG: Gradient Attack on Transformer-based Language Models
Deng, Jieren
Wang, Yijue
Li, Ji
Wang, Chenghong
Shang, Chao
Liu, Hang
Rajasekaran, Sanguthevar
Ding, Caiwen
[J]. FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, EMNLP 2021, 2021, : 3600 - 3610
[8] AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, Katikapalli Subramanyam
Rajasekharan, Ajit
Sangeetha, Sivanesan
[J]. JOURNAL OF BIOMEDICAL INFORMATICS, 2022, 126
[9] Transformer-based language models for mental health issues: A survey
Greco, Candida M.
Simeri, Andrea
Tagarelli, Andrea
Zumpano, Ester
[J]. PATTERN RECOGNITION LETTERS, 2023, 167 : 204 - 211
[10] Pre-trained transformer-based language models for Sundanese
Wongso, Wilson
Lucky, Henry
Suhartono, Derwin
[J]. JOURNAL OF BIG DATA, 2022, 9 (01)

← 1 2 3 4 5 →