mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences

被引：0

作者：

Uthus, David ^{[1
]}

Ontanion, Santiago ^{[1
]}

Ainslie, Joshua ^{[1
]}

Guo, Mandy ^{[1
]}

机构：

[1] Google Res, Mountain View, CA 94043 USA

来源：

FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (EMNLP 2023) | 2023年

关键词：

D O I：

暂无

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

We present our work on developing a multilingual, efficient text-to-text transformer that is suitable for handling long inputs. This model, called mLongT5, builds upon the architecture of LongT5, while leveraging the multilingual datasets used for pretraining mT5 and the pretraining tasks of UL2. We evaluate this model on a variety of multilingual summarization and question-answering tasks, and the results show stronger performance for mLongT5 when compared to existing multilingual models such as mBART or M-BERT.

引用

页码：9380 / 9386

页数：7

共 50 条

[21] Combating Phishing in the Age of Fake News: A Novel Approach with Text-to-Text Transfer Transformer
Ma, Yifeng
Dobbie, Gillian
Arachchilage, Nalin
PROCEEDINGS OF THE 1ST WORKSHOP ON SECURITY-CENTRIC STRATEGIES FOR COMBATING INFORMATION DISORDER, SCID 2024, 2024,
[22] Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
Mastropaolo, Antonio
Scalabrimo, Simone
Cooper, Nathan
Palacio, David Nader
Poshyvanyk, Denys
Oliveto, Rocco
Bavota, Gabriele
2021 IEEE/ACM 43RD INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING (ICSE 2021), 2021, : 336 - 347
[23] T5G2P: Text-to-Text Transfer Transformer Based Grapheme-to-Phoneme Conversion
Rezackova, Marketa
Tihelka, Daniel
Matousek, Jindrich
IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2024, 32 : 3466 - 3476
[24] T5G2P: Using Text-to-Text Transfer Transformer for Grapheme-to-Phoneme Conversion
Rezackova, Marketa
Svec, Jan
Tihelka, Daniel
INTERSPEECH 2021, 2021, : 6 - 10
[25] Ensemble-NQG-T5: Ensemble Neural Question Generation Model Based on Text-to-Text Transfer Transformer
Hwang, Myeong-Ha
Shin, Jikang
Seo, Hojin
Im, Jeong-Seon
Cho, Hee
Lee, Chun-Kwon
APPLIED SCIENCES-BASEL, 2023, 13 (02):
[26] AraT5: Text-to-Text Transformers for Arabic Language Generation
Nagoudi, El Moatez Billah
Elmadany, AbdelRahim
Abdul-Mageed, Muhammad
PROCEEDINGS OF THE 60TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2022), VOL 1: (LONG PAPERS), 2022, : 628 - 647
[27] Text-To-Text Transfer Transformer Based Method for Generating Startup Scenarios for New Equipment in Power Grids
Tao, Wenbiao
Wang, Liang
Meng, Qingmeng
Li, Rui
Han, Peng
Shi, Yuxin
Shan, Lianfei
Geng, Xiaofei
APPLIED ARTIFICIAL INTELLIGENCE, 2024, 38 (01)
[28] Practical Transformer-based Multilingual Text Classification
Wang, Cindy
Banko, Michele
2021 CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, NAACL-HLT 2021, 2021, : 121 - 129
[29] Bootstrap an End-to-end ASR System by Multilingual Training, Transfer Learning, Text-to-text Mapping and Synthetic Audio
Giollo, Manuel
Gunceler, Deniz
Liu, Yulan
Willett, Daniel
INTERSPEECH 2021, 2021, : 2416 - 2420
[30] End-to-End generation of Multiple-Choice questions using Text-to-Text transfer Transformer models
Rodriguez-Torrealba, Ricardo
Garcia-Lopez, Eva
Garcia-Cabot, Antonio
EXPERT SYSTEMS WITH APPLICATIONS, 2022, 208

← 1 2 3 4 5 →