mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences

被引：0

作者：

Uthus, David ^{[1
]}

Ontanion, Santiago ^{[1
]}

Ainslie, Joshua ^{[1
]}

Guo, Mandy ^{[1
]}

机构：

[1] Google Res, Mountain View, CA 94043 USA

来源：

FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (EMNLP 2023) | 2023年

关键词：

D O I：

暂无

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

We present our work on developing a multilingual, efficient text-to-text transformer that is suitable for handling long inputs. This model, called mLongT5, builds upon the architecture of LongT5, while leveraging the multilingual datasets used for pretraining mT5 and the pretraining tasks of UL2. We evaluate this model on a variety of multilingual summarization and question-answering tasks, and the results show stronger performance for mLongT5 when compared to existing multilingual models such as mBART or M-BERT.

引用

页码：9380 / 9386

页数：7

共 50 条

[1] mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, Linting
Constant, Noah
Roberts, Adam
Kale, Mihir
Al-Rfou, Rami
Siddhant, Aditya
Barua, Aditya
Raffel, Colin
2021 CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS: HUMAN LANGUAGE TECHNOLOGIES (NAACL-HLT 2021), 2021, : 483 - 498
[2] mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Chi, Zewen
Dong, Li
Ma, Shuming
Huang, Shaohan
Mao, Xian-Ling
Huang, Heyan
Wei, Furu
2021 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (EMNLP 2021), 2021, : 1671 - 1683
[3] A Text-to-Text Model for Multilingual Offensive Language Identification
Ranasinghe, Tharindu
Zampieri, Marcos
13TH INTERNATIONAL JOINT CONFERENCE ON NATURAL LANGUAGE PROCESSING AND THE 3RD CONFERENCE OF THE ASIA-PACIFIC CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, IJCNLP-AACL 2023, 2023, : 375 - 384
[4] Homograph Disambiguation with Text-to-Text Transfer Transformer
Rezackova, Marketa
Tihelka, Daniel
Matousek, Jindrich
INTERSPEECH 2024, 2024, : 2785 - 2789
[5] ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation
Long Phan
Hieu Tran
Hieu Nguyen
Trinh, Trieu H.
NAACL 2022: THE 2022 CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS: HUMAN LANGUAGE TECHNOLOGIES: PROCEEDINGS OF THE STUDENT RESEARCH WORKSHOP, 2022, : 136 - 142
[6] Adapting Pretrained Text-to-Text Models for Long Text Sequences
Xiong, Wenhan
Gupta, Anchit
Toshniwal, Shubham
Mehdad, Yashar
Yih, Wen-tau
FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS - EMNLP 2023, 2023, : 5566 - 5578
[7] Distractor Generation Through Text-to-Text Transformer Models
de-Fitero-Dominguez, David
Garcia-Lopez, Eva
Garcia-Cabot, Antonio
del-Hoyo-Gabaldon, Jesus-Angel
Moreno-Cediel, Antonio
IEEE ACCESS, 2024, 12 : 25580 - 25589
[8] HaT5: Hate Language Identification using Text-to-Text Transfer Transformer
Sabry, Sana Sabah
Adewumi, Tosin
Abid, Nosheen
Kovacs, Gyorgy
Liwicki, Foteini
Liwicki, Marcus
2022 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN), 2022,
[9] TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer
Berabi, Berkay
He, Jingxuan
Raychev, Veselin
Vechev, Martin
INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 139, 2021, 139
[10] Exploring the limits of transfer learning with a unified text-to-text transformer
Raffel, Colin
Shazeer, Noam
Roberts, Adam
Lee, Katherine
Narang, Sharan
Matena, Michael
Zhou, Yanqi
Li, Wei
Liu, Peter J.
Journal of Machine Learning Research, 2020, 21

← 1 2 3 4 5 →