mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences

被引：0

作者：

Uthus, David ^{[1
]}

Ontanion, Santiago ^{[1
]}

Ainslie, Joshua ^{[1
]}

Guo, Mandy ^{[1
]}

机构：

[1] Google Res, Mountain View, CA 94043 USA

来源：

FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (EMNLP 2023) | 2023年

关键词：

D O I：

暂无

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

We present our work on developing a multilingual, efficient text-to-text transformer that is suitable for handling long inputs. This model, called mLongT5, builds upon the architecture of LongT5, while leveraging the multilingual datasets used for pretraining mT5 and the pretraining tasks of UL2. We evaluate this model on a variety of multilingual summarization and question-answering tasks, and the results show stronger performance for mLongT5 when compared to existing multilingual models such as mBART or M-BERT.

引用

页码：9380 / 9386

页数：7

共 50 条

[41] Vision Transformer for Fast and Efficient Scene Text Recognition
Atienza, Rowel
DOCUMENT ANALYSIS AND RECOGNITION - ICDAR 2021, PT I, 2021, 12821 : 319 - 334
[42] ACT: an Attentive Convolutional Transformer for Efficient Text Classification
Li, Pengfei
Zhong, Peixiang
Mao, Kezhi
Wang, Dongzhe
Yang, Xuefeng
Liu, Yunfeng
Yin, Jianxiong
See, Simon
THIRTY-FIFTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, THIRTY-THIRD CONFERENCE ON INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE AND THE ELEVENTH SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2021, 35 : 13261 - 13269
[43] Fine-Tuning Transformer Models Using Transfer Learning for Multilingual Threatening Text Identification
Rehan, Muhammad
Malik, Muhammad Shahid Iqbal
Jamjoom, Mona Mamdouh
IEEE ACCESS, 2023, 11 : 106503 - 106515
[44] Assessing the Effectiveness of Multilingual Transformer-based Text Embeddings for Named Entity Recognition in Portuguese
de Lima Santos, Diego Bernardes
de Carvalho Dutra, Frederico Giffoni
Parreiras, Fernando Silva
Brandao, Wladmir Cardoso
PROCEEDINGS OF THE 23RD INTERNATIONAL CONFERENCE ON ENTERPRISE INFORMATION SYSTEMS (ICEIS 2021), VOL 1, 2021, : 473 - 483
[45] Efficient Text-to-Code Retrieval with Cascaded Fast and Slow Transformer Models
Gotmare, Akhilesh Deepak
Li, Junnan
Joty, Shafiq
Hoi, Steven C. H.
PROCEEDINGS OF THE 31ST ACM JOINT MEETING EUROPEAN SOFTWARE ENGINEERING CONFERENCE AND SYMPOSIUM ON THE FOUNDATIONS OF SOFTWARE ENGINEERING, ESEC/FSE 2023, 2023, : 388 - 400
[46] ET-DM: Text to image via diffusion model with efficient Transformer
Li, Huan
Xu, Feng
Lin, Zheng
DISPLAYS, 2023, 80
[47] Efficient Processing of Long Sequence Text Data in Transformer: An Examination of Five Different Approaches
Jia, Zihao
Lee, Philseok
ORGANIZATIONAL RESEARCH METHODS, 2025,
[48] An Efficient Text-Independent Speaker Identification Using Feature Fusion and Transformer Model
Khan, Arfat Ahmad
Jahangir, Rashid
Alroobaea, Roobaea
Alyahyan, Saleh Yahya
Almulhi, Ahmed H.
Alsafyani, Majed
Wechtaisong, Chitapong
CMC-COMPUTERS MATERIALS & CONTINUA, 2023, 75 (02): : 4085 - 4100
[49] CRET: Cross-Modal Retrieval Transformer for Efficient Text-Video Retrieval
Ji, Kaixiang
Liu, Jiajia
Hong, Weixiang
Zhong, Liheng
Wang, Jian
Chen, Jingdong
Chu, Wei
PROCEEDINGS OF THE 45TH INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL (SIGIR '22), 2022, : 949 - 959
[50] When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants
Diwan, Anuj
Choi, Eunsol
Harwath, David
61ST CONFERENCE OF THE THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, ACL 2023, VOL 2, 2023, : 1639 - 1650

← 1 2 3 4 5 →