Optimizing low-rank adaptation with decomposed matrices and adaptive rank allocation

被引：0

作者：

Zhang, Dacao ^{[1
]}

Yang, Fan ^{[1
]}

Zhang, Kun ^{[1
]}

Li, Xin ^{[2
]}

Wei, Si ^{[2
]}

Hong, Richang ^{[1
]}

Wang, Meng ^{[1
]}

机构：

[1] Hefei Univ Technol, Sch Comp Sci Informat Engn, Hefei 230601, Peoples R China

[2] iFLYTEK Co Ltd, Artificial Intelligence Res Inst, Hefei 230088, Peoples R China

来源：

FRONTIERS OF COMPUTER SCIENCE | 2025年 / 19卷 / 05期

基金：

中国国家自然科学基金;

关键词：

D O I：

10.1007/s11704-024-40317-w

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

In this paper, we argue that the same rank setting of LoRA will inhibit its potential. In response, we proposed two novel PEFT strategies to improve the capability of LoRA in single-task and multi-task scenarios. We also conducted extensive experiments to show the effectiveness of our proposed method.

引用

页数：3

共 50 条

[21] Decomposition matrices for low-rank unitary groups
Dudas, Olivier
Malle, Gunter
PROCEEDINGS OF THE LONDON MATHEMATICAL SOCIETY, 2015, 110 : 1517 - 1557
[22] The inertia of the symmetric approximation for low-rank matrices
Casanellas, Marta
Fernandez-Sanchez, Jesus
Garrote-Lopez, Marina
LINEAR & MULTILINEAR ALGEBRA, 2018, 66 (11): : 2349 - 2353
[23] Randomized algorithms for the low-rank approximation of matrices
Liberty, Edo
Woolfe, Franco
Martinsson, Per-Gunnar
Rolchlin, Vladimir
Tyger, Mark
PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA, 2007, 104 (51) : 20167 - 20172
[24] Active Learning and Search on Low-Rank Matrices
Sutherland, Dougal J.
Poczos, Barnabas
Schneider, Jeff
19TH ACM SIGKDD INTERNATIONAL CONFERENCE ON KNOWLEDGE DISCOVERY AND DATA MINING (KDD'13), 2013, : 212 - 220
[25] Structured low-rank approximation for nonlinear matrices
Antonio Fazzi
Numerical Algorithms, 2023, 93 : 1561 - 1580
[26] On best uniform approximation by low-rank matrices
Georgieva, I.
Hofreither, C.
LINEAR ALGEBRA AND ITS APPLICATIONS, 2017, 518 : 159 - 176
[27] ON A PROBLEM OF WEIGHTED LOW-RANK APPROXIMATION OF MATRICES
Dutta, Aritra
Li, Xin
SIAM JOURNAL ON MATRIX ANALYSIS AND APPLICATIONS, 2017, 38 (02) : 530 - 553
[28] Compressive PCA for Low-Rank Matrices on Graphs
Shahid, Nauman
Perraudin, Nathanael
Puy, Gilles
Vandergheynst, Pierre
IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, 2017, 3 (04): : 695 - 710
[29] On the distance to low-rank matrices in the maximum norm
Budzinskiy, Stanislav
LINEAR ALGEBRA AND ITS APPLICATIONS, 2024, 688 : 44 - 58
[30] Local reconstruction of low-rank matrices and subspaces
David, Roee
Goldenberg, Elazar
Krauthgamer, Robert
RANDOM STRUCTURES & ALGORITHMS, 2017, 51 (04) : 607 - 630

← 1 2 3 4 5 →