Sanity Checks for Lottery Tickets: Does Your Winning Ticket ReallyWin the Jackpot?

被引：0

作者：

Ma, Xiaolong ^{[1
]}

Yuan, Geng ^{[1
]}

Shen, Xuan ^{[1
]}

Chen, Tianlong ^{[2
]}

Chen, Xuxi ^{[2
]}

Chen, Xiaohan ^{[2
]}

Liu, Ning ^{[3
]}

Qin, Minghai

Liu, Sijia ^{[4
]}

Wang, Zhangyang ^{[2
]}

Wang, Yanzhi ^{[1
]}

机构：

[1] Northeastern Univ, Boston, MA 02115 USA

[2] Univ Texas Austin, Austin, TX 78712 USA

[3] Midea Grp, Foshan, Peoples R China

[4] Michigan State Univ, E Lansing, MI 48824 USA

来源：

ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 34 (NEURIPS 2021) | 2021年 / 34卷

基金：

美国国家科学基金会;

关键词：

D O I：

暂无

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

There have been long-standing controversies and inconsistencies over the experiment setup and criteria for identifying the "winning ticket" in literature. To reconcile such, we revisit the definition of lottery ticket hypothesis, with comprehensive and more rigorous conditions. Under our new definition, we show concrete evidence to clarify whether the winning ticket exists across the major DNN architectures and/or applications. Through extensive experiments, we perform quantitative analysis on the correlations between winning tickets and various experimental factors, and empirically study the patterns of our observations. We find that the key training hyperparameters, such as learning rate and training epochs, as well as the architecture characteristics such as capacities and residual connections, are all highly correlated with whether and when the winning tickets can be identified. Based on our analysis, we summarize a guideline for parameter settings in regards of specific architecture characteristics, which we hope to catalyze the research progress on the topic of lottery ticket hypothesis. Our codes are publicly available at: https://github.com/boone891214/sanity-check-LTH.

引用

页数：12