Deep Neural Networks Pruning via the Structured Perspective Regularization

被引:0
|
作者
Cacciola, Matteo [1 ]
Frangioni, Antonio [2 ]
Li, Xinlin [3 ]
Lodi, Andrea [1 ,4 ]
机构
[1] Polytech Montreal, CERC, Montreal, PQ, Canada
[2] Univ Pisa, Pisa, PI, Italy
[3] Huawei Montreal Res Ctr, Montreal, PQ, Canada
[4] Cornell Tech & Technion IIT, Jacobs Technion Cornell Inst, New York, NY 10044 USA
来源
基金
加拿大自然科学与工程研究理事会;
关键词
pruning; artificial neural networks; optimization;
D O I
10.1137/22M1542313
中图分类号
O29 [应用数学];
学科分类号
070104 ;
摘要
In machine learning, artificial neural networks (ANNs) are a very powerful tool, broadly used in many applications. Often, the selected (deep) architectures include many layers, and therefore a large number of parameters, which makes training, storage, and inference expensive. This motivated a stream of research about compressing the original networks into smaller ones without excessively sacrificing performances. Among the many proposed compression approaches, one of the most popular is pruning, whereby entire elements of the ANN (links, nodes, channels,...) and the corresponding weights are deleted. Since the nature of the problem is inherently combinatorial (what elements to prune and what not), we propose a new pruning method based on operational research tools. We start from a natural mixed-integer-programming model for the problem, and we use the perspective reformulation technique to strengthen its continuous relaxation. Projecting away the indicator variables from this reformulation yields a new regularization term, which we call the structured perspective regularization, that leads to structured pruning of the initial architecture. We test our method on some ResNet architectures applied to CIFAR-10, CIFAR-100, and ImageNet datasets, obtaining competitive performances w.r.t. the state of the art for structured pruning.
引用
收藏
页码:1051 / 1077
页数:27
相关论文
共 50 条
  • [1] Structured Pruning for Deep Convolutional Neural Networks via Adaptive Sparsity Regularization
    Shao, Tuanjie
    Shin, Dongkun
    [J]. 2022 IEEE 46TH ANNUAL COMPUTERS, SOFTWARE, AND APPLICATIONS CONFERENCE (COMPSAC 2022), 2022, : 982 - 987
  • [2] Structured Pruning for Efficient Convolutional Neural Networks via Incremental Regularization
    Wang, Huan
    Hu, Xinyi
    Zhang, Qiming
    Wang, Yuehai
    Yu, Lu
    Hu, Haoji
    [J]. IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, 2020, 14 (04) : 775 - 788
  • [3] Structured Pruning of Convolutional Neural Networks via L1 Regularization
    Yang, Chen
    Yang, Zhenghong
    Khattak, Abdul Mateen
    Yang, Liu
    Zhang, Wenxin
    Gao, Wanlin
    Wang, Minjuan
    [J]. IEEE ACCESS, 2019, 7 : 106385 - 106394
  • [4] Structured Pruning of Deep Convolutional Neural Networks
    Anwar, Sajid
    Hwang, Kyuyeon
    Sung, Wonyong
    [J]. ACM JOURNAL ON EMERGING TECHNOLOGIES IN COMPUTING SYSTEMS, 2017, 13 (03)
  • [5] Structured Pruning of Neural Networks with Budget-Aware Regularization
    Lemaire, Carl
    Achkar, Andrew
    Jodoin, Pierre-Marc
    [J]. 2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019), 2019, : 9100 - 9108
  • [6] Automatic Pruning Rate Derivation for Structured Pruning of Deep Neural Networks
    Sakai, Yasufumi
    Iwakawa, Akinori
    Tabaru, Tsuguchika
    Inoue, Atsuki
    Kawaguchi, Hiroshi
    [J]. 2022 26TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), 2022, : 2561 - 2567
  • [7] Structured Pruning for Deep Convolutional Neural Networks: A Survey
    He, Yang
    Xiao, Lingao
    [J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2024, 46 (05) : 2900 - 2919
  • [8] Deep neural networks regularization for structured output prediction
    Belharbi, Soufiane
    Herault, Romain
    Chatelain, Clement
    Adam, Sebastien
    [J]. NEUROCOMPUTING, 2018, 281 : 169 - 177
  • [9] Harmonious Coexistence of Structured Weight Pruning and Ternarization for Deep Neural Networks
    Yang, Li
    He, Zhezhi
    Fan, Deliang
    [J]. THIRTY-FOURTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, THE THIRTY-SECOND INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE AND THE TENTH AAAI SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2020, 34 : 6623 - 6630
  • [10] Towards Robustness of Deep Neural Networks via Regularization
    Li, Yao
    Min, Martin Renqiang
    Lee, Thomas
    Yu, Wenchao
    Kruus, Erik
    Wang, Wei
    Hsieh, Cho-Jui
    [J]. 2021 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2021), 2021, : 7476 - 7485