RelativeNAS: Relative Neural Architecture Search via Slow-Fast Learning

被引:29
|
作者
Tan, Hao [1 ]
Cheng, Ran [1 ]
Huang, Shihua [1 ]
He, Cheng [1 ]
Qiu, Changxiao [2 ]
Yang, Fan [2 ]
Luo, Ping [3 ]
机构
[1] Southern Univ Sci & Technol, Univ Key Lab Evolving Intelligent Syst Guangdong, Dept Comp Sci & Engn, Shenzhen 518055, Peoples R China
[2] Huawei Technol Co Ltd, Hisilicon Res Dept, Shenzhen 518055, Peoples R China
[3] Univ Hong Kong, Dept Comp Sci, Hong Kong, Peoples R China
基金
中国国家自然科学基金;
关键词
Computer architecture; Statistics; Sociology; Search problems; Optimization; Neural networks; Estimation; AutoML; convolutional neural network (CNN); neural architecture search (NAS); population-based search; slow-fast learning; NETWORKS;
D O I
10.1109/TNNLS.2021.3096658
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Despite the remarkable successes of convolutional neural networks (CNNs) in computer vision, it is time-consuming and error-prone to manually design a CNN. Among various neural architecture search (NAS) methods that are motivated to automate designs of high-performance CNNs, the differentiable NAS and population-based NAS are attracting increasing interests due to their unique characters. To benefit from the merits while overcoming the deficiencies of both, this work proposes a novel NAS method, RelativeNAS. As the key to efficient search, RelativeNAS performs joint learning between fast learners (i.e., decoded networks with relatively lower loss value) and slow learners in a pairwise manner. Moreover, since RelativeNAS only requires low-fidelity performance estimation to distinguish each pair of fast learner and slow learner, it saves certain computation costs for training the candidate architectures. The proposed RelativeNAS brings several unique advantages: 1) it achieves state-of-the-art performances on ImageNet with top-1 error rate of 24.88%, that is, outperforming DARTS and AmoebaNet-B by 1.82% and 1.12%, respectively; 2) it spends only 9 h with a single 1080Ti GPU to obtain the discovered cells, that is, 3.75x and 7875x faster than DARTS and AmoebaNet, respectively; and 3) it provides that the discovered cells obtained on CIFAR-10 can be directly transferred to object detection, semantic segmentation, and keypoint detection, yielding competitive results of 73.1% mAP on PASCAL VOC, 78.7% mIoU on Cityscapes, and 68.5% AP on MSCOCO, respectively. The implementation of RelativeNAS is available at https://github.com/EMI-Group/RelativeNAS.
引用
收藏
页码:475 / 489
页数:15
相关论文
共 50 条
  • [31] NASE: Learning Knowledge Graph Embedding for Link Prediction via Neural Architecture Search
    Kou, Xiaoyu
    Luo, Bingfeng
    Hu, Huang
    Zhang, Yan
    CIKM '20: PROCEEDINGS OF THE 29TH ACM INTERNATIONAL CONFERENCE ON INFORMATION & KNOWLEDGE MANAGEMENT, 2020, : 2089 - 2092
  • [32] Generalizable Reconstruction for Accelerating MR Imaging via Federated Learning With Neural Architecture Search
    Wu, Ruoyou
    Li, Cheng
    Zou, Juan
    Liu, Xinfeng
    Zheng, Hairong
    Wang, Shanshan
    IEEE TRANSACTIONS ON MEDICAL IMAGING, 2025, 44 (01) : 106 - 117
  • [33] Neural Network Design: Learning from Neural Architecture Search
    van Stein, Bas
    Wang, Hao
    Back, Thomas
    2020 IEEE SYMPOSIUM SERIES ON COMPUTATIONAL INTELLIGENCE (SSCI), 2020, : 1341 - 1349
  • [34] Large deviation principle for slow-fast rough differential equations via controlled rough paths
    Yang, Xiaoyu
    Xu, Yong
    PROCEEDINGS OF THE ROYAL SOCIETY OF EDINBURGH SECTION A-MATHEMATICS, 2025,
  • [35] Fast architecture, slow architecture: learning through cross-curricular workshops
    Cabrero-Olmos, Raquel
    CUADERNOS DE PROYECTOS ARQUITECTONICOS, 2019, (09): : 74 - 85
  • [36] Proxy Data Generation for Fast and Efficient Neural Architecture Search
    Minje Park
    Journal of Electrical Engineering & Technology, 2023, 18 : 2307 - 2316
  • [37] SqueezeNAS: Fast Neural Architecture Search for Faster Semantic Segmentation
    Shaw, Albert
    Hunter, Daniel
    Iandola, Forrest
    Sidhu, Sammy
    2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION WORKSHOPS (ICCVW), 2019, : 2014 - 2024
  • [38] LIGHTSPEECH: LIGHTWEIGHT AND FAST TEXT TO SPEECH WITH NEURAL ARCHITECTURE SEARCH
    Luo, Renqian
    Tan, Xu
    Wang, Rui
    Qin, Tao
    Li, Jinzhu
    Zhao, Sheng
    Chen, Enhong
    Liu, Tie-Yan
    2021 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP 2021), 2021, : 5699 - 5703
  • [39] Efficient Neural Architecture Search via Proximal Iterations
    Yao, Quanming
    Xu, Ju
    Tu, Wei-Wei
    Zhu, Zhanxing
    THIRTY-FOURTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, THE THIRTY-SECOND INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE AND THE TENTH AAAI SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2020, 34 : 6664 - 6671
  • [40] Accelerating Neural Architecture Search via Proxy Data
    Na, Byunggook
    Mok, Jisoo
    Choe, Hyeokjun
    Yoon, Sungroh
    PROCEEDINGS OF THE THIRTIETH INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, IJCAI 2021, 2021, : 2848 - 2854