user

Complexity control by gradient descent in deep networks

被引：0
|
作者：
Tomaso Poggio
Qianli Liao
Andrzej Banburski
机构：
[1] MIT,Center for Brains, Minds, and Machines
来源：
Nature Communications | / 11卷
关键词：
D O I：
暂无
中图分类号：
学科分类号：
摘要：
Overparametrized deep networks predict well, despite the lack of an explicit complexity control during training, such as an explicit regularization term. For exponential-type loss functions, we solve this puzzle by showing an effective regularization effect of gradient descent in terms of the normalized weights that are relevant for classification.
引用
收藏
相关论文
共 50 条
  • [1] Complexity control by gradient descent in deep networks
    Poggio, Tomaso
    Liao, Qianli
    Banburski, Andrzej
    [J]. NATURE COMMUNICATIONS, 2020, 11 (01)
  • [2] Learning dynamics of gradient descent optimization in deep neural networks
    Wu, Wei
    Jing, Xiaoyuan
    Du, Wencai
    Chen, Guoliang
    [J]. SCIENCE CHINA-INFORMATION SCIENCES, 2021, 64 (05)
  • [3] Image classification with deep belief networks and improved gradient descent
    Liu, Gang
    Xiao, Liang
    Xiong, Caiquan
    [J]. 2017 IEEE INTERNATIONAL CONFERENCE ON COMPUTATIONAL SCIENCE AND ENGINEERING (CSE) AND IEEE/IFIP INTERNATIONAL CONFERENCE ON EMBEDDED AND UBIQUITOUS COMPUTING (EUC), VOL 1, 2017, : 375 - 380
  • [4] Gradient Descent Finds Global Minima of Deep Neural Networks
    Du, Simon S.
    Lee, Jason D.
    Li, Haochuan
    Wang, Liwei
    Zhai, Xiyu
    [J]. INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 97, 2019, 97
  • [5] Global Convergence of Gradient Descent for Deep Linear Residual Networks
    Wu, Lei
    Wang, Qingcan
    Ma, Chao
    [J]. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019), 2019, 32
  • [6] Learning dynamics of gradient descent optimization in deep neural networks
    Wei Wu
    Xiaoyuan Jing
    Wencai Du
    Guoliang Chen
    [J]. Science China Information Sciences, 2021, 64
  • [7] Gradient Descent Analysis: On Visualizing the Training of Deep Neural Networks
    Becker, Martin
    Lippel, Jens
    Zielke, Thomas
    [J]. PROCEEDINGS OF THE 14TH INTERNATIONAL JOINT CONFERENCE ON COMPUTER VISION, IMAGING AND COMPUTER GRAPHICS THEORY AND APPLICATIONS - VOL 3: IVAPP, 2019, : 338 - 345
  • [8] Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks
    Cui, Xiaodong
    Zhang, Wei
    Tuske, Zoltan
    Picheny, Michael
    [J]. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 31 (NIPS 2018), 2018, 31
  • [9] Learning dynamics of gradient descent optimization in deep neural networks
    Wei WU
    Xiaoyuan JING
    Wencai DU
    Guoliang CHEN
    [J]. Science China(Information Sciences), 2021, 64 (05) : 17 - 31
  • [10] Complexity of gradient descent for multiobjective optimization
    Fliege, J.
    Vaz, A. I. F.
    Vicente, L. N.
    [J]. OPTIMIZATION METHODS & SOFTWARE, 2019, 34 (05): : 949 - 959
12345