The Impact of Synchronization in Parallel Stochastic Gradient Descent

被引：0
|
作者：
Backstrom, Karl [1 ]
Papatriantafilou, Marina [1 ]
Tsigas, Philippas [1 ]
机构：
[1] Chalmers Univ Technol, Dept Comp Sci & Engn, Gothenburg, Sweden
来源：
DISTRIBUTED COMPUTING AND INTELLIGENT TECHNOLOGY, ICDCIT 2022 | 2022年 / 13145卷
基金：
瑞典研究理事会;
关键词：
Stochastic gradient descent; Lock-free; Machine Learning;
D O I：
10.1007/978-3-030-94876-4_4
中图分类号：
TP31 [计算机软件];
学科分类号：
081202 ; 0835 ;
摘要：
In this paper, we discuss our and related work in the domain of efficient parallel optimization, using Stochastic Gradient Descent, for fast and stable convergence in prominent machine learning applications. We outline the results in the context of aspects and challenges regarding synchronization, consistency, staleness and parallel-aware adaptiveness, focusing on the impact on the overall convergence.
引用
收藏
页码：60 / 75
页数：16
相关论文
共 50 条
  • [1] Performance Optimization on Model Synchronization in Parallel Stochastic Gradient Descent Based SVM
    Abeykoon, Vibhatha
    Fox, Geoffrey
    Kim, Minje
    [J]. 2019 19TH IEEE/ACM INTERNATIONAL SYMPOSIUM ON CLUSTER, CLOUD AND GRID COMPUTING (CCGRID), 2019, : 508 - 517
  • [2] Asynchronous Decentralized Parallel Stochastic Gradient Descent
    Lian, Xiangru
    Zhang, Wei
    Zhang, Ce
    Liu, Ji
    [J]. INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 80, 2018, 80
  • [3] Fast Convergence Stochastic Parallel Gradient Descent Algorithm
    Hu Dongting
    Shen Wen
    Ma Wenchao
    Liu Xinyu
    Su Zhouping
    Zhu Huaxin
    Zhang Xiumei
    Que Lizhi
    Zhu Zhuowei
    Zhang Yixin
    Chen Guoqing
    Hu Lifa
    [J]. LASER & OPTOELECTRONICS PROGRESS, 2019, 56 (12)
  • [4] Test of the stochastic parallel gradient descent algorithm in laboratory experiments
    Banakh V.A.
    Larichev A.V.
    Razenkov I.A.
    Shesternin A.N.
    [J]. Atmospheric and Oceanic Optics, 2013, 26 (4) : 337 - 344
  • [5] Wavefront error correction with stochastic parallel gradient descent algorithm
    Liu Jiaguo
    Li Lin
    Hu Xinqi
    Yu Xin
    Zhao Lei
    [J]. OPTICAL DESIGN AND TESTING III, PTS 1 AND 2, 2008, 6834
  • [6] Weighted Aggregating Stochastic Gradient Descent for Parallel Deep Learning
    Guo, Pengzhan
    Ye, Zeyang
    Xiao, Keli
    Zhu, Wei
    [J]. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, 2022, 34 (10) : 5037 - 5050
  • [7] High Performance Parallel Stochastic Gradient Descent in Shared Memory
    Sallinen, Scott
    Satish, Nadathur
    Smelyanskiy, Mikhail
    Sury, Samantika S.
    Re, Christopher
    [J]. 2016 IEEE 30TH INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM (IPDPS 2016), 2016, : 873 - 882
  • [8] Performance of stochastic parallel gradient descent algorithm in coherent combination
    [J]. 1600, Chinese Optical Society (36):
  • [9] Adaptive Gradient Estimation Stochastic Parallel Gradient Descent Algorithm for Laser Beam Cleanup
    Ma, Shiqing
    Yang, Ping
    Lai, Boheng
    Su, Chunxuan
    Zhao, Wang
    Yang, Kangjian
    Jin, Ruiyan
    Cheng, Tao
    Xu, Bing
    [J]. PHOTONICS, 2021, 8 (05)
  • [10] Adaptive stochastic parallel gradient descent approach for efficient fiber coupling
    Hu, Qintao
    Zhen, Liangli
    Yao, Mao
    Zhu, Shiwei
    Zhou, Xi
    Zhou, Guozhong
    [J]. OPTICS EXPRESS, 2020, 28 (09) : 13141 - 13154
12345