Pipeline Parallelism With Elastic Averaging

被引：0
|
作者：
Jang, Bongwon [1 ]
Yoo, In-Chul [1 ]
Yook, Dongsuk [1 ]
机构：
[1] Korea University, Artificial Intelligence Laboratory, Department of Computer Science and Engineering, Seoul,02841, Korea, Republic of
来源：
IEEE Access | 2024年 / 12卷
关键词：
To accelerate the training speed of massive DNN models on large-scale datasets; distributed training techniques; including data parallelism and model parallelism; have been extensively studied. In particular; pipeline parallelism; which is derived from model parallelism; has been attracting attention. It splits the model parameters across multiple computing nodes and executes multiple mini-batches simultaneously. However; naive pipeline parallelism suffers from the issues of weight inconsistency and delayed gradients; as the model parameters used in the forward and backward passes do not match; causing unstable training and low performance. In this study; we propose a novel pipeline parallelism technique called EA-Pipe to address the weight inconsistency and delayed gradient problems. EA-Pipe applies an elastic averaging method; which has been studied in the context of data parallelism; to pipeline parallelism. The proposed method maintains multiple model replicas to solve the weight inconsistency problem; and synchronizes the model replicas using an elasticity-based moving average method to mitigate the delayed gradient problem. To verify the efficacy of the proposed method; we conducted three image classification experiments on the CIFAR-10/100 and ImageNet datasets. The experimental results show that EA-Pipe not only accelerates training speed but also demonstrates more stable learning property compared to existing pipeline parallelism techniques. Especially; in the experiments using the CIFAR-100 and ImageNet datasets; EA-Pipe recorded error rates that were 2.58% and 2.19% lower; respectively; than the baseline pipeline parallelization method. © 2013 IEEE;
D O I：
暂无
中图分类号：
学科分类号：
摘要：
引用
收藏
页码：5477 / 5489
相关论文
共 50 条
  • [31] Restoration of Legacy Parallelism: Transforming Pthreads into Farm and Pipeline Patterns
    Vladimir Janjic
    Christopher Brown
    Adam D. Barwell
    International Journal of Parallel Programming, 2021, 49 : 886 - 910
  • [32] Towards fully adaptive pipeline parallelism for heterogeneous distributed environments
    Gonzalez-Velez, Horacio
    Cole, Murray
    PARALLEL AND DISTRIBUTED PROCESSING AND APPLICATIONS, 2006, 4330 : 916 - +
  • [33] A Language-Based Tuning Mechanism for Task and Pipeline Parallelism
    Otto, Frank
    Schaefer, Christoph A.
    Dempe, Matthias
    Tichy, Walter F.
    EURO-PAR 2010 - PARALLEL PROCESSING, PART II, 2010, 6272 : 328 - 340
  • [34] Hydrodynamics of flow in an elastic pipeline
    Volobuev A.N.
    Tolstonogov A.P.
    Journal of Engineering Physics and Thermophysics, 2004, 77 (5) : 972 - 978
  • [35] Study of vibrations in an elastic pipeline
    Volobuev, A.N.
    Tolstonogov, A.P.
    Izvestiya Vysshikh Uchebnykh Zavedenij. Aviatsionnaya Tekhnika, 2004, (02): : 47 - 50
  • [36] Shock waves in an elastic pipeline
    Volobuev A.N.
    Tolstonogov A.P.
    Journal of Engineering Physics and Thermophysics, 2008, 81 (03) : 513 - 519
  • [37] Unleashing Parallelism in Elastic Circuits with Faster Token Delivery
    Elakhras, Ayatallah
    Guerrieri, Andrea
    Josipovic, Lana
    Ienne, Paolo
    2022 32ND INTERNATIONAL CONFERENCE ON FIELD-PROGRAMMABLE LOGIC AND APPLICATIONS, FPL, 2022, : 253 - 261
  • [38] Improving Utilization and Parallelism of Hadoop Cluster by Elastic Containers
    Xu, Yinggen
    Chen, Wei
    Wang, Shaoqi
    Zhou, Xiaobo
    Jiang, Changjun
    IEEE CONFERENCE ON COMPUTER COMMUNICATIONS (IEEE INFOCOM 2018), 2018, : 180 - 188
  • [39] Towards Efficient Elastic Parallelism for Deep Learning Processor
    Cheng, Jinyu
    Qian, Ruyi
    Shi, Qinwen
    Hu, Gaomei
    Ciao, Mengjuan
    Huo, Qirun
    Xu, Yuanchao
    2022 IEEE INTL CONF ON PARALLEL & DISTRIBUTED PROCESSING WITH APPLICATIONS, BIG DATA & CLOUD COMPUTING, SUSTAINABLE COMPUTING & COMMUNICATIONS, SOCIAL COMPUTING & NETWORKING, ISPA/BDCLOUD/SOCIALCOM/SUSTAINCOM, 2022, : 363 - 370
  • [40] Averaging of elastic and shrinkage properties for viscoelastic composites
    Orlik, J
    PROGRESS AND TRENDS IN RHEOLOGY V, 1998, : 439 - 440
12345