Performance Modeling for Distributed Training of Convolutional Neural Networks

被引：2

作者：

Castello, Adrian ^{[1
]}

Catalan, Mar ^{[1
]}

Dolz, Manuel F. ^{[1
]}

Mestre, Jose, I ^{[1
]}

Quintana-Orti, Enrique S. ^{[2
]}

Duato, Jose ^{[2
]}

机构：

[1] Univ Jaume 1, Castellon de La Plana, Spain

[2] Univ Politecn Valencia, Valencia, Spain

来源：

2021 29TH EUROMICRO INTERNATIONAL CONFERENCE ON PARALLEL, DISTRIBUTED AND NETWORK-BASED PROCESSING (PDP 2021) | 2021年

关键词：

Deep neural networks (DNNs); distributed training; analytical modeling; clusters; COLLECTIVE COMMUNICATION;

D O I：

10.1109/PDP52278.2021.00024

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

We perform a theoretical analysis comparing the scalability of data versus model parallelism, applied to the distributed training of deep convolutional neural networks (CNNs), along live axes: batch size, node (floating-point) arithmetic performance, node memory bandwidth, network link bandwidth, and cluster dimension. Our study relies on analytical performance models that can he configured to reproduce the components and organization of the CNN model as well as the hardware configuration of the target distributed platform. In addition, we provide evidence of the accuracy of the analytical models by performing a validation against a Python library for distributed deep learning training.

引用

页码：99 / 108

页数：10

共 50 条

[41] Investigation of training performance of convolutional neural networks evolved by genetic algorithms using an activity function
Betere, Job Isaac
Kinjo, Hiroshi
Nakazono, Kunihiko
Oshiro, Naoki
ARTIFICIAL LIFE AND ROBOTICS, 2020, 25 (01) : 1 - 7
[42] Investigation of training performance of convolutional neural networks evolved by genetic algorithms using an activity function
Job Isaac Betere
Hiroshi Kinjo
Kunihiko Nakazono
Naoki Oshiro
Artificial Life and Robotics, 2020, 25 : 1 - 7
[43] Training Convolutional Neural Networks with Limited Training Data for Ear Recognition in the Wild
Emersic, Ziga
Stepec, Dejan
Struc, Vitomir
Peer, Peter
2017 12TH IEEE INTERNATIONAL CONFERENCE ON AUTOMATIC FACE AND GESTURE RECOGNITION (FG 2017), 2017, : 987 - 994
[44] Training Convolutional Neural Networks for Translational Invariance on SAR ATR
Mahngre-Hansen, David
Engholm, Rasmus
Pedersen, Morten Osiergaard
11TH EUROPEAN CONFERENCE ON SYNTHETIC APERTURE RADAR (EUSAR 2016), 2016, : 459 - 462
[45] Training Strategy for Convolutional Neural Networks in Pedestrian Gender Classification
Ng, Choon-Boon
Tay, Yong-Haur
Goi, Bok-Min
SECOND INTERNATIONAL WORKSHOP ON PATTERN RECOGNITION, 2017, 10443
[46] Effective Training of Convolutional Neural Networks for Insect Image Recognition
Martineau, Maxime
Raveaux, Romain
Chatelain, Clement
Conte, Donatello
Venturini, Gilles
ADVANCED CONCEPTS FOR INTELLIGENT VISION SYSTEMS, ACIVS 2018, 2018, 11182 : 426 - 437
[47] Band-limited Training and Inference for Convolutional Neural Networks
Dziedzic, Adam
Paparrizos, John
Krishnan, Sanjay
Elmore, Aaron
Franklin, Michael
INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 97, 2019, 97
[48] A Gradient Boosting Approach for Training Convolutional and Deep Neural Networks
Emami, Seyedsaman
Martinez-Munoz, Gonzalo
IEEE OPEN JOURNAL OF SIGNAL PROCESSING, 2023, 4 : 313 - 321
[49] Reconstruction Combined Training for Convolutional Neural Networks on Character Recognition
Chen, Li
Wang, Song
Fan, Wei
Sun, Jun
Satoshi, Naoi
2015 13TH IAPR INTERNATIONAL CONFERENCE ON DOCUMENT ANALYSIS AND RECOGNITION (ICDAR), 2015, : 431 - 435
[50] Automated Training of Deep Convolutional Neural Networks for Cell Segmentation
Sadanandan, Sajith Kecheril
Ranefall, Petter
Le Guyader, Sylvie
Wahlby, Carolina
SCIENTIFIC REPORTS, 2017, 7

← 1 2 3 4 5 →