MEMORY REDUCTION METHOD FOR DEEP NEURAL NETWORK TRAINING

被引：0

作者：

Shirahata, Koichi ^{[1
]}

Tomita, Yasumoto ^{[1
]}

Ike, Atsushi ^{[1
]}

机构：

[1] Fujitsu Labs Ltd, Nakahara Ku, 4-1-1 Kamikodanaka, Kawasaki, Kanagawa 2118588, Japan

来源：

2016 IEEE 26TH INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING (MLSP) | 2016年

关键词：

Deep Neural Networks; Memory Management; Accelerators;

D O I：

暂无

中图分类号：

TM [电工技术]; TN [电子技术、通信技术];

学科分类号：

0808 ; 0809 ;

摘要：

Training deep neural networks requires a large amount of memory, making very deep neural networks difficult to fit on accelerator memories. In order to overcome this limitation, we present a method to reduce the amount of memory for training a deep neural network. The method enables to suppress memory increase during the backward pass, by reusing the memory regions allocated for the forward pass. Experimental results exhibit our method reduced the occupied memory size in training by 44.7% on VGGNet with no accuracy affection. Our method also enabled training speedup by increasing the mini batch size up to double.

引用

页数：6

共 50 条

[41] Training Behavior of Deep Neural Network in Frequency Domain
Xu, Zhi-Qin John
Zhang, Yaoyu
Xiao, Yanyang
NEURAL INFORMATION PROCESSING (ICONIP 2019), PT I, 2019, 11953 : 264 - 274
[42] Development of an Optimised Dataset for Training a Deep Neural Network
Newman, Callum
Petzing, Jon
Goh, Yee Mey
Justham, Laura
ADVANCES IN MANUFACTURING TECHNOLOGY XXXIV, 2021, 15 : 15 - 20
[43] Vector Analysis of Deep Neural Network Training Process
Podoprosvetov, Alexey
Smolin, Vladimir
Sokolov, Sergey
DEEP LEARNING THEORY AND APPLICATIONS, PT I, DELTA 2024, 2024, 2171 : 219 - 237
[44] Deep Neural Network Training Emphasizing Central Frames
Kurata, Gakuto
Willett, Daniel
16TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH COMMUNICATION ASSOCIATION (INTERSPEECH 2015), VOLS 1-5, 2015, : 3595 - 3599
[45] Accelerating Data Loading in Deep Neural Network Training
Yang, Chih-Chieh
Cong, Guojing
2019 IEEE 26TH INTERNATIONAL CONFERENCE ON HIGH PERFORMANCE COMPUTING, DATA, AND ANALYTICS (HIPC), 2019, : 235 - 245
[46] Cooperative Initialization based Deep Neural Network Training
Singh, Pravendra
Varshney, Munender
Namboodiri, Vinay P.
2020 IEEE WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION (WACV), 2020, : 1130 - 1139
[47] Soft Memory Box: A Virtual Shared Memory Framework for Fast Deep Neural Network Training in Distributed High Performance Computing
Ahn, Shinyoung
Kim, Joongheon
Lim, Eunji
Kang, Sungwon
IEEE ACCESS, 2018, 6 : 26493 - 26504
[48] Reduction of training computation by network optimization of Integration Neural Network approximator
Iwata, Yoshiharu
Wakamatsu, Hidefumi
2023 IEEE/SICE INTERNATIONAL SYMPOSIUM ON SYSTEM INTEGRATION, SII, 2023,
[49] A novel method to stop neural network training
Iyer, MS
Rhinehart, RR
PROCEEDINGS OF THE 2000 AMERICAN CONTROL CONFERENCE, VOLS 1-6, 2000, : 929 - 933
[50] A hybrid method for neural-network training
Voglis, C
Lagaris, IE
ADVANCES IN SCATTERING AND BIOMEDICAL ENGINEERING, PROCEEDINGS, 2004, : 431 - 438

← 1 2 3 4 5 →