MEMORY REDUCTION METHOD FOR DEEP NEURAL NETWORK TRAINING

被引：0

作者：

Shirahata, Koichi ^{[1
]}

Tomita, Yasumoto ^{[1
]}

Ike, Atsushi ^{[1
]}

机构：

[1] Fujitsu Labs Ltd, Nakahara Ku, 4-1-1 Kamikodanaka, Kawasaki, Kanagawa 2118588, Japan

来源：

2016 IEEE 26TH INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING (MLSP) | 2016年

关键词：

Deep Neural Networks; Memory Management; Accelerators;

D O I：

暂无

中图分类号：

TM [电工技术]; TN [电子技术、通信技术];

学科分类号：

0808 ; 0809 ;

摘要：

Training deep neural networks requires a large amount of memory, making very deep neural networks difficult to fit on accelerator memories. In order to overcome this limitation, we present a method to reduce the amount of memory for training a deep neural network. The method enables to suppress memory increase during the backward pass, by reusing the memory regions allocated for the forward pass. Experimental results exhibit our method reduced the occupied memory size in training by 44.7% on VGGNet with no accuracy affection. Our method also enabled training speedup by increasing the mini batch size up to double.

引用

页数：6

共 50 条

[1] Memory Efficient Deep Neural Network Training
Shilova, Alena
[J]. EURO-PAR 2021: PARALLEL PROCESSING WORKSHOPS, 2022, 13098 : 515 - 519
[2] Logarithmic Compression for Memory Footprint Reduction in Neural Network Training
Hirose, Kazutoshi
Uematsu, Ryota
Ando, Kota
Orimo, Kentaro
Ueyoshi, Kodai
Ikebe, Masayuki
Asai, Tetsuya
Takamaeda-Yamazaki, Shinya
Motomura, Masato
[J]. 2017 FIFTH INTERNATIONAL SYMPOSIUM ON COMPUTING AND NETWORKING (CANDAR), 2017, : 291 - 297
[3] EPMC: efficient parallel memory compression in deep neural network training
Zailong Chen
Shenghong Yang
Chubo Liu
Yikun Hu
Kenli Li
Keqin Li
[J]. Neural Computing and Applications, 2022, 34 : 757 - 769
[4] EPMC: efficient parallel memory compression in deep neural network training
Chen, Zailong
Yang, Shenghong
Liu, Chubo
Hu, Yikun
Li, Kenli
Li, Keqin
[J]. NEURAL COMPUTING & APPLICATIONS, 2022, 34 (01): : 757 - 769
[5] Fast and robust analog in-memory deep neural network training
Rasch, Malte J.
Carta, Fabio
Fagbohungbe, Omobayode
Gokmen, Tayfun
[J]. NATURE COMMUNICATIONS, 2024, 15 (01)
[6] Memory Saving Method for Enhanced Convolution of Deep Neural Network
Li, Ling
Tong, Yuqi
Zhang, Hangyu
Wan, Dayu
[J]. 2018 11TH INTERNATIONAL SYMPOSIUM ON COMPUTATIONAL INTELLIGENCE AND DESIGN (ISCID), VOL 1, 2018, : 185 - 188
[7] Deep Neural Network Training Method Based on Individual Differences of Training Samples
Li, Xiang
Liu, Ming
Liu, Ming-Hui
Jiang, Qing
Cao, Yang
[J]. Ruan Jian Xue Bao/Journal of Software, 2022, 33 (12): : 4534 - 4544
[8] Distributed Deep Learning Framework based on Shared Memory for Fast Deep Neural Network Training
Lim, Eun-Ji
Ahn, Shin-Young
Park, Yoo-Mi
Choi, Wan
[J]. 2018 INTERNATIONAL CONFERENCE ON INFORMATION AND COMMUNICATION TECHNOLOGY CONVERGENCE (ICTC), 2018, : 1239 - 1242
[9] A Memory-Efficient Hybrid Parallel Framework for Deep Neural Network Training
Li, Dongsheng
Li, Shengwei
Lai, Zhiquan
Fu, Yongquan
Ye, Xiangyu
Cai, Lei
Qiao, Linbo
[J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, 2024, 35 (04) : 577 - 591
[10] FloatPIM: In-Memory Acceleration of Deep Neural Network Training with High Precision
Imani, Mohsen
Gupta, Saransh
Kim, Yeseong
Rosing, Tajana
[J]. PROCEEDINGS OF THE 2019 46TH INTERNATIONAL SYMPOSIUM ON COMPUTER ARCHITECTURE (ISCA '19), 2019, : 802 - 815

← 1 2 3 4 5 →