Neural Networks Training on Graphics Processing Unit (GPU) Using Dynamic Parallelism (DP)

Artificial Neural Networks (ANN) are a crucial foundation for deep learning and many machine learning algorithms. Training an ANN is computationally intensive and inherently parallel, thus may be accelerated by a Graphics Processing Unit (GPU). Due to the dependency across different ANN layers, which is created by the nature of Back Propagation (BP) algorithm, it is quite challenging to design a highly efficient ANN training algorithm on GPU. In this work, we investigate and demonstrate the technology, Dynamic Parallelism (DP) and will further speed up an ANN training task on GPU. We implemented a generic ANN framework on GPU that consists of an arbitrary number of layers and an arbitrary number of nodes in each layer. In two sets of experiments, we trained the generic ANN on GPU for handwritten digit recognition with DP enabled and disabled. We observed that training ANNs on GPU with DP enabled achieved up to 12.7x performance gain, compared with that with DP disabled on GPU. After being trained on GPU, our neural network achieved an accuracy rate of 96% in handwritten digit recognition.