Activation Functions and Their Characteristics in Deep Neural Networks

Deep neural networks have gained remarkable achievements in many research areas, especially in computer vision, and natural language processing. The great successes of deep neural networks depend on several aspects in which the development of activation function is one of the most important elements. Being aware of this, a number of researches have concentrated on the performance improvements after the revision of a certain activation function in some specified neural networks. We have noticed that there are few papers to review thoroughly the activation functions employed by the neural networks. Therefore, considering the impact of improving the performance of neural networks with deep architectures, the status and the developments of commonly used activation functions will be investigated in this paper. More specifically, the definitions, the impacts on the neural networks, and the advantages and disadvantages of quite a few activation functions will be discussed in this paper. Furthermore, experimental results on the dataset MNIST are employed to compare the performance of different activation functions.