Performance analysis of nonlinear activation function in convolution neural network for image classification

Deep learning architectures which are exceptionally deep have exhibited to be incredibly powerful models for image processing. As the architectures become deep, it introduces challenges and difficulties in the training process such as overfitting, computational cost, and exploding/vanishing gradients and degradation. A new state-of-the-art densely connected architecture, called DenseNets, has exhibited an exceptionally outstanding result for image classification. However, it still computationally costly to train DenseNets. The choice of the activation function is also an important aspect in training of deep learning networks because it has a considerable impact on the training and performance of a network model. Therefore, an empirical analysis of some of the nonlinear activation functions used in deep learning is done for image classification. The activation functions evaluated include ReLU, Leaky ReLU, ELU, SELU and an ensemble of SELU and ELU. Publicly available datasets Cifar-10, SVHN, and PlantVillage are used for evaluation.