Design of activation function in CNN for image classification

A new combinatorial activation function called relu-softsign was proposed aiming at the problem that the derivative of the commonly used activation function relu in the convolutional neural network is constant to zero at the x negative axis, which makes it easy to cause neuron necrosis during training, and the existing combinatorial activation function relu-softplus can only use the small learning rate in the case of model convergence, which leads to slow convergence. The image classification effect was improved. The role of the activation function during training was analyzed, and the key points that need to be considered in the design of the activation function were given. The relu and softsign functions were combined piecewise in the positive and negative semi axis of the x axis according to these points, so that the derivative of x negative semi axis was no longer constant to zero. Then comparision with the single activation function and relu-softplus combination activation function was conducted on the MNIST, PI100, CIFAR-100 and Caltech256 datasets. The experimental results show that the combinatorial activation function relu-softsign improves the model classification accuracy, simply and effectively mitigates the irreversible " necrosis" phenomenon of neurons. The convergence speed of the model is accelerated, especially on complex data sets. © 2019, Zhejiang University Press. All right reserved.