An Evaluation of Parametric Activation Functions for Deep Learning

Parametric activation functions, such as PReLU and PELU, are a relatively new subdomain of neural network nonlinearities. In this paper, we present a comparison of these methods across several topologies. We find that parameterizing activation functions in neural networks do not tend to overfit and tend to converge more quickly than non-parametric activations. This is especially important in environments where time and resources are limited, such as in embedded and mobile systems. We also introduce the Bendable Linear Unit (BLU), which synthesizes many useful properties of other activations, including PReLU, ELU, and SELU. Our experiments indicate that parametric activations that can approximate the identity function can autonomously learn to make residual connections in deep networks. BLU outperforms other activations on the CIFAR-10 task when using a topology without explicit residual connections. BLU also achieves the highest predictive accuracy of compared activations on the CIFAR-100 task when training with a time limit.