Analysis of Activation Functions for Particle Swarm Optimised Feedforward Neural Networks

Previous studies of feedforward neural networks (FFNNs) have found that asymptotically bounded activation functions used by particle swarm optimised (PSO) FFNNs have a significant impact on the swarm behaviour and FFNN performance. A number of alternative activation functions have however been developed that offer potential advantages over popularly used functions. The purpose of this study is to compare the Elliot, rectified linear, leaky rectified linear and softplus functions with the sigmoid and hyperbolic tangent functions on classification and regression problems. It is shown that the rectified linear function has equal performance to the sigmoid and hyperbolic tangent without the disadvantages of the bounded activation functions. Adaptive versions of the functions are compared on unscaled data sets using the PSO lambda-gamma algorithm. It is shown that shallower gradients are beneficial to accuracy, but that the FFNNs have inferior generalisation capability when compared to networks trained on scaled data.