Activation Function Perturbations in Artificial Neural Networks Effects on Robustness

Artificial Neural Networks (ANNs) are powerful models that can learn underlying nonlinear structures within data, such as images, sounds, and sentences. However, researchers have found a significant unsolved problem with ANNs: small perturbations in input data or within the network's parameters can cause the network to output incorrect predictions or classifications. This vulnerability becomes even more dangerous as models are loaded onto special-purpose chips and computing devices that may be vulnerable to attackers. To address this issue, we investigate the effects of activation function perturbations using foundational mathematical theory within neural networks. We compare our theoretical results with two feed-forward neural networks trained and evaluated on the MNIST dataset. Our findings suggest that even subtle perturbations in activation functions and parameters can have a significant impact on the performance of ANNs. Our methods are effective at both strengthening and destroying ANNs.