Empirical Loss Landscape Analysis of Neural Network Activation Functions

被引：1

作者：

Bosman, Anna Sergeevna ^{[1
]}

Engelbrecht, Andries ^{[2
,3
]}

Helbig, Marde ^{[4
]}

机构：

[1] Univ Pretoria, Dept Comp Sci, Pretoria, South Africa

[2] Univ Stellenbosch, Stellenbosch, South Africa

[3] Gulf Univ Sci & Technol, Ctr Appl Math & Bioinformat, Kuwait, Kuwait

[4] Griffith Univ, Sch Informat & Commun Technol, Southport, Qld, Australia

来源：

PROCEEDINGS OF THE 2023 GENETIC AND EVOLUTIONARY COMPUTATION CONFERENCE COMPANION, GECCO 2023 COMPANION | 2023年

基金：

新加坡国家研究基金会;

关键词：

neural networks; activation functions; loss landscape; fitness landscape analysis;

D O I：

10.1145/3583133.3596321

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Activation functions play a significant role in neural network design by enabling non-linearity. The choice of activation function was previously shown to influence the properties of the resulting loss landscape. Understanding the relationship between activation functions and loss landscape properties is important for neural architecture and training algorithm design. This study empirically investigates neural network loss landscapes associated with hyperbolic tangent, rectified linear unit, and exponential linear unit activation functions. Rectified linear unit is shown to yield the most convex loss landscape, and exponential linear unit is shown to yield the least flat loss landscape, and to exhibit superior generalisation performance. The presence of wide and narrow valleys in the loss landscape is established for all activation functions, and the narrow valleys are shown to correlate with saturated neurons and implicitly regularised network configurations.

引用

页码：2029 / 2037

页数：9

共 50 条

[41] Multimodal transistors as ReLU activation functions in physical neural network classifiers
Pesch, Isin Surekcigil
Bestelink, Eva
de Sagazan, Olivier
Mehonic, Adnan
Sporea, Radu A.
SCIENTIFIC REPORTS, 2022, 12 (01)
[42] EvoDNN - Evolving Weights, Biases, and Activation Functions in a Deep Neural Network
Cui, Peiyu
Wiese, Kay C.
2022 IEEE CONFERENCE ON COMPUTATIONAL INTELLIGENCE IN BIOINFORMATICS AND COMPUTATIONAL BIOLOGY (IEEE CIBCB 2022), 2022, : 34 - 42
[43] Discrete time neural network synthesis using interaction activation functions
Novakovic, BM
SENSOR FUSION AND DISTRIBUTED ROBOTIC AGENTS, 1996, 2905 : 30 - 38
[44] Probabilistic Neural Network With Complex Exponential Activation Functions in Image Recognition
Savchenko, Andrey
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2020, 31 (02) : 651 - 660
[45] Hardware implementation of neural network with Sigmoidal activation functions using CORDIC
Tiwari, Vipin
Khare, Nilay
MICROPROCESSORS AND MICROSYSTEMS, 2015, 39 (06) : 373 - 381
[46] Neural Network Activation Functions with Electro-optic Absorption Modulators
George, Jonathan
Mehrabian, Armin
Amin, Rubab
Prucnal, Paul R.
El-Ghazawi, Tarek
Sorger, Volker J.
2018 IEEE INTERNATIONAL CONFERENCE ON REBOOTING COMPUTING (ICRC), 2018, : 104 - 108
[47] A Model of Deep Neural Network for Iris Classification With Different Activation Functions
Eldem, Ayse
Eldem, Huseyin
Ustun, Deniz
2018 INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE AND DATA PROCESSING (IDAP), 2018,
[48] Boosting neural network feature extraction by reduced accuracy activation functions
Raudys, A
PATTERN RECOGNITION, 2003, 36 (06) : 1343 - 1354
[49] Speeding up MLP execution by approximating neural network activation functions
Cancelliere, R
NEURAL NETWORKS FOR SIGNAL PROCESSING VIII, 1998, : 294 - 303
[50] Activation Functions and their Impact on the Training and Performance of Convolutional Neural Network Models
Onwujekwe, Gerald
Yoon, Victoria
AMCIS 2020 PROCEEDINGS, 2020,

← 1 2 3 4 5 →