Comparative analysis of activation functions in neural networks

被引：4

作者：

Kamalov, Firuz ^{[1
]}

Nazir, Amril ^{[2
]}

Safaraliev, Murodbek ^{[3
]}

Cherukuri, Aswani Kumar ^{[4
]}

Zgheib, Rita ^{[5
]}

机构：

[1] Canadian Univ Dubai, Dept Elect Engn, Dubai, U Arab Emirates

[2] Zayed Univ, Dept Informat Syst, Abu Dhabi, U Arab Emirates

[3] Ural Fed Univ, Automated Elect Syst Dept, Ekaterinburg, Russia

[4] Vellore Inst Technol, Sch IT & Engn, Vellore, Tamil Nadu, India

[5] Canadian Univ Dubai, Dept Comp Sci, Dubai, U Arab Emirates

来源：

2021 28TH IEEE INTERNATIONAL CONFERENCE ON ELECTRONICS, CIRCUITS, AND SYSTEMS (IEEE ICECS 2021) | 2021年

关键词：

activation function; neural networks; ReLU; sigmoid; loss function;

D O I：

10.1109/ICECS53924.2021.9665646

中图分类号：

TM [电工技术]; TN [电子技术、通信技术];

学科分类号：

0808 ; 0809 ;

摘要：

Although the impact of activations on the accuracy of neural networks has been covered in the literature, there is little discussion about the relationship between the activations and the geometry of neural network model. In this paper, we examine the effects of various activation functions on the geometry of the model within the feature space. In particular, we investigate the relationship between the activations in the hidden and output layers, the geometry of the trained neural network model, and the model performance. We present visualizations of the trained neural network models to help researchers better understand and intuit the effects of activation functions on the models.

引用

页数：6

共 50 条

[1] Neural networks with periodic and monotonic activation functions:: a comparative study in classification problems
Sopena, JM
Romero, E
Alquézar, R
[J]. NINTH INTERNATIONAL CONFERENCE ON ARTIFICIAL NEURAL NETWORKS (ICANN99), VOLS 1 AND 2, 1999, (470): : 323 - 328
[2] Wavelets as activation functions in Neural Networks
Herrera, Oscar
Priego, Belem
[J]. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS, 2022, 42 (05) : 4345 - 4355
[3] Simple activation functions for neural and fuzzy neural networks
Mendil, B
Benmahammed, K
[J]. ISCAS '99: PROCEEDINGS OF THE 1999 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS, VOL 5: SYSTEMS, POWER ELECTRONICS, AND NEURAL NETWORKS, 1999, : 347 - 350
[4] Comparative Performance Analysis of Neural Networks Architectures on H2O Platform for Various Activation Functions
Kochura, Yuriy
Stirenko, Sergii
Gordienko, Yuri
[J]. 2017 IEEE INTERNATIONAL YOUNG SCIENTISTS FORUM ON APPLIED PHYSICS AND ENGINEERING (YSF), 2017, : 70 - 73
[5] Simple Electromagnetic Analysis Against Activation Functions of Deep Neural Networks
Takatoi, Go
Sugawara, Takeshi
Sakiyama, Kazuo
Li, Yang
[J]. APPLIED CRYPTOGRAPHY AND NETWORK SECURITY WORKSHOPS, ACNS 2020, 2020, 12418 : 181 - 197
[6] Analysis of Activation Functions for Particle Swarm Optimised Feedforward Neural Networks
van Wyk, Andrich B.
Engelbrecht, Andries P.
[J]. 2016 IEEE CONGRESS ON EVOLUTIONARY COMPUTATION (CEC), 2016, : 423 - 430
[7] A comparative performance analysis of different activation functions in LSTM networks for classification
Amir Farzad
Hoda Mashayekhi
Hamid Hassanpour
[J]. Neural Computing and Applications, 2019, 31 : 2507 - 2521
[8] A comparative performance analysis of different activation functions in LSTM networks for classification
Farzad, Amir
Mashayekhi, Hoda
Hassanpour, Hamid
[J]. NEURAL COMPUTING & APPLICATIONS, 2019, 31 (07): : 2507 - 2521
[9] Quantum activation functions for quantum neural networks
Marco Maronese
Claudio Destri
Enrico Prati
[J]. Quantum Information Processing, 21
[10] A Comparison of Activation Functions in Artificial Neural Networks
Bircanoglu, Cenk
Arica, Nafiz
[J]. 2018 26TH SIGNAL PROCESSING AND COMMUNICATIONS APPLICATIONS CONFERENCE (SIU), 2018,

← 1 2 3 4 5 →