Layered Neural Networks with GELU Activation, a Statistical Mechanics Analysis

Frederieke Richert; Michiel Straat; Elisa Oostwal; Michael Biehl

Layered Neural Networks with GELU Activation, a Statistical Mechanics Analysis

Frederieke Richert
, Michiel Straat
, Elisa Oostwal
, Michael Biehl^*

^*Corresponding author for this work

Intelligent Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

5 Citations (Scopus)

302 Downloads (Pure)

Abstract

Understanding the influence of activation functions on the
learning behaviour of neural networks is of great practical interest. The
GELU, being similar to swish and ReLU, is analysed for soft committee
machines in the statistical physics framework of off-line learning. We find
phase transitions with respect to the relative training set size, which are
always continuous. This result rules out the hypothesis that convexity is
necessary for continuous phase transitions. Moreover, we show that even
a small contribution of a sigmoidal function like erf in combination with
GELU leads to a discontinuous transition.

Original language	English
Title of host publication	Proceedings ESANN 2023
Subtitle of host publication	European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning
Editors	Michel Verleysen
Publisher	i6doc.com publication
Pages	435-440
Number of pages	6
ISBN (Print)	978-2-87587-088-9
Publication status	Published - 2023
Event	European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning: ESANN 2023 - Bruges, Belgium Duration: 4-Oct-2023 → 6-Oct-2023

Conference

Conference	European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning
Country/Territory	Belgium
City	Bruges
Period	04/10/2023 → 06/10/2023

Access to Document

Layered Neural Networks with GELU Activation, a Statistical Mechanics AnalysisFinal publisher's version, 1.64 MBLicence: Taverne

Handle.net

Persistent link

Cite this

@inproceedings{1484215e140e4002953233d91ec6d5c3,

title = "Layered Neural Networks with GELU Activation, a Statistical Mechanics Analysis",

abstract = "Understanding the influence of activation functions on the learning behaviour of neural networks is of great practical interest. The GELU, being similar to swish and ReLU, is analysed for soft committee machines in the statistical physics framework of off-line learning. We find phase transitions with respect to the relative training set size, which are always continuous. This result rules out the hypothesis that convexity is necessary for continuous phase transitions. Moreover, we show that even a small contribution of a sigmoidal function like erf in combination with GELU leads to a discontinuous transition. ",

author = "Frederieke Richert and Michiel Straat and Elisa Oostwal and Michael Biehl",

year = "2023",

language = "English",

isbn = "978-2-87587-088-9",

pages = "435--440",

editor = "Verleysen, \{Michel \}",

booktitle = "Proceedings ESANN 2023",

publisher = "i6doc.com publication",

note = "European Symposium on Artificial Neural Networks, Computational Intelligence and<br/>Machine Learning : ESANN 2023 ; Conference date: 04-10-2023 Through 06-10-2023",

}

Richert, F, Straat, M, Oostwal, E & Biehl, M 2023, Layered Neural Networks with GELU Activation, a Statistical Mechanics Analysis. in M Verleysen (ed.), Proceedings ESANN 2023: European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. i6doc.com publication, pp. 435-440, European Symposium on Artificial Neural Networks, Computational Intelligence and
Machine Learning, Bruges, Belgium, 04/10/2023.

Layered Neural Networks with GELU Activation, a Statistical Mechanics Analysis. / Richert, Frederieke; Straat, Michiel; Oostwal, Elisa et al.
Proceedings ESANN 2023: European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. ed. / Michel Verleysen. i6doc.com publication, 2023. p. 435-440.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Layered Neural Networks with GELU Activation, a Statistical Mechanics Analysis

AU - Richert, Frederieke

AU - Straat, Michiel

AU - Oostwal, Elisa

AU - Biehl, Michael

PY - 2023

Y1 - 2023

N2 - Understanding the influence of activation functions on the learning behaviour of neural networks is of great practical interest. The GELU, being similar to swish and ReLU, is analysed for soft committee machines in the statistical physics framework of off-line learning. We find phase transitions with respect to the relative training set size, which are always continuous. This result rules out the hypothesis that convexity is necessary for continuous phase transitions. Moreover, we show that even a small contribution of a sigmoidal function like erf in combination with GELU leads to a discontinuous transition.

AB - Understanding the influence of activation functions on the learning behaviour of neural networks is of great practical interest. The GELU, being similar to swish and ReLU, is analysed for soft committee machines in the statistical physics framework of off-line learning. We find phase transitions with respect to the relative training set size, which are always continuous. This result rules out the hypothesis that convexity is necessary for continuous phase transitions. Moreover, we show that even a small contribution of a sigmoidal function like erf in combination with GELU leads to a discontinuous transition.

UR - https://michielstraat.com/files/papers/RichertESANN2023.pdf

M3 - Conference contribution

SN - 978-2-87587-088-9

SP - 435

EP - 440

BT - Proceedings ESANN 2023

A2 - Verleysen, Michel

PB - i6doc.com publication

T2 - European Symposium on Artificial Neural Networks, Computational Intelligence and<br/>Machine Learning

Y2 - 4 October 2023 through 6 October 2023

ER -

Layered Neural Networks with GELU Activation, a Statistical Mechanics Analysis

Abstract

Conference

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this