The Role of the Learning Rate in Layered Neural Networks with ReLU Activation Function

Otávio Cistolo Citton; Frederieke Richert; Michael Biehl

doi:10.14428/esann/2025.ES2025-94

The Role of the Learning Rate in Layered Neural Networks with ReLU Activation Function

Otávio Cistolo Citton, Frederieke Richert, Michael Biehl

Intelligent Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Using the statistical physics framework, we study the online learning dynamics in a particular case of shallow feed-forward neural networks with ReLU activation. By expanding the activation function in terms of Hermite polynomials we derive analytical results for the evolution of order parameters for any learning rate. Moreover, we compare our results with online gradient descent simulations and show how our method describes the typical learning curves. We also present results on how the learning rate affects the overall behavior of the network and its equilibria,
showing different learning regimes and critical values of the learning rate.

Original language	English
Title of host publication	European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning
Subtitle of host publication	33rd ESANN 2025
Editors	Michel Verleysen
Publisher	Ciaco - i6doc.com
Pages	437-442
Number of pages	6
DOIs	https://doi.org/10.14428/esann/2025.ES2025-94
Publication status	Published - Apr-2025
Event	European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning - Brugge, Belgium Duration: 23-Apr-2025 → 25-Apr-2025 Conference number: 35 https://www.esann.org

Conference

Conference	European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning
Abbreviated title	ESANN 2025
Country/Territory	Belgium
City	Brugge
Period	23/04/2025 → 25/04/2025
Internet address	https://www.esann.org

Access to Document

10.14428/esann/2025.ES2025-94

Handle.net

Persistent link

Embargoed Document

The Role of the Learning Rate in Layered Neural Networks with ReLU Activation Function
Final publisher's version, 2.64 MB
Licence: Taverne
Embargo ends: 26/10/2025
Request copy

Cite this

@inproceedings{9bfc43895b634b53a2adce31260c2f41,

title = "The Role of the Learning Rate in Layered Neural Networks with ReLU Activation Function",

abstract = "Using the statistical physics framework, we study the online learning dynamics in a particular case of shallow feed-forward neural networks with ReLU activation. By expanding the activation function in terms of Hermite polynomials we derive analytical results for the evolution of order parameters for any learning rate. Moreover, we compare our results with online gradient descent simulations and show how our method describes the typical learning curves. We also present results on how the learning rate affects the overall behavior of the network and its equilibria,showing different learning regimes and critical values of the learning rate.",

author = "{Cistolo Citton}, Ot{\'a}vio and Frederieke Richert and Michael Biehl",

year = "2025",

month = apr,

doi = "10.14428/esann/2025.ES2025-94",

language = "English",

pages = "437--442",

editor = "Michel Verleysen",

booktitle = "European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning",

publisher = "Ciaco - i6doc.com",

note = "European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2025 ; Conference date: 23-04-2025 Through 25-04-2025",

url = "https://www.esann.org",

}

Cistolo Citton, O , Richert, F & Biehl, M 2025, The Role of the Learning Rate in Layered Neural Networks with ReLU Activation Function. in M Verleysen (ed.), European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning: 33rd ESANN 2025. Ciaco - i6doc.com, pp. 437-442, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Brugge, Belgium, 23/04/2025. https://doi.org/10.14428/esann/2025.ES2025-94

The Role of the Learning Rate in Layered Neural Networks with ReLU Activation Function. / Cistolo Citton, Otávio ; Richert, Frederieke ; Biehl, Michael.
European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning: 33rd ESANN 2025. ed. / Michel Verleysen. Ciaco - i6doc.com, 2025. p. 437-442.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - The Role of the Learning Rate in Layered Neural Networks with ReLU Activation Function

AU - Cistolo Citton, Otávio

AU - Richert, Frederieke

AU - Biehl, Michael

N1 - Conference code: 35

PY - 2025/4

Y1 - 2025/4

N2 - Using the statistical physics framework, we study the online learning dynamics in a particular case of shallow feed-forward neural networks with ReLU activation. By expanding the activation function in terms of Hermite polynomials we derive analytical results for the evolution of order parameters for any learning rate. Moreover, we compare our results with online gradient descent simulations and show how our method describes the typical learning curves. We also present results on how the learning rate affects the overall behavior of the network and its equilibria,showing different learning regimes and critical values of the learning rate.

AB - Using the statistical physics framework, we study the online learning dynamics in a particular case of shallow feed-forward neural networks with ReLU activation. By expanding the activation function in terms of Hermite polynomials we derive analytical results for the evolution of order parameters for any learning rate. Moreover, we compare our results with online gradient descent simulations and show how our method describes the typical learning curves. We also present results on how the learning rate affects the overall behavior of the network and its equilibria,showing different learning regimes and critical values of the learning rate.

U2 - 10.14428/esann/2025.ES2025-94

DO - 10.14428/esann/2025.ES2025-94

M3 - Conference contribution

SP - 437

EP - 442

BT - European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning

A2 - Verleysen, Michel

PB - Ciaco - i6doc.com

T2 - European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning

Y2 - 23 April 2025 through 25 April 2025

ER -

The Role of the Learning Rate in Layered Neural Networks with ReLU Activation Function

Abstract

Conference

Access to Document

Handle.net

Embargoed Document

Fingerprint

Cite this