Soft mode in the dynamics of over-realizable online learning for soft committee machines

Frederieke Richert*, Roman Worschech*, Bernd Rosenow

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

3 Citations (Scopus)
43 Downloads (Pure)

Abstract

Overparametrized deep neural networks trained by stochastic gradient descent are successful in performing many tasks of practical relevance. One aspect of overparametrization is the possibility that the student network has a larger expressivity than the data generating process. In the context of a student-teacher scenario, this corresponds to the so-called over-realizable case, where the student network has a larger number of hidden units than the teacher. For online learning of a two-layer soft committee machine in the over-realizable case, we present evidence that the approach to perfect learning occurs in a power-law fashion rather than exponentially as in the realizable case. All student nodes learn and replicate one of the teacher nodes if teacher and student outputs are suitably rescaled and if the numbers of student and teacher hidden units are commensurate.
Original languageEnglish
Article number L052302
Number of pages6
JournalPhysical Review E
Volume105
DOIs
Publication statusPublished - 17-May-2022
Externally publishedYes

Fingerprint

Dive into the research topics of 'Soft mode in the dynamics of over-realizable online learning for soft committee machines'. Together they form a unique fingerprint.

Cite this