Structures of neural network effective theories

Banta, Ian; Cai, Tianji; Craig, Nathaniel; Zhang, Zhengkang

doi:10.1103/PhysRevD.109.105007

Title: Structures of neural network effective theories

Journal Article · 06 May 2024 · Physical Review. D.

DOI: https://doi.org/10.1103/PhysRevD.109.105007 · OSTI ID:2346270

;

; Craig, Nathaniel;

We develop a diagrammatic approach to effective field theories (EFTs) corresponding to deep neural networks at initialization, which dramatically simplifies computations of finite-width corrections to neuron statistics. The structures of EFT calculations make it transparent that a single condition governs criticality of all connected correlators of neuron preactivations. Understanding of such EFTs may facilitate progress in both deep learning and field theory simulations. Published by the American Physical Society 2024

View Journal Article

Cite

Export

Save

Sponsoring Organization:: USDOE

Grant/Contract Number:: SC0011702

OSTI ID:: 2346270

Alternate ID(s):: OSTI ID: 2404908

Journal Information:: Physical Review. D., Journal Name: Physical Review. D. Journal Issue: 10 Vol. 109; ISSN PRVDAQ; ISSN 2470-0010

Publisher:: American Physical SocietyCopyright Statement

Country of Publication:: United States

Language:: English

References (13)

Wide neural networks of any depth evolve as linear models under gradient descent ^* Lee, Jaehoon; Xiao, Lechao; Schoenholz, Samuel S. Journal of Statistical Mechanics: Theory and Experiment, Vol. 2020, Issue 12 https://doi.org/10.1088/1742-5468/abc62b	journal	December 2020
Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group Erdmenger, Johanna; Grosvenor, Kevin; Jefferson, Ro SciPost Physics, Vol. 12, Issue 1 https://doi.org/10.21468/SciPostPhys.12.1.041	journal	January 2022
Predicting the outputs of finite deep neural networks trained with noisy gradients Naveh, Gadi; Ben David, Oded; Sompolinsky, Haim Physical Review E, Vol. 104, Issue 6 https://doi.org/10.1103/PhysRevE.104.064301	journal	December 2021
Neural network field theories: non-Gaussianity, actions, and locality Demirtas, Mehmet; Halverson, James; Maiti, Anindita Machine Learning: Science and Technology, Vol. 5, Issue 1 https://doi.org/10.1088/2632-2153/ad17d3	journal	January 2024
Unified field theoretical approach to deep and recurrent neuronal networks Segadlo, Kai; Epping, Bastian; van Meegen, Alexander Journal of Statistical Mechanics: Theory and Experiment, Vol. 2022, Issue 10 https://doi.org/10.1088/1742-5468/ac8e57	journal	October 2022
The Principles of Deep Learning Theory Roberts, Daniel A.; Yaida, Sho; Hanin, Boris https://doi.org/10.1017/9781009023405	book	May 2022
The edge of chaos: quantum field theory and deep neural networks Grosvenor, Kevin; Jefferson, Ro SciPost Physics, Vol. 12, Issue 3 https://doi.org/10.21468/SciPostPhys.12.3.081	journal	March 2022
Learning curves for overparametrized deep neural networks: A field theory perspective Cohen, Omry; Malka, Or; Ringel, Zohar Physical Review Research, Vol. 3, Issue 2 https://doi.org/10.1103/PhysRevResearch.3.023034	journal	April 2021
Non-perturbative renormalization for the neural network-QFT correspondence Erbin, H.; Lahoche, V.; Ousmane Samary, D. Machine Learning: Science and Technology, Vol. 3, Issue 1 https://doi.org/10.1088/2632-2153/ac4f69	journal	February 2022
Neural networks and quantum field theory Halverson, James; Maiti, Anindita; Stoner, Keegan Machine Learning: Science and Technology, Vol. 2, Issue 3 https://doi.org/10.1088/2632-2153/abeca3	journal	April 2021
Quantum field-theoretic machine learning Bachtis, Dimitrios; Aarts, Gert; Lucini, Biagio Physical Review D, Vol. 103, Issue 7 https://doi.org/10.1103/PhysRevD.103.074510	journal	April 2021
Bayesian Learning for Neural Networks Neal, Radford M. Lecture Notes in Statistics https://doi.org/10.1007/978-1-4612-0745-0	book	January 1996
Ti k Z-Feynman: Feynman diagrams with Ti k Z Ellis, Joshua P. Computer Physics Communications, Vol. 210 https://doi.org/10.1016/j.cpc.2016.08.019	journal	January 2017