Wide neural networks with bottlenecks are deep Gaussian processes

Journal Article · Tue Sep 01 00:00:00 EDT 2020 · Journal of Machine Learning Research

OSTI ID:1820877

Agrawal, Devanshu ^[1]; Papamarkou, Theodore ^[2]; Hinkle, Jacob ^[2]

---

1. Univ. of Tennessee, Knoxville, TN (United States); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
2. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

There has recently been much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width. However, these results do not apply to architectures that require one or more of the hidden layers to remain narrow. In this paper, we consider the wide limit of BNNs where some hidden layers, called "bottlenecks", are held at finite width. The result is a composition of GPs that we term a "bottleneck neural network Gaussian process" (bottleneck NNGP). Although intuitive, the subtlety of the proof is in showing that the wide limit of a composition of networks is in fact the composition of the limiting GPs. We also analyze theoretically a single-bottleneck NNGP, finding that the bottleneck induces dependence between the outputs of a multi-output network that persists through extreme post-bottleneck depths, and prevents the kernel of the network from losing discriminative power at extreme post-bottleneck depths.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE
• Grant/Contract Number: AC05-00OR22725; AC02-06CH11357; AC52-07NA27344; AC52-06NA25396
• OSTI ID: 1820877
• Journal Information: Journal of Machine Learning Research, Journal Name: Journal of Machine Learning Research Journal Issue: 175 Vol. 21; ISSN 1532-4435
• Publisher: JMLR
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Bayesian neural networks
deep learning
Gaussian processes
kernels
phase transitions