Thermodynamics of Restricted Boltzmann Machines and Related Learning Dynamics
- TAU-Inria Saclay, Université Paris-Saclay, Laboratoire de Recherche en Informatique (France)
We investigate the thermodynamic properties of a restricted Boltzmann machine (RBM), a simple energy-based generative model used in the context of unsupervised learning. Assuming the information content of this model to be mainly reflected by the spectral properties of its weight matrix W, we try to make a realistic analysis by averaging over an appropriate statistical ensemble of RBMs. First, a phase diagram is derived. Otherwise similar to that of the Sherrington–Kirkpatrick (SK) model with ferromagnetic couplings, the RBM’s phase diagram presents a ferromagnetic phase which may or may not be of compositional type depending on the kurtosis of the distribution of the components of the singular vectors of W. Subsequently, the learning dynamics of the RBM is studied in the thermodynamic limit. A “typical” learning trajectory is shown to solve an effective dynamical equation, based on the aforementioned ensemble average and explicitly involving order parameters obtained from the thermodynamic analysis. In particular, this let us show how the evolution of the dominant singular values of W, and thus of the unstable modes, is driven by the input data. At the beginning of the training, in which the RBM is found to operate in the linear regime, the unstable modes reflect the dominant covariance modes of the data. In the non-linear regime, instead, the selected modes interact and eventually impose a matching of the order parameters to their empirical counterparts estimated from the data. Finally, we illustrate our considerations by performing experiments on both artificial and real data, showing in particular how the RBM operates in the ferromagnetic compositional phase.
- OSTI ID:
- 22783711
- Journal Information:
- Journal of Statistical Physics, Vol. 172, Issue 6; Other Information: Copyright (c) 2018 Springer Science+Business Media, LLC, part of Springer Nature; http://www.springer-ny.com; Country of input: International Atomic Energy Agency (IAEA); ISSN 0022-4715
- Country of Publication:
- United States
- Language:
- English
Similar Records
Training Restricted Boltzmann Machines With a D-Wave Quantum Annealer
Magnetic interactions in TDAE-C{sub 60}