Spectral Thresholds for Order and Gibbs Sampling Relaxation in a Spin System with a Given Coupling Matrix

Statistical physics describes the properties of a typical spin system drawn from an ensemble, whereas in learning systems the properties of a specific system, such as a trained model, are of interest. We show that for a spin system defined by a given coupling matrix $J$, the spectrum of $J$ locates the phase transition in the inverse temperature $\beta$. For $\beta <1/\gamma (J)$, where $\gamma (J)={\lambda }_{\max }-{\lambda }_{\min }$, the system is disordered: the mean squared overlap of two replicas is of $O(1/N)$, as for independent spins. When $J$ has a delocalized top eigenvector and a bulk of small operator norm, the system is ordered for $\beta >1/{\lambda }_{\max }$: the mean squared overlap is of order one. In a restricted Boltzmann machine (RBM) with weight matrix $W$ and singular values ${\mu }_1\ge {\mu }_2\ge \cdots$, the spectrum also determines how fast Gibbs sampling relaxes. For $\beta <1/{\mu }_1$ each singular value sets a time scale: the projection of the configuration on the $k$-th singular direction of $W$ has a variance of $1/(1-\beta {\mu }_k)$ and a corresponding decorrelation time scale, and is uncorrelated with the other projections. Experiments on synthetic couplings confirm these predictions. In an RBM trained on MNIST, the relaxation times and the onset of order follow the same formulas with ${\mu }_k$ multiplied by the measured average response of the units to their fields.