acceptodds
Under review as a conference paper at ICLR 2027

Spectral Thresholds for Order and Gibbs Sampling Relaxation in a Spin System with a Given Coupling Matrix

Abstract

Statistical physics describes the properties of a typical spin system drawn from an ensemble, whereas in learning systems the properties of a specific system, such as a trained model, are of interest. We show that for a spin system defined by a given coupling matrix , the spectrum of locates the phase transition in the inverse temperature . For , where , the system is disordered: the mean squared overlap of two replicas is of , as for independent spins. When has a delocalized top eigenvector and a bulk of small operator norm, the system is ordered for : the mean squared overlap is of order one. In a restricted Boltzmann machine (RBM) with weight matrix and singular values , the spectrum also determines how fast Gibbs sampling relaxes. For each singular value sets a time scale: the projection of the configuration on the -th singular direction of has a variance of and a corresponding decorrelation time scale, and is uncorrelated with the other projections. Experiments on synthetic couplings confirm these predictions. In an RBM trained on MNIST, the relaxation times and the onset of order follow the same formulas with multiplied by the measured average response of the units to their fields.

Primary area:
learning theory
Keywords:
spin systems, restricted Boltzmann machine, phase transition, Gibbs sampling, relaxation time, statistical physics of neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zfkVOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I want to alert the ACs that I'm not 'qualified' to review this paper (I'm happy to have it replaced with another work).

This paper is treating a toy physics problem, and also appears almost 100% LLM generated with 60 pages of unnecessary math formatting.

Critical strengths

N/A

Critical weaknesses

N/A

Questions for authors

N/A

Reviewer_vLZFOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies equilibrium correlations and Gibbs-sampling relaxation in binary spin systems with a fixed coupling matrix. It combines spectral disorder bounds with sufficient conditions for order under delocalized dominant patterns and a small residual coupling. For low-rank bipartite systems, it derives estimates for the variance, cross-covariance, and integrated autocorrelation time of projections onto singular directions. Synthetic experiments examine these predictions across system sizes. An MNIST-trained RBM, with biases discarded after training, is used to test a response-corrected approximation for a weight matrix with a substantial spectral bulk.

  • This work addresses a well-defined sampling problem and offers meaningful theoretical insights, which I find interesting. However, concerns about the finite-size bounds and Gibbs sampler conversion make it difficult to reach a definitive assessment. I currently lean toward weak rejection but remain open to other reviewers' perspectives.
Critical strengths
  1. The paper addresses a clear sampling problem and provides quantitative results for particular coupling matrices. Understanding which collective directions decorrelate slowly is relevant to sampling from spin systems and RBMs. The paper develops this question beyond an overall fast-relaxation bound. The order results treat a delocalized dominant pattern with a controlled residual coupling, while Theorem 4.4 gives directional variance, cross-covariance, and relaxation estimates for several patterns with unequal strengths. For the single-pattern bipartite model, the disorder and order thresholds approach the same value as the residual coupling vanishes. The resulting analysis provides a meaningful theoretical increment to the established connection between spectra, order, and sampling difficulty.

  1. The synthetic experiments examine directional predictions with multiple sizes and appropriate control directions. Figure 3 uses three unequal pattern strengths and seven sizes from n = 128 to 8192. At the largest size, the reported directional times are 10.1 +/- 0.6, 3.71 +/- 0.11, and 2.20 +/- 0.06, compared with the once-per-sweep predictions of approximately 10.01, 3.59, and 2.21. Projections orthogonal to the patterns serve as controls. Table 4 also reports the corresponding variances, and Appendix B examines cross-direction correlations. The comparisons with exact calculations on small systems in Table 2 provide an additional check on the estimators.
Critical weaknesses
  1. The accuracy claims need a clearer distinction between finite-size error bounds, asymptotic predictions, and empirical agreement. Theorem 4.4(ii) bounds the relative error in directional relaxation time through the Poincare constant lambda_P. The text supplies a uniform lower bound for beta mu_1 < 1/2 and for the flat single-pattern case throughout beta mu_1 < 1. Thus, the general statement that each singular direction has time scale 1/(1 - beta mu_k) throughout the latter interval requires qualifications. The abstract and the introductory description should make the model assumptions and the conditions for a controlled time approximation explicit.

The explicit finite-size constants also appear very conservative at the experimental sizes. Substituting the Figure 3 setting, theta_1 = 0.9 and r = 3, into Eqs. (100), (112), and (133) gives C_1 approximately 4.97 x 10^5. Even with the best possible delocalization, kappa_* = 1/n, the variance error bound in Theorem 4.4(i) is approximately 1437 at n = 8192, against a predicted variance of 10 for the top direction. With these supplied constants, the sufficient condition in part (ii) requires kappa_* <= 4.25 x 10^-8, whereas any unit vector in this dimension has kappa_4 >= 1/8192, approximately 1.22 x 10^-4. These calculations indicate that the displayed bounds do not certify the observed accuracy at the plotted sizes. Please report which bounds are informative in the experiments, provide sharper estimates where possible, and distinguish the numerical support for the asymptotic formula from a useful finite-size accuracy guarantee.


  1. The experimental validation needs clearer sampler specifications and a more precise assessment of the trained-RBM approximation. Appendix B states that some runs use alternating block Gibbs updates and converts their times to the single-site sweeps used in the theory. The printed conversion factor, 1/[2(1 - beta mu_k)], needs a derivation and verification because the two samplers have different transition operators. For example, an independent exact finite-state calculation for the flat pattern with n = 8 per block and beta mu = 0.9 gives an integrated autocorrelation time of approximately 4.758 for continuous-time single-site updates with rate-one clocks, and 2.390 for full alternating block updates under the discrete convention 1/2 + sum of positive-lag correlations. Their ratio is approximately 1.99, whereas the printed factor is 5. This comparison uses a specified exact small system and autocorrelation convention; it establishes a concrete inconsistency to resolve, not that every large-system result is necessarily wrong. Please explain whether the displayed expression denotes a time-unit conversion or a predicted block-chain correlation time, identify the figures and runs using block updates, and verify the implementation. A direct single-site versus block comparison on the same small system would resolve this ambiguity and establish whether any reported comparison is affected. For the trained weight matrix, Eq. (11) additionally uses a joint-Gaussian approximation and replaces individual response coefficients by layer averages. These steps are separate from Theorem 4.4. Appendix A.8 reports measured residuals, but their size alone does not give a small relative-time error without suitable control of lambda_P. Please quantify prediction errors and response heterogeneity, and test the correction on additional independently trained matrices. The interpretation should also consistently identify the target as the model after its biases have been discarded; this changes the model distribution from that used during training.
Questions for authors
  1. How strong are the finite-size guarantees at the sizes used in Figure 3? Can you provide numerical values of the variance and time error bounds for the plotted settings? Please address the conservative constants above and specify which claims are supported by a controlled finite-size bound and which are supported by asymptotic reasoning and empirical agreement.

  1. Which experiments use block Gibbs sampling, and how are their times converted to single-site sweeps? Please derive the conversion stated on page 43 and explain its relationship to the block-chain integrated autocorrelation time. Please provide one direct comparison of the two samplers on the same model, state the time conventions and conversion implemented in the code, and clarify whether any correction changes the plotted results.

  1. How reliable is the response-corrected formula across independently trained RBMs? Please report directional prediction errors for additional training seeds and characterize the variation of individual response coefficients within each layer. This would help establish when the layer-average approximation in Eq. (11) remains accurate.
Reviewer_kGXVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper relates two properties of a spin system with a given coupling matrix JJ to the spectrum of JJ (or, in bipartite systems such as RBMs, to the singular values of the weight matrix WW): whether the system is ordered or disordered, as measured by the overlap order parameter, and the relaxation times of Gibbs sampling, which determine how hard the system is to sample.

The main results are: (i) a threshold for disorder and fast relaxation; (ii) a threshold for order when JJ is a delocalized rank-one spike; (iii) a disorder threshold for an RBM with one pattern; and (iv) a relationship between relaxation times and singular values in an RBM with a few delocalized patterns. The results are illustrated on a few toy models and an RBM trained on MNIST.

Critical strengths

I find the question -- how the coupling matrix determines relaxation and order -- natural and interesting. The derived thresholds are simple and build physical intuition, as does the idea that each singular value sets the time scale of Gibbs sampling along its direction. I can see how this could help in designing and understanding sampling in RBMs and related models.

Critical weaknesses
  • It is not clear how much of this is new. Props. 3.2 and 4.1 are Eldan et al. (2022), Prop. 3.5 follows from Mikulincer & Sohn (2025), per-direction relaxation times recall work on linearized Glauber dynamics (cf. Levin et al., 2010; Decelle et al., 2017, 2018). The new results are some limited, order is proved only for a delocalized spike + negligible bulk, and the relaxation times only for a few patterns with no bulk.
  • The writing is generally clear, but the main text is mostly theorem statements with little intuition, the order parameter is not explained, and the figures are hard to read at a glance. In the MNIST experiment and figure, I could not tell what is predicted and what is verified: the corrected formula uses quantities measured from the same simulations. Moreover, the biases are removed and the trained model appears to lie in the ordered regime, so the theory does not apply to the trained model itself.
Questions for authors

Clarifications:

  • Which statements are new relative to Eldan et al. (2022), Mikulincer & Sohn (2025), Levin et al. (2010), and Decelle et al. (2017, 2018)?
  • In the MNIST experiment, which quantities are inputs and which are predictions? Can χσ,χτ\chi_\sigma,\chi_\tau be obtained without sampling at the same β\beta? If not, what does the formula add over measuring the relaxation times directly?
  • In the MNIST experiment, why are the biases removed? Is it possible to apply the theory to the actual model, with biases? Which unit convention was used in training (±1\pm1 or {0,1}\lbrace0,1\rbrace)?
  • The proof of Thm. 3.3 works for any decomposition J=λww⊤+JbulkJ=\lambda ww^\top+J_{\rm bulk}. Is the spike strength, rather than λmax⁡\lambda_{\max}, the relevant parameter, and can the result tolerate a fixed small bulk?
  • Since E[qab2]\mathbb{E}[q_{ab}^2] is of order one for both pattern and spin-glass order, how is the type of order identified, e.g., for the trained RBM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.