acceptodds
Under review as a conference paper at ICLR 2027

Noise-Debiased Thermodynamic Variance for Local Learning Coefficient Probes

Abstract

Local learning coefficient (LLC) probes offer a singularity-aware view of neural-network training, but mean-energy methods require a local loss baseline that is ambiguous at transient checkpoints. Thermodynamic variance avoids this input; under mini-batch evaluation, however, direct variance mixes cross-state loss fluctuations with same-state noise. We operationalize this route with the Shift-Invariant Variance Estimator (SIVE), which estimates and subtracts the latter component using repeated evaluations. Conditional on any fixed retained path, unclipped SIVE is unbiased for noiseless path variance without requiring MCMC stationarity. The finite-scale diagnostic remains indexed by localization scale —even a locally linear loss has tether-dependent variance—while interpretation as a Real Log Canonical Threshold (RLCT) requires additional stationary low-temperature conditions. Toy experiments recover calibrated finite-scale targets. At the primary localization scale, all five MNIST MLP trajectories exhibit a mid-training trough followed by a rebound in SIVE, while Raw Variance decreases from Epoch 40 to 100 in every trajectory. Across four localization scales, the joint early-drop/late-rise criterion is met in 19 of 20 trajectory–scale pairs. At Epoch 40, the estimated observation-noise correction accounts for of Raw Variance. Same-state debiasing thus reveals a reproducible turning structure masked by time-varying observation noise.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
local learning coefficient, real log canonical threshold, singular learning theory, thermodynamic variance, stochastic gradient Langevin dynamics, observation-noise debiasing, neural network training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wZ4uOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes to estimate a local learning coefficient along training curves by running a Langevin chain tethered to non-stationary points along the path, and computing the variance of the loss along it. This is based on a 7-year-old non-peer-reviewed arXiv paper, Imai 2019. From this variance the mini-batch evaluation noise is subtracted by empirical estimation of the batch noise (Theorem 1, the ANOVA within-group correction). It is validated on two toy energies with injected noise and on five MLPs on MNIST. On the latter, the curve shows a dip at epoch ~40 if the batch noise is accounted for, but that dip disappears if not. The significance of this observation is undiscussed

Critical strengths

None.

Critical weaknesses
  1. The paper is bearly readable, and presents no question. The abstract says a measurement can be fixed and describes a “turning structure” in a curve whose meaning is unclear and its discussion is deferred to future work. notation is used before it's defined and necessary contrived vocabulary is used which the generic ICLR reader is unfamiliar with. In addition the paper adds its own undefined and non standard terms (“local chart”, “Gaussian tether”, “retention rule”) to refer to standard and common concepts. Pages of formalism are devoted to bookkeeping (a burn-in window, a Bessel correction to the variance) while the conditions that would make the statistic an LLC (Eq. 15) are never checked.
  2. No contribution. The base estimator this paper claims to improve and treats as well known, is an unpublished 2019 preprint with 15 citations. The paper’s addition is the ANOVA within-group correction (Searle et al. 1992, cited by the authors) for a noise its own protocol creates: more than half the samples are evaluated per step, while evaluating on all would remove the noise identically for at less than twice the cost. Theorem 1 is just linearity of expectation.
Questions for authors

What is the actual problem you were trying to solve? how did your results answer it? what did we learn?

Reviewer_MjYoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The local learning coefficient (LLC) is a recently introduced complexity measure for neural networks, that seeks to quantify the volume scaling around a local minimum in the loss function, even if that local minimum is singular (as is often the case in neural networks). More generally, one may want to estimate the LLC during the course of training in order to detect transitions between different training regimes. One way of doing that estimation is via a quantity that looks like an effective mean energy, but is defined with respect to a particular reference point on the landscape. This may lead to estimates that are sensitive to the specific choice of the reference point. The paper introduces an alternative approach based on the loss variance, which is insensitive to reference point. However in practice this variance is calculated using minibatching, so it has to be corrected ("noise-debiased") to remove the minibatch contribution. The authors provide an algorithm (SIVE) to implement this debiasing.

Critical strengths

The authors have identified potential problems in LLC estimation (reference point dependence and artifacts due to mini-batching in using variance as an alternative), and developed a solution. The solution is validated on toy models (two simple 2D landscapes) and implemented in a dense network trained on MNIST.

Critical weaknesses

The paper does not conclusively demonstrate that the identified problems with LLC estimation are major issues in any practical sense: does using a mean energy approach lead to qualitatively different conclusions than SIVE when trying to characterize training regimes?

The SIVE approach outperforms the mean energy method on the toy model, but the results in the dense network case (Fig. 1B) are hard to interpret: we see five different estimates (including SIVE) but it is not clear whether any of them are more accurate than any other. Without a ground truth, it is hard to evaluate whether the claimed benefits of SIVE bear out in real-world contexts.

The paper is written in a highly technical, jargon-rich style that obscures rather than clarifies the underlying material. For example the additional noise due to minibatching is pitched as a "stochastic deployment gap", which inflates the significance of a something that is a fairly straightforward. It is unlikely that anyone would ever use the "raw" variance without doing some sort of minibatch correction, so the contribution of the SIVE approach seems somewhat incremental and focused on a minor technical point.

Questions for authors
  1. Can the authors provide examples where errors in LLC estimation via mean energy methods lead to meaningful errors in identifying and describing training regimes? In other words, is there a significant problem that motivates the necessity of SIVE? Pointing out the different trends in Fig. 1B are not really enough, because it is unclear whether any of these trends are correct in the first place.

  2. Following up on the point above, providing a ground truth estimate in Fig. 1B is critical, because otherwise it is hard to see whether SIVE is doing better than the alternatives. One possibility is using the full data instead of minibatching, which is somewhat cumbersome but not impossible for the small network used in the example. At the very least this full-data case would have no minibatching noise, so should agree with SIVE if the latter algorithm succeeded in removing the noise.

Reviewer_aWR9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a Shift-Invariant Variance Estimator of the loss under noisy parameter-changes at given checkpoints. The method is shown to be unbiased theoretically, and evaluated on a toy settings where the optima are known, and on a multi-layer perceptron trained on MNIST.

Critical strengths

None.

Critical weaknesses

The paper feels AI-generated and not carefully checked. After spending an hour trying to understand this paper, I still don't understand what problem SIVE tries to address. Algorithm 1 is readable, but it is still unclear to me what it is useful for. The cited Imai paper (which provides the starting point of this paper, according to the introduction), is about estimating the Bayesian information criterions, but this paper seems to be unrelated, or at least never mentions this goal.

Questions for authors

I gave up at some point listing all the weaknesses, but here are a few points I noted:

  • abstract and main contribution statement are full of unexplained jargon and difficult to understand.
  • missing references, e.g. in the first sentence of the second paragraph of the introduction for SLT and RLCT, first sentence of background and probe setup
  • Acronyms SGLD and SIVE not introduced in the introduction
  • What is a DNN runner?
  • Eq. 1: what is the exact definition of the localized tempered posterior q?
  • What is X^n in the definition of \ell_n above Eq. 3?
  • What is \phi(\theta) in Eq. 3?
  • Where are the bands in Fig 1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.