- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzes when the Gaussian localization penalty in the sampling-based LLC estimator of Lau et al. (2025) isolates a target solution without altering its local learning coefficient. This is formulated as a two-sided argument: the localization must be strong enough to suppress remote low-loss regions, yet weak enough not to impose its own contraction scale. Under a joint resolution of the loss and localizer, the paper shows that any localization strength that grows faster than logarithmically but slower than any polynomial in the inverse temperature results in asymptotic convergence to the target LLC without any model-specific knowledge. In addition, the paper separates the error into localization, empirical, and numerical errors and provides the exact response identities. Experimentally, the theoretical results are validated in a two-dimensional setting with a known LLC, and the LLC trajectories of an MNIST grokking MLP and a CIFAR-10 ResNet20 with learning-rate drops are analyzed.
Critical strengths
S1. LLC is a theoretically well-founded framework with practical applications. The localization strength in LLC estimation is currently chosen heuristically. Therefore, the clear, model-agnostic answer the paper provides is of practical significance. Theorem 2 gives a precise sufficient growth class. The counterexamples make the two-sidedness tight. This gives the main formal contribution of the paper.
S2. Numerically, a toy experiment validates the predicted regimes, and the DNN experiments on real data demonstrate localization sensitivity and reproducible qualitative patterns.
Critical weaknesses
W1. The guarantee is asymptotic in , but all DNN experiments are run at one fixed , which does not give a complete picture that aligns with the theoretical results. The finite-scale quantities are derived but are not computed or presented for the DNN experiments. Therefore, the empirical part of the headline claim that “theory-guided localization yields reliable LLC measurements” feels overstated; instead, what is shown corresponds more to qualitative trends that are robust to .
W2. The DNN experiments do not exhibit a case where validity changes a conclusion. The two emphasized signals (a late decline in MNIST and an increase followed by a plateau on CIFAR) appear under every tested . Similarly, the only known-truth validation is a 2-D toy experiment; a higher-dimensional model with a known LLC would test the finite- loss–localizer competition where it actually matters.
W3. Some of the illustrative plots (e.g., Fig. 2) are harder to understand, and the effort needed to understand them outweighs their contribution to understanding. Similarly, Fig. 1b uses variables that are not introduced beforehand, and it is not clear how the curves are created. In addition, the caption describes grokking, whereas the panel seems to display learning-rate phases.
W4. The main theorem’s assumptions are not connected adequately to the empirical DNN checkpoints. The theory defines around a population minimizer and requires analytic resolved geometry. The experiments instead freeze intermediate training checkpoints in ReLU networks. Neither the minimizer assumption nor the required analytic setting is established. As in W1, this makes the connection between the numerical results and the theory weaker.
If the discrepancies between the theory and the simulations and experiments can be resolved, I would be willing to increase my score.
Questions for authors
Q1. How do the suggested values from the paper compare with those used in prior LLC studies, and which fall outside the sufficient guarantees of the paper?
Q2. At fixed , what finite-scale criterion justifies choosing as the primary localization strength? Since is asymptotically admissible for every fixed , how is this particular choice calibrated or shown to preserve the target LLC at the experimental scale?
3 comments
Author Response to Reviewer sHUA (2/2)
Readability in figures (W3)
We moved Fig.2 in the submitted version to Appendix C for further reference in the rebuttal version. Instead, we add Fig.2 and Fig.3 for readers to understand intuitively the two localization failures and the core idea of our theory for expressing loss, localizer, and weighted volume as local powers with the same chart. Fig. 1(b) now summarizes the theoretical main result, rather than foregrounding learning dynamics because the main contribution of this paper lies in asymptotic valid localization theory rather than empirical findings in learning dynamics with finite-.
Choice of in previous work and finite- prefactors (Q1/Q2).
Published fixed strengths ( in Lau et al., 2025; in Wang et al., 2025) differ in objective, temperature, and sampled subspace. Please note that even if it is outside Eq. (4)'s sufficient asymptotic class, it does not invalidate finite- measurements. At , with unit prefactor for the multi-stage learning rate and the grokking experiments was illustrative, not calibrated. Section 4 quadratic identity explains the absence of a universal finite- prefactor.
References
- Aoyagi, M. (2024). Consideration on the learning efficiency of multiple-layered neural networks with linear units. Neural Networks, 172, 106132.
- Lau, E., Furman, Z., Wang, G., Murfet, D., & Wei, S. (2025). The Local Learning Coefficient: A Singularity-Aware Complexity Measure. AISTATS, 244–252.
- Wang, G., Hoogland, J., van Wingerden, S., Furman, Z., & Murfet, D. (2025). Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient. ICLR 2025.
Official Comment by Authors
With the discussion phase approaching its end, we would sincerely appreciate it if you could take a moment to review the additional explanations presented in our rebuttal and share your thoughts. We trust that our responses have helped resolve the issues you raised, and we would be thankful if you could reconsider your initial evaluation in light of these clarifications. If any aspects of our work or responses remain unclear, or if you have further recommendations, we would welcome your input. We remain fully committed to addressing any outstanding concerns and incorporating further refinements into the manuscript as needed.
We are truly grateful for the time and effort you have devoted to reviewing our submission and for your valuable insights. We would greatly appreciate hearing your perspective before the discussion period closes.
Best regards, The Authors
Author Response to Reviewer sHUA (1/2)
We appreciate the reviewer's detailed feedback, in particular, the one on the “discrepancies between the theory and the simulations and experiments.” We strengthened the connection through known-truth deep linear networks (DLNs) at several temperatures and explicitly separated those tests from exploratory DNN measurements.
Higher-dimensional known-truth evaluation (W1/W2).
To connect theory and experiments more strongly, we added the truth-known DLN experiments. Fig. 5 evaluates 20 DLNs with analytic LLCs: ten approximately 100K and ten 1M parameters (Aoyagi, 2024). By changing t for population loss and n for empir ical loss of DLNs, we confirmed dependence on t or n instead of multi-layer perceptron for grokking and ResNet20 for multi-stage linear perceptron. With γ=logtloglogt, median population errors stay below 2% over t∈{104,106,108}, while empirical errors reach 3.87% and 3.01% at the largest n=108. Differences from fixed(γ=1)/linear(γ=t) localizations show that γ materially affects accuracy. These are known-truth finite-chain comparisons and we confirmed the validity of the asymptotic localization theory.
Connection between theory and experiments (W4).
To make the relationship between theory and experiments, as we explained it in the first item, we added the truth-known DLN experiments and evaluated how well we can confirm the asymptotic localization theory, in addition to the existing truth-known 2D toy model. By contrast, there are gaps in non-analytically of ReLU, training dynamics with checkpoints (not necessarily local minimum). Therefore, we cast the truth-known 2D toy model and 100k/1M DLNs for confirming the validity of our localization theory. We retain multi-stage learning rates in the main text because ResNet20 training while we increased the number of training data 1000 (submission) to 50000(rebuttal). By contrast, we moved the Grokking experiment to Appendix F because it does not have a strong relationship between asymptotic-t valid localization theory and learning dynamics where a local minimum is not always guaranteed in training.
Revision of headline claims in Abstract (W1).
In Abstract, we changed the headline claim from “theory-guided localization yields reliable LLC measurements” to "Our theory establishes conditions for asymptotically recovering the target LLC, while the experiments distinguish quantitative accuracy against known ground truth from finite-scale sensitivity in DNN training." .
Finite-t theory and diagnostics (W1/W2/W4).
Theoretically, Theorem 1 is asymptotic. Proposition D.1 gives a finite-t partition-ratio bound, while Proposition E.5 rules out universal calibration. Empirically, Appendix D derives exact response identities. In the multi-stage learning rate experiment, Fig. 6 show that the 48k–52k change is 73.43±1.20 for γ=(logt)2 versus 67.54±11.89 for γ=1, preserving the training-stage signal with less seed variability. Please note that we changed the number of training data from 1000 (submission) to 50000(rebuttal) because original ResNet20 experiments were conducted in the latter experimental setting (He et al., 2016). In Appendix F, we also provide Table 3 for computing sample-moment plug-ins across four checkpoints and four strengths, as suggested kindly by the reviewer.