acceptodds
Under review as a conference paper at ICLR 2027

The Limits of Imitation: A Theory of Recalibration in Weak-to-Strong Generalization

Abstract

A student trained only on a teacher's predictions can become more accurate than the teacher, although imitation adds no information. Relative to a prior on the target, every predictor's risk splits into ignorance, the risk of the best readout of what it reveals, and calibration error, its distance from that readout. Imitation on fresh inputs cannot reduce ignorance, whatever the downstream algorithm; this one bound contains both the teacher's unobserved signal and the Bayes limit of its labels. A student with no knowledge of its own can gain only by recalibrating its teacher. In a linear Gaussian model with a sketched teacher, fitting pseudo-labels applies a spectral filter to the teacher and pays a leak: the energy the filter removes returns as noise. In this calculus, a minimum-norm teacher is overconfident exactly in the observed directions whose signal variance falls below one threshold, and imitating slightly less than a clone helps, to first order, when that overconfidence exceeds an explicit leak rate. With high-variance signal directions in dimensions, the limiting optimal budget of a minimum-norm student equalizes the fraction of the teacher's error it keeps and the fraction of the removed error it leaks. Along a chain of identical stages, energy and knowledge decay on two clocks, with an underconfident phase between them; at equal total budget, two stages can beat every single-stage ridge. In power-law formulas, interpolating pipelines acquire knowledge at the Bayes exponent, yet their risk plateaus at pure calibration error.

Primary area:
learning theory
Keywords:
weak-to-strong generalization, calibration, knowledge distillation, self-distillation, learning theory, spectral filtering, high-dimensional regression

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_UCtSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when imitation can improve a teacher’s predictions. It attributes improvement to recalibration under a no-extra-information assumption. It derives results on sample budgets, repeated distillation, and scaling laws.

Critical strengths
  • The risk decomposition offers a clear conceptual explanation.
  • The optimal budget and equal budget depth results are concrete contributions.
  • The main derivations appear sound under the stated assumptions.
Critical weaknesses

Main weaknesses:

  • Personally speaking, I am not very convinced by the framing of Theorem 2.1 (Imitation cannot create knowledge). The assumptions already make the student a randomized post-processing of the teacher. The Bayes risk inequality then follows from standard data processing. The theorem looks correct, but I do not see much new in the result itself. Calling it “Imitation cannot create knowledge” makes it sound much broader than what is actually proved.
  • I am not convinced that the core mechanism in Section 3 is novel. Spectral shrinkage and the accompanying variance term already appear in Ildiz et al. (2025, Eqs. 8–10). The “filter + leak” decomposition in Eqs. (3.2)–(3.4) appears to recover these same quantities. Eqs. (3.6)–(3.7) then reinterpret the risk through a calibration-based decomposition. This perspective may be useful, but calling it “recalibration” does not by itself establish a new mechanism for weak-to-strong generalization. The paper should distinguish this reinterpretation from its new quantitative conditions and multistage results.
  • As I read on, I felt the paper drifting further from weak-to-strong generalization. Removing the weak-to-strong generalization framing would leave most of it intact as a paper on linear distillation and regression. The main explanation is already established in Section 3: suitable spectral shrinkage can correct some teacher errors. Yet this mechanism substantially overlaps with Ildiz et al. (2025), as the authors acknowledge. Sections 4–6 add results on budgets, depth, and scaling, but I struggled to see what additional weak-to-strong generalization-specific insights they provide. The mathematics is coherent, and the quantitative results may be useful. Still, I am not convinced that their explanatory value for weak-to-strong generalization matches the prominence of that framing.
  • I also missed any validation beyond synthetic linear regression. Figures 1–2 and Appendix N check the theory within its own model, but there are no experiments with actual pretrained language models. These checks support the calculations, yet leave the relevance to LLM-based weak-to-strong generalization unclear. Given the framing, I would have liked a controlled experiment testing whether the predicted budget or depth effects survive in that setting. Otherwise, the claims should stay closer to the linear model actually studied.

Presentation:

  • I have read quite a few weak-to-strong generalization papers, but I still struggled with “readout” in the abstract and introduction. I only understood what the authors meant after reaching its definition in Section 2. This is a central concept, so having to read ahead to understand the opening argument is frustrating. Please explain upfront that it means the posterior mean.
  • The use of “calibration” also bothers me. As an ML reader, I naturally interpret it as predictive calibration. Here, it depends on the target prior and conditions on the full model parameters. Section 3’s title, “HOW IMITATION RECALIBRATES,” makes this confusion worse. It sounds like a general claim about calibration, while the analysis uses a specific Bayesian definition. I find that framing hard to accept without a clear qualifier. The distinction should be explicit from the beginning, rather than left for the reader to discover later.
  • I also find the sentence “When the student brings no knowledge of its own, weak-to-strong generalization is recalibration” too sweeping. The theorem gives an accounting identity under a specific Bayesian definition. It does not establish that “weak-to-strong generalization is recalibration” in the usual ML sense. Calling the entire phenomenon “recalibration” blurs that distinction.
Questions for authors
  • What does Theorem 2.1 establish beyond standard data processing, and how does this justify the broader claim that “imitation cannot create knowledge”?
  • Which results in Sections 3–6 provide new insights into weak-to-strong generalization beyond the spectral shrinkage mechanism already studied by Ildiz et al. (2025)?
  • Can the authors validate the predicted budget or depth effects using pretrained language models, or clarify why these predictions should extend beyond synthetic linear regression?
5 comments
AuthorsOpenReview ↗

Response to Reviewer UCtS (1/3)

Thank you for quoting the exact sentences. The revised PDF narrows the sentences you quote, which claimed more about weak-to-strong generalization (W2S) than the theorems support, and defines the terms where they first appear (changes in blue).

W3 (drift from W2S).

Removing the weak-to-strong generalization framing would leave most of it intact as a paper on linear distillation and regression.

What makes the model weak-to-strong is an asymmetry: a linear student that sees all dd coordinates is trained on the outputs of a teacher that sees only m<dm \lt d of them. The gain depends on that width. In the spiked model of Theorem 4.2, at a fixed teacher, the best gain V(1−k/(d−k))2V(1-\sqrt{k/(d-k)})^2, where V=σ2(nw−k)/(m−nw)V=\sigma^2(n_{\rm w}-k)/(m-n_{\rm w}) is the teacher's error in the bulk and does not involve dd, increases with the student's dimension dd. The extra coordinates carry no signal, yet at the optimal budget they lower both the share of teacher error the student keeps and the share it leaks; Appendix I.4 now states this. The same calculus covers self-training on fresh inputs, as in Wu, Yang & Sun (cited in the paper as Wu et al., 2026b), which compare a student with its own teacher in the same way; what W2S adds is the width asymmetry.

Linear models with fixed representations are a common theoretical setting for W2S (Ildiz et al., 2025; Medvedev et al., 2025). Moniri & Hassani's Sections 2.2-3 instead study students with informative representations, which is the second route below.

The ledger separates two routes to a gain. A student with a state U′U' informative about the target beyond the teacher has a lower floor, K(βw,U′)K(\beta_{\rm w},U') (remark after Theorem 2.1), and can also gain by elicitation. Our model excludes that route.

The revision makes this scope explicit where the reader meets the claim:

  • The second paragraph of Section 1 now names the recalibration route. It says that a student whose representation carries information beyond the teacher, as a pretrained language model may, can also gain by elicitation, which the model excludes.
  • The abstract now defines "no knowledge of its own" as no representation informative about the target beyond the teacher's output.
  • The title names the route the paper studies, recalibration; the abstract and Section 1 now also state its condition where the reader first meets the claim.
  • The sentence you quote in W8 and the conclusion's "Every gain a student makes over its teacher is recalibration" are narrowed in the same way (W8 below).

Within that scope, three statements bear on W2S:

  1. No gain from a teacher that is nowhere overconfident. If the calibration slopes satisfy gi≥1g_i\ge1 in every direction, no minimum-norm or ridge student improves the teacher in expected risk, nor does any finite chain of them, provided budget and ridge are fixed independently of the teacher's output (Proposition 2.3). Its isotropic ridge case is Moniri & Hassani's Theorem 5.
  2. More pseudo-labels can erase a gain, for a fixed interpolation rule. When the teacher is competent (risk below r2r^2) and its overconfidence exceeds the clone leak rate (Proposition 4.1, Corollary B.3), a minimum-norm student's gain peaks at an interior budget and vanishes at cloning. In the spiked model the limiting optimum is k+k(d−k)k+\sqrt{k(d-k)} (Theorem 4.2). The analogous effect along a stopping-time axis is known (Medvedev et al., 2025).
  3. Where interpolating pipelines plateau. In the power-law formulas, interpolating pipelines plateau at a floor that is pure calibration error (Proposition 6.1). Exponent preservation itself is due to Wu et al. (2026a), and Section 1 credits it.
AuthorsOpenReview ↗

Response to Reviewer UCtS (2/3)

W2 and Q2 (Section 3 versus Ildiz et al.). For the single-stage and two-stage machinery your reading is right:

  • spectral shrinkage and its variance term are Ildiz et al.'s Eqs. (8)-(10);
  • the three-term two-stage risk of our Theorem 3.2 is their Theorem 2, and our leak law (3.4) is their Eqs. (10) and (16) in other notation;
  • Medvedev et al. (their Section 5) describe the same mechanism: the student removes the teacher's energy in low-variance directions that carry no signal.

Theorem 3.2 differs in two respects: the teacher is an interpolating fit restricted to m<dm \lt d coordinates, and the signal has a prior. The revised paper credits these at the statements themselves (the Theorem 3.2 header, after (3.4), the introduction's description of the leak, and the paragraph after Proposition 3.3). It claims no novelty for the filter-and-leak mechanism.

In the terms of your W2, the results sort into three groups:

  • Reading of a known risk. (3.6)-(3.7) rewrite a known deterministic-equivalent risk as the teacher's linear ignorance, plus the calibration error left by the student's mean filter, plus the leak. This reading is what makes the conditions below statable.
  • Quantitative conditions, absent from Ildiz et al., Wu, Yang & Sun, Moniri & Hassani and Medvedev et al.:
    • which directions of a given minimum-norm teacher are overconfident (Proposition 3.3). Derived from the single-stage law, it locates where a shrinking student can gain: in observed directions whose signal strength is below a threshold that does not involve the eigenvalue;
    • the first-order criterion for imitating less than a clone (Proposition 4.1);
    • the limiting budget k+k(d−k)k+\sqrt{k(d-k)} (Theorem 4.2).
  • Multistage results. Wu, Yang & Sun already give the chain recursion, denoising followed by forgetting, and noise and signal decaying at different rates. Beyond them, the paper has:
    • the two clocks (Proposition B.4): along a chain the linear knowledge share ci2/ρic_i^2/\rho_i decays at Fˉi2/μ\bar F_i^2/\mu while energy decays at μ\mu, so a chain on an anisotropic spectrum passes through an underconfident phase. It rests on the exact finite-dd identity of Lemma 3.1, which gives expected chain risks at every depth from products of fixed matrices;
    • the equal-budget certificate (5.6).

The revised related-work paragraph and a new table in Appendix A.3 state this separation.

W1 and Q1 (Theorem 2.1 and data processing). Theorem 2.1 has two parts: an exact identity for the gain, which is the classical calibration-refinement split of squared error in gain form (Murphy, 1973; DeGroot & Fienberg, 1983, both cited in Section 1), and an inequality, which is data processing, as you say:

EEw−EE(w^)=[CE(βw)−CE(w^)]−[K(w^)−K(βw)]≤CE(βw).\mathbb{E}\mathcal E_{\rm w}-\mathbb{E}\mathcal E(\hat w)=\big[\mathrm{CE}(\beta_{\rm w})-\mathrm{CE}(\hat w)\big]-\big[K(\hat w)-K(\beta_{\rm w})\big]\le \mathrm{CE}(\beta_{\rm w}).

What the paper adds is their use as a bound on the W2S gain:

  • in expected risk under the prior, for every procedure whose extra data are independent of the pair (target, teacher output), the gain is at most the teacher's calibration error;
  • the readout attains this bound;
  • Sections 2-5 compute the linear part CElin=∑iλiρi(1−gi)2\mathrm{CE}_{\rm lin}=\sum_i\lambda_i\rho_i(1-g_i)^2 direction by direction, which bounds the gain of every linear student.

You are right that the theorem's old name was broader than its statement. The revision renames the theorem "Imitation on fresh inputs cannot reduce ignorance". The sentence before it now says that the theorem combines orthogonality of the conditional mean with data processing, and that the identity is what the paper uses.

W6 (readout). Agreed. The readout of a predictor β\beta is the posterior mean βˉ=E[w⋆∣β]\bar\beta=\mathbb{E}[w^\star\mid\beta] of the target given the predictor. The abstract and Section 1 now say this at first use.

AuthorsOpenReview ↗

Response to Reviewer UCtS (3/3)

W7 (calibration). Agreed. The submitted abstract mentioned the prior, but the abstract and introduction used the term before Section 2 said that this calibration is Bayesian and conditions on the whole predictor. The abstract and Section 1 now state at first use that calibration here is Bayesian: it averages over a prior on the target and conditions on the whole predictor. Averaged over the prior, it implies calibration of the predicted value x⊤βx^\top\beta; on a single fixed task it need not. Section 2 now carries the same qualifier on its comparison with calibration of the predicted value, after Theorem 2.1. Section 3 is retitled "Recalibration in the linear model: filter and leak". The overview figure's middle panel is renamed "Filter and leak".

W8 (the sweeping sentence). Agreed. It now reads: "When the student brings no knowledge of its own, its expected gain over its teacher can come only from reducing calibration error, and it never exceeds the teacher's calibration error." That is what Theorem 2.1 states for expected risk under the prior; for a single fixed target, Section 2 notes that the bound is only the teacher's risk. The conclusion's sentence is narrowed in the same way.

W4 and Q3 (pretrained language models). The paper's claims are for the linear model: we have not run LLM experiments, and the revised paper states that its results do not claim that the budget or depth effects carry over to them. In an LLM W2S experiment, the student's pretrained representation may carry information about the target beyond the teacher. The experiment would then measure both routes, and the theory accounts only for recalibration.

The ledger does give one diagnostic there. Since the gain equals the drop in calibration error minus the rise in ignorance, a gain larger than the teacher's calibration error can only come from a drop in ignorance, that is, from knowledge the student brought. Using it requires an estimate of that calibration error, which is defined over a family of tasks drawn from a prior (or slopes pooled across tasks, as in the Discussion); we have not attempted it outside the model.

A test of the recalibration route itself needs a student whose representation carries no information about the target beyond the teacher. The Discussion now states this condition for the measurement it proposes.

If the narrowed framing and the separation above resolve your concerns about the W2S claims, we would appreciate it if you would reconsider your rating.

Reviewer_rr47OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes weak-to-strong generalization in a linear Gaussian model with a partially observing teacher. Bayes squared risk splits into ignorance and calibration error. Thm 2.1: imitation on fresh inputs cannot reduce ignorance, so the gain is at most the teacher's calibration error. Thm 3.2: chains of minimum-norm and ridge fits are treated exactly, with deterministic-equivalent risks split into signal distortion, retained teacher fluctuations and leak. Prop 3.3: teacher is overconfident exactly in observed low-signal directions. Thm 4.2: in a spiked model, showing the optimal pseudo-label budget. Sec 5: a two-stage chain provably beats every single ridge at equal budget. Sec 6: interpolating pipelines plateau at pure calibration error, and tuned ridge reaches the supervised exponent.

Critical strengths

Rigorous theory with clean organization. Thm 2.1 gives a single lens on weak-to-strong gains, and the bound "gain ≤ teacher calibration error" appears to be new. Thm 3.2 is carefully derived from the Gaussian surrogate of Han & Xu (2023), with a valid small-noise coupling and uniform integrability. Thm 4.2 gives a closed-form interior optimum for the imitation budget, ns⋆=k+k(d−k)n_s^\star = k + \sqrt{k(d-k)}. These turn qualitative observations, some imitation helps, too much reverts to cloning, into checkable predictions.

Critical weaknesses

Novelty is overstated relative to closely related recent work. Two-stage deterministic equivalents on fresh inputs via Han & Xu already appear in Ildiz et al. and in the multi-step recursion of Wu, Yang & Sun, which already establishes the purification and forgetting mechanism of Sec 5. The isotropic dichotomy in Prop 2.3, 2.4 and Sec 4 is essentially Moniri & Hassani Thm 5.

References:

M. Emrullah Ildiz, Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, and Samet Oymak. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. In International Conference on Learning Representations (ICLR), 2025.

Wu, Mingqi, Archer Y. Yang, and Qiang Sun. "Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting." arXiv preprint arXiv:2602.14029 (2026).

Moniri, Behrad, and Hamed Hassani. "On the mechanisms of weak-to-strong generalization: A theoretical perspective." Advances in Neural Information Processing Systems 38 (2026): 121409-121444.

Questions for authors
  1. Relative to Ildiz et al. (Thm 2) and Wu, Yang & Sun, what exactly is new in Lemma 3.1, Thm 3.2 and the purification to forgetting picture of Sec 5?
  2. How do Prop 2.3 and Sec 4 go beyond Moniri & Hassani's isotropic dichotomy and Medvedev et al.'s shrinkage-optimality bounds?
5 comments
AuthorsOpenReview ↗

Response to Reviewer rr47 (1/3)

Thank you for naming the exact results to compare. We checked each attribution against the arXiv versions (Ildiz et al. v2; Wu, Yang & Sun v1, cited in the paper as Wu et al., 2026b; Moniri & Hassani v1; Medvedev et al. v3). Your attributions are accurate for the three-term structure of Theorem 3.2, the leak law, the chain recursion with denoising and forgetting, and the isotropic ridge case of Proposition 2.3; the points below give, result by result, where the papers differ. The revised paper credits each item where it is stated, and the novelty claims now cover only what is not in these papers.

AuthorsOpenReview ↗

Response to Reviewer rr47 (2/3)

W1 and Q1 (Lemma 3.1, Theorem 3.2 and Section 5 versus Ildiz et al. and Wu, Yang & Sun).

Theorem 3.2. The three-term structure is in Ildiz et al., Theorem 2: signal distortion through the product filter, propagated teacher noise, and the student's own variance, obtained by applying Han & Xu twice. Their setting has a full-feature minimum-norm surrogate and target labels with noise variance bounded below. Wu, Yang & Sun, Theorem 4.2 at t=1t=1, covers noiseless pseudo-labels and ridge stages. Our leak law (3.4) is Ildiz et al.'s Eqs. (10) and (16) in other notation.

Theorem 3.2 uses the same technique with two substitutions:

  • an interpolating teacher (nw<mn_{\rm w} \lt m) restricted to m<dm \lt d coordinates, whose unobserved signal enters as the extra noise AmA_m. Ildiz et al. treat feature-restricted surrogates with infinite data (their Section 3.1) or as under-parameterized least squares (their Appendix C.1).
  • a random prior with a converging empirical law.

Their results also cover what Theorem 3.2 does not: non-asymptotic concentration rates, Ildiz et al.'s covariate shift, and Wu, Yang & Sun's stage-dependent covariances. The revised paper credits both papers in the Theorem 3.2 header and at Theorem D.1, and Ildiz et al. after (3.4). It makes no novelty claim for Theorem 3.2, which serves as the computational tool for the later sections.

Lemma 3.1. We do not claim it as a contribution in itself. It is an exact identity for expected second moments at every finite dd, with an elementary proof, where both papers give deterministic equivalents up to a concentration error vanishing with dimension. It computes expected chain risks at every depth from products of fixed matrices, and it is the tool behind the two clocks (Proposition B.4) and the uniform-in-depth expected risks (Proposition B.6 in Appendix B.4).

Section 5. Several parts are in Wu, Yang & Sun, with a full-feature teacher:

  • the recursion (5.1)-(5.2) is their Eq. (10), credited at (5.1) and at Theorem D.9;
  • so are denoising followed by forgetting and the U-shaped risk;
  • so are noise and signal decaying at different rates that coincide under isotropy (their Section 3.1 and Corollary B.1).

They also give a data-driven stopping rule (iterated GCV, their Theorem 4.3), which our paper does not have.

Three things in Section 5 are not in their paper:

  1. The two clocks (Proposition B.4). The separate noise and signal rates are theirs. Proposition B.4 tracks their ratio in each direction, the linear knowledge share ci2/ρic_i^2/\rho_i: along a chain of identical stages, energy is of exact order μT\mu^T, where μ\mu is the Perron root of the exact second-moment map, while the knowledge share is O((Fˉi2/μ)T)O((\bar F_i^2/\mu)^T), with μ>max⁡iFˉi2\mu>\max_i\bar F_i^2. A deep stage on an anisotropic spectrum therefore passes through an underconfident phase, a statement about its calibration that the separate rates do not make. The benchmark is KlinK_{\rm lin}, the risk of the oracle per-direction readout, which no post-hoc linear correction can beat; one scalar reaches it exactly when the calibration slopes are uniform over the directions that carry energy. General comment point 3 has the consequences for post-hoc rescaling, including where it stops helping and what is derivable from Wu, Yang & Sun.
  2. A population two-stage theorem. In the power-law family (λi=i−a\lambda_i=i^{-a}, ωi2∝i−b\omega_i^2\propto i^{-b}, a noisy minimum-norm teacher) and at population level, with no budget, Theorem D.11 proves that the best two-stage filter strictly beats every single ridge student fitted to the same teacher for b≥ab\ge a. Wu, Yang & Sun compare the best iterate with ridge trained on the labeled data, in simulation (their Fig. 3b), and leave a theorem for future work (their Section 6). Theorem D.11 settles a different comparison.
  3. An equal-budget certificate. (5.6) certifies one finite-budget instance of the formulas. Theorem J.30 transfers it to the actual estimators in a replicated-spectrum limit, each of the 4000 levels repeated q→∞q\to\infty times. In numerical optimization (not certified), with one parameter at a time moved, the advantage over the best single ridge holds at every tested aa, σ2\sigma^2, N/dN/d and nwn_{\rm w}, at bb from 1 to 3, and for first-stage shares from 0.25 to 0.95; it was not found at b=0.5b=0.5 or at shares of 0.20 and below (Table 5).
AuthorsOpenReview ↗

Response to Reviewer rr47 (3/3)

W1 (isotropic part) and Q2 (isotropic results versus Moniri & Hassani; Sections 3-4 versus Medvedev et al. and Moniri & Hassani).

Moniri & Hassani, Theorem 5. For isotropic ridge teachers and students your reading is right. Under their prior N(0,I/d)N(0, I/d) the threshold λt⋆=σ2γt\lambda_t^\star=\sigma^2\gamma_t is the posterior-mean ridge, so a teacher with λt≥λt⋆\lambda_t \ge \lambda_t^\star has calibration slope at least one. The first case of their Theorem 5 is therefore the isotropic ridge instance of Proposition 2.3(i). Their second case contains the instance of 2.3(ii), and adds exact conditions and an interval of student ridges for over-parameterized students, which we do not have. Their Theorem 4 is the isotropic, full-feature ridge analogue of Theorem 3.2. The isotropic minimum-norm statement of Proposition 2.4, iterated along a chain, matches Wu, Yang & Sun's Corollary B.1. The revised paper says so after Proposition 2.4.

Outside the isotropic ridge family, Proposition 2.3 adds:

  • anisotropic Σ\Sigma;
  • any teacher, entering only through its per-direction state (ci,ρi)(c_i,\rho_i);
  • minimum-norm students and chains;
  • finite dd;
  • a sufficient condition for improvement, (ii), that does not involve the spectrum.

Section 4 differs from their Theorem 5 in its control and geometry. They vary the student's ridge at a fixed sample size under isotropy; Section 4 varies the unlabeled budget of a minimum-norm student under anisotropy. Under isotropy, Proposition 4.1 reduces to Proposition 2.4, with no interior optimum.

Medvedev et al. The mechanism of Section 3 is in their Section 5, for early-stopped gradient flow trained on the population: the student removes the teacher's energy in low-variance directions that carry no signal. Their Theorem 3.2 uses the geometry of our Theorem 4.2: kk spikes, a flat bulk, and the target on the spikes. Both are now credited in Section 3 and Section 4.

On shrinkage optimality and Proposition 2.3:

  • A teacher that is shrink-optimal with respect to the target (their Definition 4.1) has aggregate slope at least one, a related aggregate condition.
  • For a target with Ef⋆2≤1\mathbb E f^{\star2}\le1 and a teacher shrink-optimal to it, their Theorem 4.3 concludes LST≥34LTE2L_{\rm ST}\ge\tfrac34L_{\rm TE}^2 for any student shrink-optimal to the teacher, including one that trains a head on pretrained features from the origin.
  • Along a chain of shrink-optimal students, their Corollary 4.6 gives LST≥14LTE2L_{\rm ST}\ge\tfrac14L_{\rm TE}^2.
  • Proposition 2.3(i) assumes slope at least one in every direction and concludes that no minimum-norm or ridge student whose budget and ridge do not depend on the teacher's output, or finite chain of them, improves the teacher in expected risk: a stronger hypothesis and a stronger conclusion, for a narrower class of students.

Relative to Theorem 2.1, their bounds are teacher-side limits of a different kind. Theorem 2.1 is an exact identity in expected risk under a prior, for any procedure whose extra data are independent of the pair (target, teacher output). It gives zero gain from a calibrated teacher and is attained by the readout. Their bounds need shrink-optimality but allow informative student features. Neither statement implies the other.

What Sections 3 and 4 add over both papers:

  • in the risk calculus, which directions of a given minimum-norm teacher are overconfident: Proposition 3.3, ωi2<Γw/τw\omega_i^2 \lt \Gamma_{\rm w}/\tau_{\rm w}, a threshold on signal strength rather than the eigenvalue;
  • the pseudo-label budget as the control, where they use stopping time or the ridge (the leak formula itself is Ildiz et al.'s);
  • the first-order sign criterion near cloning, O>ℓ\mathfrak O \gt \ell (Proposition 4.1);
  • the closed form ns⋆=k+k(d−k)n_s^\star=k+\sqrt{k(d-k)} (Theorem 4.2), in the proportional limit followed by L→∞L\to\infty, with all signal on the spikes inside the teacher's view.

W2 (references) and what changed in the paper.

  • The related-work paragraph of Section 1 now credits, result by result:
    • Ildiz et al.: Theorem 2, the leak law, and their feature-restricted surrogates;
    • Wu, Yang & Sun: the recursion, denoising and forgetting, the rate mismatch, iterated GCV;
    • Moniri & Hassani: Theorem 5;
    • Medvedev et al.: the mechanism, the spiked geometry, Theorem 4.3.
  • The same credits appear where the results are stated: before Theorem 2.1 (the calibration-refinement split), Theorem 3.2 (including Moniri & Hassani's Theorem 4), (3.4), after Propositions 2.4 and 3.3, before Theorem 4.2, (5.1), the paragraph on forgetting in Section 5, (6.2), and Theorems D.1 and D.9.
  • Appendix A.3 adds a table that lists, for each group of results in Sections 2-6, the prior result containing it, with theorem numbers, and what is new.
  • The Moniri & Hassani reference is updated to NeurIPS 2025.

We are glad to answer further questions about any line of the comparison.

Comments on the paper

10 comments
AuthorsOpenReview ↗

Response to Reviewer atFm (1/4)

Response to Reviewer atFm (1/4)

Thank you for the precise questions. The strongest new consequence (Q1) is a per-direction impossibility result, Proposition 2.3. We also ran both computations you asked for, Q2 and Q3, inside the paper's model; the results below include where each effect stops. A label-free scalar brings the depth-200 spiked stage from 0.8024 to 0.2832, below the teacher's 0.3364; at a well-chosen stopping depth, the best fixed scalar moves the best risk only from 0.2320 to 0.2317 (spiked) and from 0.0665 to 0.0646 (power law). The equal-budget advantage is 3.6% to 12.8% of the single-ridge risk as aa, bb, σ2\sigma^2, N/dN/d or nwn_{\rm w} is moved over the ranges below (less at first-stage shares near 0.25), and was not found at b=0.5b=0.5 or at shares of 0.20 and below.

AuthorsOpenReview ↗

Response to Reviewer atFm (2/4)

W1 and Q1 (what the ignorance-calibration viewpoint adds).

What is the strongest new prediction or consequence that follows from the ignorance–calibration viewpoint but does not follow from the existing spectral-risk and denoising–forgetting analyses?

The strongest is a per-direction impossibility result. Suppose the teacher's calibration slopes satisfy gi≥1g_i\ge1 in every direction. Then no minimum-norm or ridge student, nor any finite chain of them, improves the teacher in expected risk, at any finite dd, provided budget and ridge are fixed independently of the teacher's output (Proposition 2.3). The slope coordinates of the ledger are what make it statable: written in them, the single-stage formula gives the asymptotic version, and Lemma 3.1 gives the exact one at every finite dd, for any teacher. Its isotropic ridge case is Moniri & Hassani's Theorem 5; the per-direction, anisotropic, any-teacher form is not in Ildiz et al., Wu, Yang & Sun, Moniri & Hassani or Medvedev et al. (we checked the arXiv versions).

A second consequence, about where and when the gain comes from: in the risk calculus, a minimum-norm teacher is overconfident in an observed direction exactly when ωi2<Γw/τw\omega_i^2 \lt \Gamma_{\rm w}/\tau_{\rm w}, a threshold on signal strength, not on the eigenvalue (Proposition 3.3). To first order near cloning, imitating slightly less than a clone helps when the overconfidence exceeds an explicit leak rate, O>ℓ\mathfrak O \gt \ell, and hurts when it falls short (Proposition 4.1).

Theorem 2.1 frames both: in expected risk under the prior, no procedure whose extra data are independent of the pair (target, teacher output) gains more than the teacher's calibration error, and the readout attains this ceiling (general comment, point 2). Medvedev et al. (Theorem 4.3) cap the gain under a frequentist shrink-optimality condition; ours is Bayesian, covers any such procedure, and is attained, while theirs also allows informative student features, which ours excludes, so neither implies the other.

Inside the linear model, the asymptotic risk formulas follow from known deterministic equivalents (Ildiz et al.; Wu, Yang & Sun). What we claim is not those formulas but the statements above: Proposition 2.3 and Theorem 2.1 hold exactly at finite dd, and Propositions 3.3 and 4.1 are built on the formulas.

For chains, the ledger adds one exact criterion that is new in the revision. The excess risk of a rescaled stage κβ(T)\kappa\beta^{(T)} over the oracle linear readout is ∑iπi(gi−κ)2\sum_i\pi_i(g_i-\kappa)^2, with πi=λiρi\pi_i=\lambda_i\rho_i. So the best scalar κ⋆\kappa^\star is the π\pi-weighted mean slope, and what it leaves is the weighted variance of the slopes: one scalar reaches the readout exactly when the slopes are uniform over the directions that carry energy. In Wu, Yang & Sun's spiked model (cited in the paper as Wu et al., 2026b) the late systematic error comes from a deterministic shrinkage of the signal by the factor (s/(s+τ))t+1(s/(s+\tau))^{t+1} in their Theorem 3.2 (their ss and τ\tau). A rescaling correction is therefore derivable there. In a spiked model, the energy that survives late sits mainly on spikes that share one slope, so one scalar nearly suffices: in our spiked chain it is within 10−310^{-3} of the readout from depth 47 on. On a power-law spectrum the slopes differ, and the criterion explains the gap in Table 2: one scalar removes only 52% of the late calibration error at depth 200 and 64% at depth 1000. Separately, from depth 1099 on, ignorance growth leaves even the rescaled stage above the teacher (Appendix B.2).

On the attributions in W1, we agree. The decomposition is the orthogonality of the conditional mean, and the monotonicity is data processing (general comment, point 2). The single-stage and two-stage risk formulas, including the leak law, are in Ildiz et al.'s Theorems 1 and 2. The chain recursion, denoising followed by forgetting, and a stopping rule are in Wu, Yang & Sun. General comment point 1 lists, result by result, what is theirs, and the revised paper credits each one where it is stated (Theorem 3.2, (3.4), (5.1), the paragraph on forgetting in Section 5) and in Section 1.

Q2, W1c and W2d (post-hoc rescaling of late stages; W1c and W2d are the last points of your W1 and W2). A scalar recovers much of the late-stage loss, up to a limit that depends on the chain; at a well-chosen stopping depth it adds little. We evaluated the deterministic calculus (the formulas behind Table 1 in Appendix B) on that table's two anisotropic chains:

  • power law: d=1000d=1000, (a,b)=(1.2,1)(a,b)=(1.2,1);
  • spiked: d=1000d=1000, k=50k=50, L=100L=100.
AuthorsOpenReview ↗

Response to Reviewer atFm (3/4)

Each stage uses n=d/2n=d/2 fresh inputs. The table compares four predictors:

  • plain: stage TT as trained;
  • best fixed scalar: κ⋆β(T)\kappa^\star\beta^{(T)}, an oracle, since it needs one correlation with the target;
  • teacher-fitted scalar: stage TT regressed on the teacher's outputs over unlabeled inputs; it needs no labels;
  • oracle readout: the best linear per-direction readout, with risk KlinK_{\rm lin}.
Chain, depth TTplainbest fixed scalarteacher-fitted scalaroracle readout KlinK_{\rm lin}
Spiked, 0 (teacher)0.33640.26530.33640.2267
Spiked, 20 (best plain depth)0.23200.23200.27830.2318
Spiked, 2000.80240.23720.28320.2371
Power law, 0 (teacher)0.23530.19750.23530.0632
Power law, 47 (best plain depth)0.06650.06510.06720.0640
Power law, 2000.12460.09350.09370.0652
Power law, 10000.46040.22550.22550.0917

Past the best depth, the two chains behave differently:

  • Spiked chain. The slopes are nearly uniform, so the best scalar is within 10−310^{-3} of the readout from depth 47 on. At depth 200 the scalars are moderate (κ⋆=7.2\kappa^\star=7.2, teacher-fitted 8.9), and the label-free scalar brings the risk from 0.8024 to 0.2832, below the teacher's 0.3364. Far past the energy half-life the scalar becomes huge (κ⋆=4.7×104\kappa^\star=4.7\times10^4 at depth 1000), and recovery there relies on exactly noiseless pseudo-labels.
  • Power-law chain. The slopes are not uniform, so the best scalar removes 52% of the late calibration error at depth 200 and 64% at depth 1000. Ignorance also grows: from depth 1099 on, even the best scalar leaves the stage above the teacher's risk.

At and near the best depth, rescaling helps little:

  • The best fixed scalar barely improves the best achievable risk: power law, 0.0665 at depth 47 becomes 0.0646 at depth 56; spiked, 0.2320 at depth 20 becomes 0.2317 at depth 9.
  • At the best plain depth, the label-free scalar is worse than no rescaling in both chains: 0.2783 against 0.2320, and 0.0672 against 0.0665.
  • The label-free scalar's best values over depth are 0.2781 in the spiked chain (depth 10), far above the plain best, and 0.0662 in the power-law chain (depth 60), slightly below it.

So rescaling is insurance against an unknown stopping depth, at a price. With the label-free scalar, no stage up to depth 200 in the spiked chain, or up to depth 1000 in the power-law chain, has higher risk than the teacher (0.3364 and 0.2353), while the plain stage reaches 0.8024 and 0.4604. The price is paid at the best depth, as above. It is not a selection rule and does not replace one such as Wu, Yang & Sun's iterated GCV.

Oracle versus available information: the per-direction readout needs the prior. The best fixed scalar needs one correlation with the target, which a labeled set estimates. The teacher-fitted scalar needs only unlabeled inputs (in the simulations we compute it with the population covariance). Simulations of the actual estimators (200 trials for the spiked chain, 80 for the power law) give, at depth 200:

Depth 200plainscalar fitted on 30 labelsridge on those 30 labelsscalar fitted on 100 labelsridge on those 100 labels
Spiked0.80690.25550.58710.24440.1866
Power law0.12390.09770.11900.09050.0588

What the labeled simulations show:

  • Spiked chain, 30 labels. Past the best depth, 30 labels fix the scale but are far too few to learn the target alone.
  • Power-law chain, 30 labels. The fitted scalar is within noise of the posterior mean computed from the same labels (0.1024).
  • Near the best depth. A scalar fitted on 10 labels hurts in both chains. At 30 labels it hurts in the spiked chain (depth 20: 0.2404 becomes 0.2502) and is within the standard errors in the power-law chain.
  • 100 labels. At depth 200, a scalar fitted on held-out labels beats a ridge trained on the same labels when they are far fewer than the teacher's 100 (30 here). With 100 labels, a ridge trained on them directly, its penalty tuned on the exact risk (an oracle choice), beats a scalar fitted on the same labels at every depth.
  • Calculus versus simulation. The simulated plain risk matches the calculus within two standard errors at every recorded depth.

The revision adds all of this, caveats included, to Section 5 (paragraph "Recalibrating a deep chain") and Appendix B.2 (Tables 2 and 3). The calculus values agree with direct iteration to 1.2×10−121.2\times10^{-12}.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.