- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when imitation can improve a teacher’s predictions. It attributes improvement to recalibration under a no-extra-information assumption. It derives results on sample budgets, repeated distillation, and scaling laws.
Critical strengths
- The risk decomposition offers a clear conceptual explanation.
- The optimal budget and equal budget depth results are concrete contributions.
- The main derivations appear sound under the stated assumptions.
Critical weaknesses
Main weaknesses:
- Personally speaking, I am not very convinced by the framing of Theorem 2.1 (Imitation cannot create knowledge). The assumptions already make the student a randomized post-processing of the teacher. The Bayes risk inequality then follows from standard data processing. The theorem looks correct, but I do not see much new in the result itself. Calling it “Imitation cannot create knowledge” makes it sound much broader than what is actually proved.
- I am not convinced that the core mechanism in Section 3 is novel. Spectral shrinkage and the accompanying variance term already appear in Ildiz et al. (2025, Eqs. 8–10). The “filter + leak” decomposition in Eqs. (3.2)–(3.4) appears to recover these same quantities. Eqs. (3.6)–(3.7) then reinterpret the risk through a calibration-based decomposition. This perspective may be useful, but calling it “recalibration” does not by itself establish a new mechanism for weak-to-strong generalization. The paper should distinguish this reinterpretation from its new quantitative conditions and multistage results.
- As I read on, I felt the paper drifting further from weak-to-strong generalization. Removing the weak-to-strong generalization framing would leave most of it intact as a paper on linear distillation and regression. The main explanation is already established in Section 3: suitable spectral shrinkage can correct some teacher errors. Yet this mechanism substantially overlaps with Ildiz et al. (2025), as the authors acknowledge. Sections 4–6 add results on budgets, depth, and scaling, but I struggled to see what additional weak-to-strong generalization-specific insights they provide. The mathematics is coherent, and the quantitative results may be useful. Still, I am not convinced that their explanatory value for weak-to-strong generalization matches the prominence of that framing.
- I also missed any validation beyond synthetic linear regression. Figures 1–2 and Appendix N check the theory within its own model, but there are no experiments with actual pretrained language models. These checks support the calculations, yet leave the relevance to LLM-based weak-to-strong generalization unclear. Given the framing, I would have liked a controlled experiment testing whether the predicted budget or depth effects survive in that setting. Otherwise, the claims should stay closer to the linear model actually studied.
Presentation:
- I have read quite a few weak-to-strong generalization papers, but I still struggled with “readout” in the abstract and introduction. I only understood what the authors meant after reaching its definition in Section 2. This is a central concept, so having to read ahead to understand the opening argument is frustrating. Please explain upfront that it means the posterior mean.
- The use of “calibration” also bothers me. As an ML reader, I naturally interpret it as predictive calibration. Here, it depends on the target prior and conditions on the full model parameters. Section 3’s title, “HOW IMITATION RECALIBRATES,” makes this confusion worse. It sounds like a general claim about calibration, while the analysis uses a specific Bayesian definition. I find that framing hard to accept without a clear qualifier. The distinction should be explicit from the beginning, rather than left for the reader to discover later.
- I also find the sentence “When the student brings no knowledge of its own, weak-to-strong generalization is recalibration” too sweeping. The theorem gives an accounting identity under a specific Bayesian definition. It does not establish that “weak-to-strong generalization is recalibration” in the usual ML sense. Calling the entire phenomenon “recalibration” blurs that distinction.
Questions for authors
- What does Theorem 2.1 establish beyond standard data processing, and how does this justify the broader claim that “imitation cannot create knowledge”?
- Which results in Sections 3–6 provide new insights into weak-to-strong generalization beyond the spectral shrinkage mechanism already studied by Ildiz et al. (2025)?
- Can the authors validate the predicted budget or depth effects using pretrained language models, or clarify why these predictions should extend beyond synthetic linear regression?
5 comments
Response to Reviewer UCtS (2/3)
W2 and Q2 (Section 3 versus Ildiz et al.). For the single-stage and two-stage machinery your reading is right:
- spectral shrinkage and its variance term are Ildiz et al.'s Eqs. (8)-(10);
- the three-term two-stage risk of our Theorem 3.2 is their Theorem 2, and our leak law (3.4) is their Eqs. (10) and (16) in other notation;
- Medvedev et al. (their Section 5) describe the same mechanism: the student removes the teacher's energy in low-variance directions that carry no signal.
Theorem 3.2 differs in two respects: the teacher is an interpolating fit restricted to coordinates, and the signal has a prior. The revised paper credits these at the statements themselves (the Theorem 3.2 header, after (3.4), the introduction's description of the leak, and the paragraph after Proposition 3.3). It claims no novelty for the filter-and-leak mechanism.
In the terms of your W2, the results sort into three groups:
- Reading of a known risk. (3.6)-(3.7) rewrite a known deterministic-equivalent risk as the teacher's linear ignorance, plus the calibration error left by the student's mean filter, plus the leak. This reading is what makes the conditions below statable.
- Quantitative conditions, absent from Ildiz et al., Wu, Yang & Sun, Moniri & Hassani and Medvedev et al.:
- which directions of a given minimum-norm teacher are overconfident (Proposition 3.3). Derived from the single-stage law, it locates where a shrinking student can gain: in observed directions whose signal strength is below a threshold that does not involve the eigenvalue;
- the first-order criterion for imitating less than a clone (Proposition 4.1);
- the limiting budget (Theorem 4.2).
- Multistage results. Wu, Yang & Sun already give the chain recursion, denoising followed by forgetting, and noise and signal decaying at different rates. Beyond them, the paper has:
- the two clocks (Proposition B.4): along a chain the linear knowledge share decays at while energy decays at , so a chain on an anisotropic spectrum passes through an underconfident phase. It rests on the exact finite- identity of Lemma 3.1, which gives expected chain risks at every depth from products of fixed matrices;
- the equal-budget certificate (5.6).
The revised related-work paragraph and a new table in Appendix A.3 state this separation.
W1 and Q1 (Theorem 2.1 and data processing). Theorem 2.1 has two parts: an exact identity for the gain, which is the classical calibration-refinement split of squared error in gain form (Murphy, 1973; DeGroot & Fienberg, 1983, both cited in Section 1), and an inequality, which is data processing, as you say:
What the paper adds is their use as a bound on the W2S gain:
- in expected risk under the prior, for every procedure whose extra data are independent of the pair (target, teacher output), the gain is at most the teacher's calibration error;
- the readout attains this bound;
- Sections 2-5 compute the linear part direction by direction, which bounds the gain of every linear student.
You are right that the theorem's old name was broader than its statement. The revision renames the theorem "Imitation on fresh inputs cannot reduce ignorance". The sentence before it now says that the theorem combines orthogonality of the conditional mean with data processing, and that the identity is what the paper uses.
W6 (readout). Agreed. The readout of a predictor is the posterior mean of the target given the predictor. The abstract and Section 1 now say this at first use.
Response to Reviewer UCtS (3/3)
W7 (calibration). Agreed. The submitted abstract mentioned the prior, but the abstract and introduction used the term before Section 2 said that this calibration is Bayesian and conditions on the whole predictor. The abstract and Section 1 now state at first use that calibration here is Bayesian: it averages over a prior on the target and conditions on the whole predictor. Averaged over the prior, it implies calibration of the predicted value ; on a single fixed task it need not. Section 2 now carries the same qualifier on its comparison with calibration of the predicted value, after Theorem 2.1. Section 3 is retitled "Recalibration in the linear model: filter and leak". The overview figure's middle panel is renamed "Filter and leak".
W8 (the sweeping sentence). Agreed. It now reads: "When the student brings no knowledge of its own, its expected gain over its teacher can come only from reducing calibration error, and it never exceeds the teacher's calibration error." That is what Theorem 2.1 states for expected risk under the prior; for a single fixed target, Section 2 notes that the bound is only the teacher's risk. The conclusion's sentence is narrowed in the same way.
W4 and Q3 (pretrained language models). The paper's claims are for the linear model: we have not run LLM experiments, and the revised paper states that its results do not claim that the budget or depth effects carry over to them. In an LLM W2S experiment, the student's pretrained representation may carry information about the target beyond the teacher. The experiment would then measure both routes, and the theory accounts only for recalibration.
The ledger does give one diagnostic there. Since the gain equals the drop in calibration error minus the rise in ignorance, a gain larger than the teacher's calibration error can only come from a drop in ignorance, that is, from knowledge the student brought. Using it requires an estimate of that calibration error, which is defined over a family of tasks drawn from a prior (or slopes pooled across tasks, as in the Discussion); we have not attempted it outside the model.
A test of the recalibration route itself needs a student whose representation carries no information about the target beyond the teacher. The Discussion now states this condition for the measurement it proposes.
If the narrowed framing and the separation above resolve your concerns about the W2S claims, we would appreciate it if you would reconsider your rating.
Response to Reviewer UCtS (1/3)
Thank you for quoting the exact sentences. The revised PDF narrows the sentences you quote, which claimed more about weak-to-strong generalization (W2S) than the theorems support, and defines the terms where they first appear (changes in blue).
W3 (drift from W2S).
What makes the model weak-to-strong is an asymmetry: a linear student that sees all d coordinates is trained on the outputs of a teacher that sees only m<d of them. The gain depends on that width. In the spiked model of Theorem 4.2, at a fixed teacher, the best gain V(1−k/(d−k))2, where V=σ2(nw−k)/(m−nw) is the teacher's error in the bulk and does not involve d, increases with the student's dimension d. The extra coordinates carry no signal, yet at the optimal budget they lower both the share of teacher error the student keeps and the share it leaks; Appendix I.4 now states this. The same calculus covers self-training on fresh inputs, as in Wu, Yang & Sun (cited in the paper as Wu et al., 2026b), which compare a student with its own teacher in the same way; what W2S adds is the width asymmetry.
Linear models with fixed representations are a common theoretical setting for W2S (Ildiz et al., 2025; Medvedev et al., 2025). Moniri & Hassani's Sections 2.2-3 instead study students with informative representations, which is the second route below.
The ledger separates two routes to a gain. A student with a state U′ informative about the target beyond the teacher has a lower floor, K(βw,U′) (remark after Theorem 2.1), and can also gain by elicitation. Our model excludes that route.
The revision makes this scope explicit where the reader meets the claim:
Within that scope, three statements bear on W2S: