- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies weak-to-strong generalization in random-feature networks, where a finite-width teacher is trained on ground-truth labels and an infinitely wide student is trained on the teacher's predictions. The authors derive deterministic expressions for the teacher and student population errors and use a bias–variance decomposition to explain the role of early stopping. The main result shows quadratic improvement for pure spherical-harmonic targets. The paper also studies different stopping times, mixed targets, and extensions to other activation functions.
Critical strengths
- Clear theoretical setting.
- Interesting quadratic improvement result.
- Useful mixed-target examples.
Critical weaknesses
- Section 6 claims that, for and , $ \overline L_{\mathrm{ST}}(T_\gamma) \lesssim d^{k_0-\alpha}\overline L_{\mathrm{TE}}. $ I do not think this rate holds throughout the stated stopping-time window.
For example, consider , with a target having nonzero degree-one and degree-two components of comparable energy. The teacher error is of order . At , the degree-four contribution to the variance trace gives , so the student variance alone is at least of order . However, Eq. (6.1) would imply a student error of order at most .
This also seems consistent with the nonquadratic regime described in Theorem 4. In Appendix D.2, the variance is shown to be small compared with the teacher error, but that does not necessarily make it smaller than the bias or establish the particular rate in Eq. (6.1).
The weaker conclusion that polynomial improvement is possible may still hold. My concern is with the claimed quantitative rate and its interpretation in Figure 3.
- Claim 2 states that, for even k, .
However, the ReLU harmonic coefficients give .
For fixed k, this has the claimed order. But if k=d with even d, the ratio approaches 1/4, rather than zero.
My concern is that the proof of Theorem 2 also considers harmonic degrees growing with d, so the fixed-degree estimate cannot be applied uniformly over those ranges. This does not necessarily invalidate Theorem 2, but the high-degree tail estimates need a different justification.
Questions for authors
-
Could you clarify how the variance term is controlled for ? In particular, how does the claimed bound apply when ? Would a stopping-time-dependent rate be more appropriate?
-
Is the eigenvalue-ratio bound intended only for fixed harmonic degrees? If so, how are the contributions from degrees growing with d controlled in the proof of Theorem 2?
-
What additional assumptions or concentration estimates would be required to obtain relative deterministic-equivalent accuracy at the quadratic student-error scale? Could the authors further clarify which conclusions are established for the original random-feature model?
3 comments
- What additional assumptions or concentration estimates would be required to obtain relative deterministic-equivalent accuracy at the quadratic student-error scale? Could the authors further clarify which conclusions are established for the original random-feature model?
What is established for the original model. For the random model itself, the only established statement is Medvedev et al.'s lower bound (their Thm 4.3), which applies to our setting. All our other results are theorems about the DE. They transfer to the random model under Assumption 1, which is exactly the relative accuracy you describe, for both losses at the stopping times.
What a proof would take. We do not prove Assumption 1. With population training and an infinitely wide student, the only randomness is the teacher's weights , i.i.d. uniform on , and both losses are functionals of the i.i.d. random features . We expect a proof to follow the route used for multiplicative deterministic equivalents in random-feature ridge regression (Defilippis, Loureiro & Misiakiewicz 2024; Misiakiewicz & Saeed 2024):
- Leave-one-out. Removing one neuron at a time turns the resolvent functionals that define the losses into an approximate fixed-point (Dyson) equation, whose error terms are fluctuations of quadratic forms in a single feature. Equivalently, in a cumulant expansion (App. B, Eq. (B.4)), the second cumulants give the Dyson equation and the higher cumulants are the error terms.
- Concentration. For spherical features the quadratic forms are controlled degree by degree: hypercontractivity of spherical harmonics handles the fixed low degrees, and the high-degree part concentrates near a multiple of the identity (Mei, Misiakiewicz & Montanari 2022). The log-Sobolev inequality on the sphere gives concentration of the losses around their means.
- Stability. Stability of the fixed-point equation for turns the approximate equation into bounds on both losses. To reach relative error at the scale of the student error, this has to be done multiplicatively, as in the works above, rather than with additive bounds.
For the teacher, the infinite-data limit of Defilippis et al.'s bound is our teacher deterministic equivalent, so what remains there is checking their hypotheses in our polynomial scaling. The student is the new part: it is a time-dependent two-resolvent functional, and the error must be relative at the scale .
Where the difficulty lies. The demanding requirement is relative accuracy at the scale of the student error, . This is hardest in the variance-limited phases, where the random-matrix variance term itself must be captured to leading order. Each step is standard; the work is in carrying the error terms through the two-stage, double-resolvent structure. This is a substantial undertaking that we leave open. The paper's population simulations agree with the DE.
Thanks for your response, it addressed my concerns. I will raise my score to 3.
Thank you for the careful feedback.
You are right. The variance term dominates when γ>(α+k0)/2, and we wrongly dropped it. The corrected rate (see the revisions we made in the paper updated draft) depends on the stopping time, as you suggest: LST(Tγ)≲(dk0−α+d2γ−2αlog2d) LTE. As you anticipated, polynomial improvement holds throughout the window. The corrected text of Section 6 is in our response to UhSz, and Fig. 3 and App. D.2.1 are corrected accordingly.
Thank you. The ratio ((k−1)/(d+k+1))2 is exact; our proof dropped a factor (k−1)2. We restated Claim 2 with the exact ratio. It is ≲d−2 for fixed k, and <1 for all k, so the sequence is decreasing.
The proof of Theorem 2 used the claim for growing degrees only in the tail terms (iii) and (iv). We now split these at k=logd instead of k=d:
The same repair applies to the corresponding terms in Theorem 4. The conclusions of Theorems 2 and 4 are unchanged.
Yes, the stopping-time-dependent rate above is the correct one. The variance is V(Tγ)≍d2γ−2αLTE, up to a factor log2d when a degree lies in (γ,α]. It comes from the degrees k∈(γ,α] that the student has partly learned by time Tγ, together with the degree γ itself.
At (5,2,4) with your two-mode target:
So LST(T4)≍d−5. This matches your lower bound and Theorem 4, and is still polynomial improvement over d−3.
Yes, the d−2 bound is for fixed degrees. Growing degrees are controlled as described under Weakness 2: the exact ratio up to logd, then monotonicity and Parseval beyond.