- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies weak-to-strong generalization (W2SG) in fully converged, ridgeless two-stage learning. The main observation is that a student can outperform its teacher even when both use the same hypothesis class and training procedure. The authors attribute this to the finite number of pseudo-labeled samples, which can filter out part of the teacher's error. They analyze this effect under power-law covariance and extend the framework to random-feature models, where either sample size or student width can become the active bottleneck.
The paper provides deterministic-equivalent risk expressions and derives phase boundaries for several asymptotic regimes.
Critical strengths
-
Clear and useful mechanism.
-
Substantial scalar analysis.
-
Correct exact geometric results.
-
Informative conditional phase formulas.
Critical weaknesses
- My main concern is the connection between the deterministic-equivalent calculations and the actual ridgeless risks. Several derivations start from positive-ridge deterministic equivalents and then take the ridgeless limit. Since the covariance spectrum also changes with dimension, I am not convinced that the cited results directly justify these limit exchanges.
This matters because the phase diagrams depend on the signs of risk differences, not simply on approximations to the individual risks. It would help to state precisely which approximation guarantees apply in the spectral regimes considered.
- Remark 3.5 states that the fixed-sample and proportional regimes give a complete picture across the possible scalings of m.
I am not fully convinced by this conclusion at the critical exponent . Theorem C.23 gives a zero leading-order gain in the sparse-growing regime, which by itself does not determine the eventual sign. Expanding the finite-p expression in equation (36) suggests competing contributions of orders 1/m and m/p. These become comparable when m is of order , potentially producing an additional transition for some noise levels.
This does not invalidate the stated fixed-m or proportional calculations, but it suggests that the completeness claim in Remark 3.5 may be too strong.
Questions for authors
-
Can the authors clarify which deterministic-equivalent results justify the ridgeless limit under the dimension-dependent covariance spectra? Is uniform control available in the relevant regimes?
-
At , could there be a transition when ? If so, how should Remark 3.5 be modified?
-
For Case II, can the authors clarify whether the phase boundaries are established for the actual expected risk or for the master deterministic equivalent, particularly when the predicted gain vanishes with dimension?
5 comments
Question 3. For Case II, can the authors clarify whether the phase boundaries are established for the actual expected risk or for the master deterministic equivalent, particularly when the predicted gain vanishes with dimension?
Response: Our original Case II analysis followed the deterministic-equivalence framework of Atanasov et al. (2026), which yielded the phase diagram in Theorem 4.3 for the master deterministic equivalent. However, as the reviewer correctly points out, the phase boundaries of the deterministic equivalent do not, by themselves, establish the corresponding sign conclusions for the actual expected risks, particularly when the gain vanishes with dimension.
To address this concern, we have replaced the deterministic-equivalent analysis in Appendix D with a direct finite-dimensional analysis of the actual expected risks, starting from exact risk identities for the two bottleneck regimes. In particular, Theorem D.9 establishes the regular phase diagram directly for the actual expected population risks. The resulting normalized limits yield exactly the same phase boundaries as those stated in Theorem 4.3. Whenever the corresponding limiting numerator is nonzero, its sign determines the eventual sign of the actual expected gain. At exact zeros of the limiting numerator, Theorem D.9 does not by itself determine the lower-order sign. The interpolation boundary and joint critical regime are treated separately in Appendix D.5.
We have made this distinction explicit in Appendix D and will update the corresponding main-text notation in the final revision.
We hope that these additional results and clarifications address the reviewer's concerns and strengthen the theoretical justification of our main results.
Dear wxeL,
We would like to kindly follow up on our response to your review and the corresponding revisions to the manuscript.
We sincerely appreciate the concerns you raised, particularly regarding the justification of the ridgeless limits, the possibility of an additional critical scaling at , and the distinction between deterministic-equivalent and actual expected-risk results in Case II. Your comments prompted us to substantially strengthen the theoretical analysis.
In particular, we have made the following revisions:
-
Case I — Ridgeless analysis: We developed a direct finite-dimensional expected-risk analysis, establishing the required uniform estimates and controlled approximation bounds. The revised derivation no longer relies on exchanging the positive-ridge and high-dimensional limits.
-
Critical exponent : We added a second-order analysis identifying the additional critical window, including the corresponding transition for the actual expected risk improvement. We also revised Remark 3.5 accordingly.
-
Case II — Random-feature model: We replaced the deterministic-equivalent-based phase analysis with a direct finite-dimensional analysis of the actual expected risks. The revised Appendix D establishes the regular phase diagram for the actual expected-risk difference and separately addresses the interpolation boundary and critical scaling regimes.
Importantly, in the latest revision, we have incorporated these changes throughout the manuscript, including the main text, appendices, and experimental section, rather than limiting the revisions to the technical appendices. We have updated the corresponding statements, explanations, and notation to maintain consistency across the paper.
We would be very grateful if you could take another look at our response and the latest revised manuscript and let us know whether these changes adequately address your concerns, or whether any issues remain.
Thank you again for your time, careful assessment, and constructive feedback. We genuinely appreciate your contribution to improving our work.
Best regards,
The Authors
We thank the reviewer for their comments.
General Response
We have revisited the role of the deterministic-equivalence framework of Atanasov et al. in our analysis. While this framework remains useful for identifying the corresponding master deterministic equivalents, we agree with the reviewer that relying on it alone does not fully justify the asymptotic behavior of the actual expected risks.
To address these concerns, we have strengthened Appendices C and D by developing a direct finite-dimensional ridgeless expected-risk analysis (please see the revised manuscript). Specifically, we derive exact risk decompositions and identities for the actual estimators and establish the uniform estimates needed to pass to the relevant asymptotic regimes. This allows us to establish the sign conclusions for the actual expected-risk differences, rather than relying solely on those of their deterministic equivalents.
These additions preserve the phase boundaries and threshold formulas reported in the main text, while strengthening their theoretical justification and clarifying the regimes in which the conclusions hold for the actual expected risks. Finally, we will update the main-text notation by replacing the DE notation with actual-risk notation wherever justified by the new results. The main theorem statements, phase boundaries, and threshold formulas remain unchanged.
We describe the corresponding additions in response to each question below.
Question 1. Can the authors clarify which deterministic-equivalent results justify the ridgeless limit under the dimension-dependent covariance spectra? Is uniform control available in the relevant regimes?
Response: We thank the reviewer for identifying this technical issue. Our original proportional-regime analysis follows the deterministic-equivalence framework of Atanasov et al. (2026). However, as the reviewer correctly points out, invoking this framework alone does not fully resolve the distinction between deterministic equivalents and actual expected risks, particularly when the covariance spectrum changes with dimension and the sign of a difference between two risks is of interest.
To address this concern, we have substantially strengthened Appendix C by deriving the proportional-regime expected-risk formula directly for the finite-dimensional ridgeless estimators. Thus, the revised argument does not require exchanging a positive-ridge limit with the high-dimensional limit.
In particular, Lemmas C.3–C.5 establish the uniform fixed-point, spectral, and quadratic-form estimates, while Theorems C.6 and C.7 establish the weighted pseudoinverse and Stage-II projection estimates required for the direct ridgeless derivation. Theorem C.8 then establishes the finite-dimensional expected-risk decomposition with an explicit, uniformly controlled remainder. Theorem C.10 derives the corresponding power-law asymptotic limit. We also treat the other sampling regimes separately: Theorem C.26 controls the approximation error in the sparse-growing regime, while Theorem C.37 establishes the fixed-sample expected-risk expansion.
Consequently, the revised analysis provides direct control of the actual expected-risk improvement in each regime, rather than relying solely on deterministic-equivalent formulas for the individual risks.
Question 2. At α=2, could there be a transition when m≍p? If so, how should Remark 3.5 be modified?
Response: We agree that the vanishing leading-order gain at α=2 does not determine the eventual sign, and that the competing contributions of orders 1/m and m/p suggest an additional transition. Motivated by this observation, we have added a second-order analysis in Appendix C. Proposition C.25 identifies the critical m≍p window for the scalar approximation. More importantly, Theorem C.30 establishes the corresponding transition for the actual expected gain, accounting for the finite-sample Stage-II variance correction. In particular, it identifies the shifted critical threshold for the actual expected risk.
We have accordingly revised Remark 3.5 to acknowledge this additional critical window and to distinguish the transition predicted by the scalar approximation from the one established for the actual expected gain. Thus, the fixed-sample and proportional regimes alone do not fully describe the behavior at the critical exponent α=2.