- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies weak-to-strong generalization when the weak teacher knows a hard feature only partially, instead of not knowing it at all. In their data model, samples contain an easy feature, a hard feature, or both. The teacher always labels samples with the easy feature correctly. On samples with only the hard feature, it is right more often when the feature is stronger (larger ). For a two-layer ReLU student trained by gradient descent, the authors show three things. Early in training, the student's response to the hard feature approaches a level set by a strength-weighted ratio between correctly and wrongly labeled hard-only samples. Later, the student memorizes the wrong labels through each sample's own noise, while keeping part of the hard feature. As a result, the test error still goes to zero after the student fits all the weak labels.
Critical strengths
- A natural question: real weak teachers usually know hard features partially, not completely or not at all.
- The theory covers the whole training process, and synthetic experiments match it closely.
Critical weaknesses
- The hard feature is only a weaker linear direction, and teacher errors are random noise, unlike real hard features.
- Novelty is limited. With random flips below 1/2, two-layer ReLU networks learning the signal while memorizing wrong labels through sample-specific noise was already shown in [1]. The setup of easy, hard and overlap features also follows [2]. The new part is mainly that the flip probability depends on .
[1] Y. Kou, Z. Chen, Y. Chen, and Q. Gu. Benign Overfitting in Two-layer ReLU Convolutional Neural Networks. In ICML, PMLR 202:17615–17659, 2023.
[2] J. Oh, J. Song, and C. Yun. From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning. In NeurIPS, 2025.
Questions for authors
Which steps in Appx. C are new compared with [1] and [2]? Are Lemmas C14 and C15 new?
[1] Y. Kou, Z. Chen, Y. Chen, and Q. Gu. Benign Overfitting in Two-layer ReLU Convolutional Neural Networks. In ICML, PMLR 202:17615–17659, 2023.
[2] J. Oh, J. Song, and C. Yun. From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning. In NeurIPS, 2025.
2 comments
We sincerely thank all reviewers for their valuable comments and constructive feedback on our paper. As the reviewer discussion period is coming to an end, we would greatly appreciate it if you could take another look at our responses and share any further feedback or concerns.
Response to Reviewer WCbY
We sincerely thank the reviewer for the constructive feedback. Please see our responses to the reviewer’s questions.
We represent the hard feature by a single direction so that we can track how much of it the two-layer ReLU student learns and retains. A feature is hard in our model because the teacher has learned it only partly. Limited model capacity or differences between the teacher’s and student’s training distributions can leave the teacher with a small response bh along that direction, even when the feature is clearly expressed in some samples.
Noise memorized during teacher training can also affect predictions on new samples. This motivates the output-noise term ϵ, which we model as independent across samples. On a hard-only sample, the teacher’s score aligned with the true label is 2αbh+ϵ. When the feature strength α is smaller, the same noise distribution is more likely to overwhelm the hard-feature contribution and flip the label. The probability of an error therefore depends on the sample’s feature strength.
Table 1 in Common Response C1 shows that trained Qwen teachers are less accurate on hard-only samples than on easy-only and overlap samples, while their average hard-only accuracy remains above chance. Table 2 also shows that average teacher accuracy is higher in the highest-α quintile than in the lowest on each dataset. These results support the kind of partial knowledge our model captures. Modeling shared misconceptions would require accounting for how teacher errors are related across samples.
Our contribution goes beyond making the flip probability depend on α: we quantitatively connect early hard-feature saturation to later preservation when clean and flipped feature updates compete. Lemmas C14 and C15 provide the key preservation arguments for this regime.
As the reviewer notes, [1] establishes benign overfitting in two-layer ReLU networks: the network learns a shared signal while fitting incorrect labels through sample-specific noise. Its data model assumes random label flips with probability below 1/2, while its theorem requires p≤1/C for a sufficiently large constant C (Condition 4.1). The signal coefficient increases throughout the analyzed interval (Proposition 5.2). Our model instead represents partial teacher knowledge through label correctness that depends on hard-feature strength. In our regime, supporting and opposing hard-feature updates can be comparable, leading to early saturation and a feature response that may subsequently decrease during noise memorization. Thus, the continued feature growth established in [1] does not provide the preservation guarantee needed here.
We adopt an easy/hard/overlap construction similar to [2], but study a different source of supervision. In [2], a linear teacher cannot recognize the nonlinear hard-feature patterns, and correctly labeled overlap samples supply the hard-feature learning signal. Its data-scarce proof relies on clean hard-feature updates dominating, while its data-abundant result guarantees generalization at an early stopping time before substantial sample-specific noise memorization. In our setting, partial teacher knowledge makes hard-only labels informative even without overlap samples. We ask how much knowledge the student extracts from these competing labels and retains during memorization.
Lemmas C14 and C15 supply the two technical steps needed to address this new question. Lemma C14 establishes a noise-response ordering: the smallest accumulated noise response among flipped hard-only samples is at least as large as the largest among clean hard-only samples. This comparison matters because noise memorization weakens both the opposing updates from flipped samples and the supporting updates from clean samples. Lemma C15 combines this ordering with R>1 to control both effects and establish a positive lower bound on the hard-feature response during memorization. Under our assumptions, this bound and the early saturation level both scale with logR. This allows us to quantify the fraction of early hard-feature knowledge retained as the student fits the weak labels.
[1] Y. Kou, Z. Chen, Y. Chen, and Q. Gu. Benign Overfitting in Two-layer ReLU Convolutional Neural Networks. ICML, 2023.
[2] J. Oh, J. Song, and C. Yun. From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning. NeurIPS, 2025.