- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper study the relation between the size of student's training data and the strength of the teacher models. The main finding of the paper can be summarized as the following two phenomenon:
- (1) The strength of teacher models that weak-to-strong generalization can happen increase with the size of student's training data.
- (2) The strength of the optimal teacher models increase with the size of student's training data, and eventually the optimal teacher model become the groundtruth label.
The authors conducted experiments on both language and vision tasks to support the above findings, and support the results with theoretical analysis on linear models.
Critical strengths
-
1.This paper systemetically study the relation between the size of student's training data and the strength of the teacher models, especially experimentally. I think this is an interesting question that worth study.
-
- The presentation of the paper is clear, it is relative easy to follow the main setups and results of the paper.
Critical weaknesses
Experimental results
(1) The setups of different experiments and the theoretical results are not consistent with each other. In particular, in vision experiments on ImageNet and Cifar-100, the student's training data is a subset of teacher's training data. For example, in the ImageNet experiments, the teacher models are ImageNet-1K pretrained AlexNet and ResNet-18, and the student's training data is a subset of ImageNet-1K.
However, in the experiments on language tasks (CLINC150, SCIQ, math reasoning), the student's training data is independent of the teacher's. Moreover, in the theoretical analysis, the student's training data is assumed to be independent of the teacher's.
Note that using dependent student training data could cause different behaviour, since a weak teacher on the test set could be strong on the training set (harmful overfitting), so the quality of teacher's label in this case is not the same as the case where the student's data is independent of the teacher's data. I suggest the authors to make the experiments consistent.
(2) It seems that not all the experimental findings can transfer across tasks and domains. For the vision experiments, despite the data dependency issue in point (1), the experimental results support the main findings under these specific setups. However, on the language experiments, the results are not convincing enough.
In the CLINC150 experiments in Fig2b, the mean seems to support the results, but the standard deviation (error bar) seems to be pretty large, even large than the differences between the gain of two different models. In the SCIQ experiments in Fig 8b, it supports that strength of teacher models that weak-to-strong generalization can happen increase with the size of student's training data, but seems that the hard label is always the best teacher. In the math reasoning experiments in Fig 8c, I do not see either the main findings are supported, since it does not characterize the critical dataset size that weak-to-strong can happen, and seems that either the best model or the original MATH-CoT target is the best teacher.
Theoretical results
(3) The theoreitcal results seem to be directly following from the (Ildiz et.al. 2025), which is not novel. Besides, the paper explains the empirical findings by "weaker supervision tends to yield lower student variance when data are scarce." But this is not well supported by neither the experimental results nor the theoretical results. In particular, on linear model, the paper only conduct one specific setup in Fig 3c, which is not clear whether this explanation is valid across different setups even on linear models. Moreover, the authors does not consider the effect of bias. In the case where the bias dominate the variance, the variance reduction may not be the main factor that cause the phenomenon found in the paper.
Questions for authors
- For an explanation of the bias reduction in the w2s generalization, (Medvedev et al., 2025) provides a possible explanation through early stopping, do you expect similar phenomenon can explain the phenomenon found in this paper?
5 comments
Experimental results
(2) Variability in CLINC150 and generality across language tasks.
CLINC150. To address the concern about variability, we report absolute student accuracies over the same five paired runs used in Fig. 2(b), and examine comparisons within each run. Since the plotted gains share a hard-label baseline, their separate error bars do not directly quantify the variability of the difference between two teacher-supervised conditions.
Student test accuracy (%) on CLINC150, reported as mean ± sample standard deviation over five paired runs. Columns indicate the number of training examples per intent. Bold entries identify supervision conditions that achieved the highest student accuracy in at least one of the five runs at the corresponding data budget.
| Supervision | 1 | 2 | 3 | 4 | 8 | 16 | 32 | 50 |
|---|---|---|---|---|---|---|---|---|
| BERT-L4-H256 | 28.95 ± 0.76 | 54.63 ± 1.11 | 64.67 ± 1.07 | 69.95 ± 1.19 | 76.60 ± 0.45 | 78.88 ± 0.81 | 79.33 ± 0.47 | 79.36 ± 0.45 |
| BERT-L6-H256 | 28.11 ± 2.43 | 58.61 ± 1.54 | 69.29 ± 1.31 | 74.73 ± 0.92 | 81.08 ± 0.60 | 83.10 ± 0.63 | 83.38 ± 0.81 | 83.58 ± 0.41 |
| BERT-L12-H256 | 25.74 ± 1.92 | 57.95 ± 1.50 | 73.02 ± 0.86 | 78.70 ± 1.03 | 85.85 ± 0.57 | 87.79 ± 0.63 | 88.60 ± 0.53 | 88.64 ± 0.41 |
| BERT-L8-H512 | 17.20 ± 2.64 | 45.48 ± 3.80 | 69.48 ± 2.72 | 81.01 ± 0.96 | 90.46 ± 0.44 | 93.30 ± 0.23 | 94.13 ± 0.24 | 94.50 ± 0.11 |
| Hard | 16.78 ± 1.17 | 45.93 ± 3.45 | 68.23 ± 3.71 | 80.88 ± 1.31 | 90.63 ± 0.51 | 94.18 ± 0.28 | 95.48 ± 0.14 | 96.14 ± 0.33 |
Across runs, supervision from lower-capacity teachers is preferred at the smallest data budgets, while higher-capacity teachers and eventually hard labels become preferable as more student data become available. At some budgets, however, different teacher choices yield similar mean student accuracies, and their relative ordering varies across runs. Thus, the overall change in supervision preference is observed consistently, although the exact ranking of conditions with similar performance can vary across runs.
SciQ. We agree that these results do not demonstrate a clear supervision crossover or a consistent advantage over hard-label training. As stated in Appendix C.2, students can outperform their respective teachers as training data increase, while hard-label training generally matches or exceeds teacher-supervised training. This setting distinguishes outperforming the teacher from outperforming direct ground-truth supervision; we do not treat it as evidence that the crossover occurs universally.
Mathematical reasoning. Figure 8(c) supports a more specific change in supervision preference. At 60 and 120 training examples, all three weak-teacher response conditions outperform training on the original MATH-CoT responses. At 240 examples, MATH-CoT approximately matches the best weak-teacher condition, and it performs best at the larger evaluated training sizes. This shows a change between teacher-generated and reference-response supervision. It does not establish a progressive preference shift among the three Qwen teachers or identify the critical sample size at which W2SG first occurs.
These results indicate that weak supervision is not necessarily beneficial even when student data are limited. Understanding which properties of weak supervision make it helpful for a given student and task, and how their effects depend on student data size, remains an important open question. Our current analysis does not establish general conditions that explain these differences across settings.
Theoretical results
(3) Scope of the theoretical analysis and the roles of bias and variance.
Building on the regression framework of Ildiz et al. (2025), we study teachers estimated from finite data using restricted input coordinates, while the responses are generated from all coordinates. Under the diagonal Gaussian design, omitted signal acts as effective noise during teacher estimation. In the evaluated setting, which fixes the teacher aspect ratio, our numerical analysis shows how the balance between bias reduction and variance cost changes across teacher dimensions as student sample size increases. It also identifies a regime in which W2S has lower risk than both its teacher and ground-truth training at the same student sample size. We agree that Figure 3(c) alone does not establish how broadly its bias–variance explanation applies. We therefore conducted an additional parameter sweep and examined both components, including settings where the teacher-preference crossover is absent.
Additional parameter sweep. We evaluated all 256 combinations of
where , and . We fixed , used noiseless responses, and evaluated , with . These are numerical evaluations of the Gaussian-sequence risk expressions.
The table reports how often the risk-minimizing changes as varies over the evaluated grid. Each cell summarizes the 16 combinations of and for the indicated .
| 1.5 | 2 | 2.5 | 3 | |
|---|---|---|---|---|
| 2 | 7/16 | 0/16 | 0/16 | 0/16 |
| 3 | 14/16 | 14/16 | 11/16 | 2/16 |
| 4 | 16/16 | 16/16 | 15/16 | 14/16 |
| 5 | 16/16 | 16/16 | 16/16 | 16/16 |
Teacher preference changes in 173 of the 256 configurations. In 172 of these, the preferred dimension is nondecreasing with (); one exhibits a nonmonotone path. In the remaining 83 configurations, the preferred dimension stays constant over the evaluated range. Thus, the preference change occurs across multiple settings, but is not universal.
Bias is part of the explanation. Throughout the evaluated settings, increasing () lowers the W2S student's squared bias at a fixed student sample size. A smaller teacher dimension therefore comes with higher bias, and is preferable only if the reduction in student variance outweighs this increase in squared bias. We compare teachers by examining how both components change between the corresponding students. Even when bias accounts for most of the total risk, differences in variance can still affect which teacher is preferable.
Dependence on parameter settings. Across the evaluated grid, changes in the preferred teacher dimension are more common at larger and smaller . The bias–variance components illustrate how different outcomes arise. In the main setting, smaller teacher dimensions incur higher squared bias but provide a variance reduction that outweighs this cost at small student sample sizes. As student data increase, the balance favors larger dimensions. In contrast, changing only from 4 to 2, while keeping , and makes both squared bias and variance decrease with throughout the evaluated grid, so is preferred at every evaluated . This comparison shows that the variance advantage of a smaller teacher is parameter-dependent, and that teacher preference must be assessed together with the corresponding bias cost.
We thank the reviewer for their time and feedback.
Critical Weaknesses:
Experimental results
(1) The supervision preference changes with student data size even when teacher and student training images are disjoint.
We agree that overlap can affect the quality of teacher-generated supervision and that the difference between our experimental settings warrants a controlled comparison. We therefore conducted additional ImageNet experiments explicitly comparing overlapping and disjoint teacher–student training sets.
We partitioned ImageNet training images into two non-overlapping pools, A and B, using a fixed class-wise random split. We trained AlexNet and ResNet18 separately on each pool for 90 epochs, from random initialization without pretrained weights. For students trained on subsets of A, teachers trained on A provide overlapping supervision, whereas teachers trained on B provide disjoint supervision.
Teacher ImageNet validation top-1 accuracy (%). Each entry corresponds to one fixed teacher.
All students are ResNet50 models trained from scratch for 200 epochs. Within each data fraction and seed, supervision conditions use the same student subset, initialization, and training configuration. Teacher-supervised students minimize KL divergence to the frozen teacher’s predictions on MixUp inputs, without an additional ground-truth loss; hard-label students use the corresponding mixed ground-truth targets. Data fractions are relative to the original full ImageNet training set, so 50% corresponds to the entire A pool. Results report final-epoch validation accuracy as mean ± sample SD over two student seeds, with teachers and the partition fixed.
With disjoint training sets, AlexNet supervision gives higher student accuracy than ResNet18 supervision at 1%, whereas ResNet18 is preferable at 5% and 20%. At 50%, hard-label training outperforms disjoint ResNet18 supervision. Each comparison holds for both student seeds. The overlapping conditions exhibit the same AlexNet-to-ResNet18 preference change between 1% and 5%.
We also reversed the assignment and trained students on pool B. In this case, teachers trained on A provide disjoint supervision, while teachers trained on B provide overlapping supervision.
The reverse-direction experiments show the same preference change under both disjoint and overlapping conditions: AlexNet supervision yields higher student accuracy at 1%, whereas ResNet18 supervision is preferable at 5%.
These results show that shared teacher–student training images are not necessary for the observed change in supervision preference in this ImageNet setting. They do not imply that overlap has no effect on accuracy, but directly address whether the preference change persists when the training images are disjoint.