acceptodds
Under review as a conference paper at ICLR 2027

Data-Size-Dependent Supervision in Weak-to-Strong Learning

Abstract

Training high-capacity models to generalize from limited data remains challenging. Teacher-generated supervision can help, but typically relies on a highly capable expert teacher. We study data-efficient weak-to-strong (W2S) generalization, asking whether pretrained weak teachers can effectively supervise stronger students with limited training data. Our experiments show that students can outperform their weak teachers at small data budgets, although their performance can saturate early as more data are added under a fixed teacher. Comparing supervision sources reveals a data-size-dependent crossover: among the sources evaluated, weaker teachers yield the best student performance at the smallest data budgets, while stronger teachers and eventually hard labels become preferable as the data budget increases. To examine this behavior, we characterize asymptotic risks in a two-stage linear regression model. Numerical evaluation of these risks reveals how the balance between bias and variance shifts with student data size, explaining changes in the preferred teacher within this model. Together, these findings highlight the potential of weak teachers for data-efficient learning and show that effective supervision depends jointly on teacher capability and the student’s data budget.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Weak-to-strong generalization, data-efficiency, knowledge distillation, supervision crossover, bias-variance analysis

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_AdrWOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper study the relation between the size of student's training data and the strength of the teacher models. The main finding of the paper can be summarized as the following two phenomenon:

  • (1) The strength of teacher models that weak-to-strong generalization can happen increase with the size of student's training data.
  • (2) The strength of the optimal teacher models increase with the size of student's training data, and eventually the optimal teacher model become the groundtruth label.

The authors conducted experiments on both language and vision tasks to support the above findings, and support the results with theoretical analysis on linear models.

Critical strengths
  • 1.This paper systemetically study the relation between the size of student's training data and the strength of the teacher models, especially experimentally. I think this is an interesting question that worth study.

    1. The presentation of the paper is clear, it is relative easy to follow the main setups and results of the paper.
Critical weaknesses

Experimental results

(1) The setups of different experiments and the theoretical results are not consistent with each other. In particular, in vision experiments on ImageNet and Cifar-100, the student's training data is a subset of teacher's training data. For example, in the ImageNet experiments, the teacher models are ImageNet-1K pretrained AlexNet and ResNet-18, and the student's training data is a subset of ImageNet-1K.

However, in the experiments on language tasks (CLINC150, SCIQ, math reasoning), the student's training data is independent of the teacher's. Moreover, in the theoretical analysis, the student's training data is assumed to be independent of the teacher's.

Note that using dependent student training data could cause different behaviour, since a weak teacher on the test set could be strong on the training set (harmful overfitting), so the quality of teacher's label in this case is not the same as the case where the student's data is independent of the teacher's data. I suggest the authors to make the experiments consistent.

(2) It seems that not all the experimental findings can transfer across tasks and domains. For the vision experiments, despite the data dependency issue in point (1), the experimental results support the main findings under these specific setups. However, on the language experiments, the results are not convincing enough.

In the CLINC150 experiments in Fig2b, the mean seems to support the results, but the standard deviation (error bar) seems to be pretty large, even large than the differences between the gain of two different models. In the SCIQ experiments in Fig 8b, it supports that strength of teacher models that weak-to-strong generalization can happen increase with the size of student's training data, but seems that the hard label is always the best teacher. In the math reasoning experiments in Fig 8c, I do not see either the main findings are supported, since it does not characterize the critical dataset size that weak-to-strong can happen, and seems that either the best model or the original MATH-CoT target is the best teacher.

Theoretical results

(3) The theoreitcal results seem to be directly following from the (Ildiz et.al. 2025), which is not novel. Besides, the paper explains the empirical findings by "weaker supervision tends to yield lower student variance when data are scarce." But this is not well supported by neither the experimental results nor the theoretical results. In particular, on linear model, the paper only conduct one specific setup in Fig 3c, which is not clear whether this explanation is valid across different setups even on linear models. Moreover, the authors does not consider the effect of bias. In the case where the bias dominate the variance, the variance reduction may not be the main factor that cause the phenomenon found in the paper.

Questions for authors
  1. For an explanation of the bias reduction in the w2s generalization, (Medvedev et al., 2025) provides a possible explanation through early stopping, do you expect similar phenomenon can explain the phenomenon found in this paper?
5 comments

We thank the reviewer for their time and feedback.

Critical Weaknesses:

Experimental results

(1) The supervision preference changes with student data size even when teacher and student training images are disjoint.

We agree that overlap can affect the quality of teacher-generated supervision and that the difference between our experimental settings warrants a controlled comparison. We therefore conducted additional ImageNet experiments explicitly comparing overlapping and disjoint teacher–student training sets.

We partitioned ImageNet training images into two non-overlapping pools, A and B, using a fixed class-wise random split. We trained AlexNet and ResNet18 separately on each pool for 90 epochs, from random initialization without pretrained weights. For students trained on subsets of A, teachers trained on A provide overlapping supervision, whereas teachers trained on B provide disjoint supervision.

Teacher ImageNet validation top-1 accuracy (%). Each entry corresponds to one fixed teacher.

Teacher architectureTrained on ATrained on B
AlexNet49.49649.556
ResNet1865.56265.946

All students are ResNet50 models trained from scratch for 200 epochs. Within each data fraction and seed, supervision conditions use the same student subset, initialization, and training configuration. Teacher-supervised students minimize KL divergence to the frozen teacher’s predictions on MixUp inputs, without an additional ground-truth loss; hard-label students use the corresponding mixed ground-truth targets. Data fractions are relative to the original full ImageNet training set, so 50% corresponds to the entire A pool. Results report final-epoch validation accuracy as mean ± sample SD over two student seeds, with teachers and the partition fixed.

Student data(A) fractionHard label (A) $ \quad$AlexNet (B) - DisjointResNet18(B) - DisjointAlexNet(A) - OverlappingResNet18(A) - Overlapping
1%10.484 ± 0.11038.746 ± 0.32036.627 ± 0.68038.463 ± 0.49436.194 ± 0.156
5%36.559 ± 0.14049.624 ± 0.07662.180 ± 0.30550.167 ± 0.16862.151 ± 0.264
20%60.949 ± 0.69751.729 ± 0.00466.755 ± 0.10952.786 ± 0.06267.359 ± 0.001
50%71.808 ± 0.144—67.523 ± 0.075——

With disjoint training sets, AlexNet supervision gives higher student accuracy than ResNet18 supervision at 1%, whereas ResNet18 is preferable at 5% and 20%. At 50%, hard-label training outperforms disjoint ResNet18 supervision. Each comparison holds for both student seeds. The overlapping conditions exhibit the same AlexNet-to-ResNet18 preference change between 1% and 5%.

We also reversed the assignment and trained students on pool B. In this case, teachers trained on A provide disjoint supervision, while teachers trained on B provide overlapping supervision.

Student data(B) fractionHard labels(B) $ \quad$AlexNet(A) - DisjointResNet18(A) - DisjointAlexNet(B) - OverlappingResNet18(B) - Overlapping
1%10.442 ± 0.50638.946 ± 0.88235.652 ± 1.68638.504 ± 0.19034.809 ± 0.700
5%35.700 ± 0.10749.678 ± 0.19262.289 ± 0.11750.388 ± 0.03162.067 ± 0.035
20%60.879 ± 0.49451.636 ± 0.20166.620 ± 0.05152.781 ± 0.01867.481 ± 0.013

The reverse-direction experiments show the same preference change under both disjoint and overlapping conditions: AlexNet supervision yields higher student accuracy at 1%, whereas ResNet18 supervision is preferable at 5%.

These results show that shared teacher–student training images are not necessary for the observed change in supervision preference in this ImageNet setting. They do not imply that overlap has no effect on accuracy, but directly address whether the preference change persists when the training images are disjoint.

Experimental results

(2) Variability in CLINC150 and generality across language tasks.

CLINC150. To address the concern about variability, we report absolute student accuracies over the same five paired runs used in Fig. 2(b), and examine comparisons within each run. Since the plotted gains share a hard-label baseline, their separate error bars do not directly quantify the variability of the difference between two teacher-supervised conditions.

Student test accuracy (%) on CLINC150, reported as mean ± sample standard deviation over five paired runs. Columns indicate the number of training examples per intent. Bold entries identify supervision conditions that achieved the highest student accuracy in at least one of the five runs at the corresponding data budget.

Supervision12348163250
BERT-L4-H25628.95 ± 0.7654.63 ± 1.1164.67 ± 1.0769.95 ± 1.1976.60 ± 0.4578.88 ± 0.8179.33 ± 0.4779.36 ± 0.45
BERT-L6-H25628.11 ± 2.4358.61 ± 1.5469.29 ± 1.3174.73 ± 0.9281.08 ± 0.6083.10 ± 0.6383.38 ± 0.8183.58 ± 0.41
BERT-L12-H25625.74 ± 1.9257.95 ± 1.5073.02 ± 0.8678.70 ± 1.0385.85 ± 0.5787.79 ± 0.6388.60 ± 0.5388.64 ± 0.41
BERT-L8-H51217.20 ± 2.6445.48 ± 3.8069.48 ± 2.7281.01 ± 0.9690.46 ± 0.4493.30 ± 0.2394.13 ± 0.2494.50 ± 0.11
Hard16.78 ± 1.1745.93 ± 3.4568.23 ± 3.7180.88 ± 1.3190.63 ± 0.5194.18 ± 0.2895.48 ± 0.1496.14 ± 0.33

Across runs, supervision from lower-capacity teachers is preferred at the smallest data budgets, while higher-capacity teachers and eventually hard labels become preferable as more student data become available. At some budgets, however, different teacher choices yield similar mean student accuracies, and their relative ordering varies across runs. Thus, the overall change in supervision preference is observed consistently, although the exact ranking of conditions with similar performance can vary across runs.

SciQ. We agree that these results do not demonstrate a clear supervision crossover or a consistent advantage over hard-label training. As stated in Appendix C.2, students can outperform their respective teachers as training data increase, while hard-label training generally matches or exceeds teacher-supervised training. This setting distinguishes outperforming the teacher from outperforming direct ground-truth supervision; we do not treat it as evidence that the crossover occurs universally.

Mathematical reasoning. Figure 8(c) supports a more specific change in supervision preference. At 60 and 120 training examples, all three weak-teacher response conditions outperform training on the original MATH-CoT responses. At 240 examples, MATH-CoT approximately matches the best weak-teacher condition, and it performs best at the larger evaluated training sizes. This shows a change between teacher-generated and reference-response supervision. It does not establish a progressive preference shift among the three Qwen teachers or identify the critical sample size at which W2SG first occurs.

These results indicate that weak supervision is not necessarily beneficial even when student data are limited. Understanding which properties of weak supervision make it helpful for a given student and task, and how their effects depend on student data size, remains an important open question. Our current analysis does not establish general conditions that explain these differences across settings.

Theoretical results

(3) Scope of the theoretical analysis and the roles of bias and variance.

Building on the regression framework of Ildiz et al. (2025), we study teachers estimated from finite data using restricted input coordinates, while the responses are generated from all coordinates. Under the diagonal Gaussian design, omitted signal acts as effective noise during teacher estimation. In the evaluated setting, which fixes the teacher aspect ratio, our numerical analysis shows how the balance between bias reduction and variance cost changes across teacher dimensions as student sample size increases. It also identifies a regime in which W2S has lower risk than both its teacher and ground-truth training at the same student sample size. We agree that Figure 3(c) alone does not establish how broadly its bias–variance explanation applies. We therefore conducted an additional parameter sweep and examined both components, including settings where the teacher-preference crossover is absent.

Additional parameter sweep. We evaluated all 256 combinations of

κw∈1.1,1.25,1.5,2,αs∈1.5,2,2.5,3,αw ∈2,3,4,5,β ∈0,1,2,3,\kappa_w \in \\{1.1,1.25,1.5,2 \\}, \quad \alpha_s \in \\{1.5,2,2.5,3\\}, \quad \alpha_w \ \in \\{2,3,4,5\\}, \quad \beta \ \in \\{0,1,2,3\\},

where λis=i−αs,λiw=i−αw\lambda_i^s = i^{-\alpha_s}, \lambda_i^w=i^{-\alpha_w}, and βi2=iβ\beta_i^2=i^\beta. We fixed p=1000p=1000, used noiseless responses, and evaluated n,pw∈100,150,…,900n,p_w \in \\{100,150,\ldots,900\\}, with m=round⁡(pw/κw)m=\operatorname{round}(p_w / \kappa_w). These are numerical evaluations of the Gaussian-sequence risk expressions.

The table reports how often the risk-minimizing pwp_w changes as nn varies over the evaluated grid. Each cell summarizes the 16 combinations of κw\kappa_w and β\beta for the indicated αw,αs\alpha_w, \alpha_s.

αw\αs\alpha_w \backslash \alpha_s1.522.53
27/160/160/160/16
314/1614/1611/162/16
416/1616/1615/1614/16
516/1616/1616/1616/16

Teacher preference changes in 173 of the 256 configurations. In 172 of these, the preferred dimension is nondecreasing with (nn); one exhibits a nonmonotone path. In the remaining 83 configurations, the preferred dimension stays constant over the evaluated range. Thus, the preference change occurs across multiple settings, but is not universal.

Bias is part of the explanation. Throughout the evaluated settings, increasing (pwp_w) lowers the W2S student's squared bias at a fixed student sample size. A smaller teacher dimension therefore comes with higher bias, and is preferable only if the reduction in student variance outweighs this increase in squared bias. We compare teachers by examining how both components change between the corresponding students. Even when bias accounts for most of the total risk, differences in variance can still affect which teacher is preferable.

Dependence on parameter settings. Across the evaluated grid, changes in the preferred teacher dimension are more common at larger αw\alpha_w and smaller αs\alpha_s. The bias–variance components illustrate how different outcomes arise. In the main setting, smaller teacher dimensions incur higher squared bias but provide a variance reduction that outweighs this cost at small student sample sizes. As student data increase, the balance favors larger dimensions. In contrast, changing only αw\alpha_w from 4 to 2, while keeping αs=2,β=2\alpha_s=2, \beta=2, and κw=1.25\kappa_w=1.25 makes both squared bias and variance decrease with pwp_w throughout the evaluated grid, so pw=900p_w=900 is preferred at every evaluated nn. This comparison shows that the variance advantage of a smaller teacher is parameter-dependent, and that teacher preference must be assessed together with the corresponding bias cost.

Reviewer_GPTYOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the question that when the student's training data is limited, should the student learn from a strong teacher, a weak teacher, or from the true label. It observes that when student's training data is very limited, a weak teacher may be the best option; as the student's training data size increases, it should use a strong teacher and eventually use the true label. This phenomenon is explained by numerical computations of a two-phase linear regression task, and empirical observations on many other deep learning tasks.

Critical strengths

The question raised in this paper is clearly stated and valuable.

The experiments are comprehensive and fair: they include different data sizes, the teacher baseline, students learning from different teachers, and observation on many different tasks.

Critical weaknesses

The theory aspect of the paper is very limited. After reading this paper, I have a feeling that I understand what the phenomenon is, but totally have no idea why it happens. "What is the optimal teacher" is not justified rigorously, so why the phenomenon happens is not rigorously justified, even for the simplest two-phase linear task. The linear task is also too specific. The characterization of the strength of a teacher in the linear task does not apply to real deep learning settings. The theory introduces Rˉα\bar{R}_\alpha, but it is not rigorous and clear why you can substitute the true expected risk by this quantity. This quantity is not analyzed rigorously either. Only showing some numerical experiments for this quantity is not enough.

Questions for authors

Can you establish rigorous theory for the phenomenon you observed in your paper? (Justify the optimal teacher for a fixed training data size.)

2 comments

We thank the reviewer for their time and feedback.

Our analysis numerically examines how teacher dimension and student sample size jointly affect W2S risk within a two-stage linear regression model. In the main setting, smaller teacher dimensions are preferable at low data because their variance advantage outweighs their higher squared bias. With more student data, the bias reduction from larger dimensions becomes more important relative to the variance cost. The reported preferred dimension minimizes the Gaussian-sequence risk over the evaluated grid. These results explain the observed preference through its bias and variance components within this setting, but do not constitute a general theorem characterizing the optimal teacher for a fixed sample size. Here, (pwp_w) controls the teacher’s access to input coordinates; it is not intended as a general characterization of teacher strength in deep networks.

Regarding (Rˉα\bar{\mathcal R}_{\alpha}), we evaluate the expected risk of the corresponding Gaussian-sequence model, following the framework of Ildiz et al. (2025). Appendix D.2 derives these risk expressions, which we use as an analytical approximation to finite-regression risk. Figure 3(a) provides numerical evidence of agreement between these risk expressions and finite-regression simulations at the evaluated parameter values.

The authors did not address my concerns.

Reviewer_cWCNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper systematically investigates how the preferred teacher size in knowledge distillation (KD) changes with the available data budget. First, experiments on ImageNet demonstrate that distillation from a weaker teacher can outperform training with hard labels when the student has access to limited training samples. Next, the paper theoretically investigates this phenomenon using an overparameterized two-stage linear regression model. Finally, the authors decompose the asymptotic risk into bias and variance terms to explain the observed shift in teacher preference.

Critical strengths

This paper provides extensive empirical evaluations across various settings, including linear regression, ImageNet (including linear probing), CIFAR-100, and CLINC150 (BERT). These experiments provide broad empirical evidence of a crossover in the preferred source of supervision as the available data budget increases.

Critical weaknesses
  • Limited Novelty. Ben-Baruch et al. [1] have already reported that smaller teachers can be preferable under low retained-data fractions across diverse experimental settings. Furthermore, this paper adopts the regression framework introduced by Ildiz et al. [2] as its primary analytical tool. Therefore, the authors should more clearly distinguish their contributions from these prior works and clarify what new insights this study provides beyond the existing findings.

  • Unclear Contribution of the Bias–Variance Analysis. One of the main contributions of this paper appears to be connecting the data-budget-dependent shift in teacher preference to a decomposition of asymptotic (or empirical) risk into bias and variance terms. However, the ImageNet experiments and the linear regression analysis exhibit different trends in their bias terms. This discrepancy makes it difficult to understand how the theoretical analysis explains the empirical observations and what the central takeaway of the decomposition is. Further discussion of these differences would help clarify the contribution and implications of the proposed analysis.

[1] Ben-Baruch et al., "Distilling the knowledge in data pruning." ICML (2025).
[2] Ildiz et al., "High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws." ICLR (2025)

Questions for authors
  • Could the authors provide additional experiments that vary (κ\kappa), or fix the number of teacher training samples (mm) while varying the accessible feature dimension (pwp_w)? The former would help establish whether the theoretical observations hold across different degrees of overparameterization. The latter would provide a setting more closely aligned with the ImageNet experiments, where the teacher is pretrained on a fixed dataset rather than being trained with a varying number of samples.

  • Could the authors align the axis definitions and scales of Figure 3(c) and Figure 4 to facilitate a more direct comparison between the two results?

3 comments

We thank the reviewer for their time and feedback.

Critical Weakness:

  • Limited Novelty

    We agree that Ben-Baruch et al. have already observed that smaller teachers can be preferable when student data are limited. Our regression analysis builds on the framework of Ildiz et al. Our additional findings concern how student bias and variance vary across supervision sources and student data sizes.

    Empirically, our cross-entropy bias–variance decomposition on ImageNet shows that, at the smallest evaluated data fraction, the weakest teacher yields both lower student bias and lower variance. As student data increase, variance differences between teachers diminish, and stronger teachers become preferable primarily through lower bias. In this setting, the low-data advantage of weaker supervision thus comes from improvements in both components, and the components that distinguish the preferred teachers change with student data size.

    Within the regression framework, we study teachers estimated from finite data using restricted input coordinates, while the responses are generated from all coordinates. Under the diagonal Gaussian design, omitted signal acts as effective noise during teacher estimation. In the evaluated setting, which fixes the teacher aspect ratio, our numerical analysis shows how the balance between bias reduction and variance cost changes across teacher dimensions as student sample size increases. It also identifies a regime in which W2S has lower risk than both its teacher and ground-truth training at the same student sample size.

    These analyses provide empirical measurements and a linear-model illustration of data-dependent teacher preference. However, the linear model captures the changing bias–variance balance but does not explain the lower bias observed under weaker supervision in the smallest-data ImageNet setting.

  • Unclear Contribution of the Bias-Variance Analysis

    We agree that the bias ordering differs between the linear and ImageNet analyses. On ImageNet, stronger teachers do not consistently yield lower bias at the smallest data fractions, and the weakest teacher has both the lowest bias and variance at 1%. Nevertheless, both analyses show that stronger supervision becomes more favorable as student data increase, through changes in the relative contributions of bias and variance. In the main linear setting, smaller teacher dimensions are preferable at low data because their variance reduction outweighs their higher squared bias; with more data, larger dimensions provide a greater bias reduction relative to the variance cost. Similarly, on ImageNet, variance differences among teachers diminish as data increase, while bias under stronger-teacher supervision decreases more substantially. Thus, the linear model captures the overall shift toward stronger supervision, while not fully explaining the empirical bias behavior at low data.

Questions For Authors:

  • Question 1. Could the authors provide additional experiments that vary (κ\kappa), or fix the number of teacher training samples (mm) while varying the accessible feature dimension (pwp_w)? The former would help establish whether the theoretical observations hold across different degrees of overparameterization. The latter would provide a setting more closely aligned with the ImageNet experiments, where the teacher is pretrained on a fixed dataset rather than being trained with a varying number of samples.

    • Weak-to-strong generalization persists with fixed teacher training data. We fix the teacher training sample size at $ m \in \{80,200 \} ,whilevaryingtheaccessibleteacherdimension(, while varying the accessible teacher dimension (p_w>m$) and student sample size (nn). All other parameters are unchanged from the main analysis. Students can still outperform their weak teachers with limited training data. Within each fixed-(mm) setting, the first evaluated student sample size at which W2SG occurs tends to increase as teacher risk decreases, among teachers for which a crossing is observed. Thus, fixing the teacher training sample size preserves the possibility of low-data W2SG and the observed tendency for stronger teachers to require more student data before being surpassed.

    • The preferred teacher dimension remains unchanged across student data sizes. In contrast to the fixed-aspect-ratio setting, the largest evaluated dimension, (pw=900p_w=900), yields the lowest W2S risk throughout the evaluated range of (nn) in both fixed-(mm) settings. With (mm) fixed, increasing (pwp_w) gives the teacher access to more coordinates but no additional training examples. In these settings, the newly accessible signal components remain strongly attenuated in the teacher’s mean prediction, so the student’s squared bias changes only modestly. Meanwhile, observing more coordinates reduces the omitted signal that acts as noise during teacher estimation. The resulting student variance decreases, making larger teacher dimensions preferable even at small (nn). Under the original fixed κw\kappa_w, increasing (pwp_w) also increases (mm), allowing the teacher to recover substantially more signal. When student data are scarce, this produces a limited reduction in student bias but a substantial variance increase; with more student data, the bias reduction becomes sufficient to favor larger teachers. Thus, the crossover depends on how teacher dimension and training sample size vary together in the evaluated settings, whereas low-data W2SG can occur under either protocol.

  • Question 2. Could the authors align the axis definitions and scales of Figure 3(c) and Figure 4 to facilitate a more direct comparison between the two results?

    • The additional figure replots the linear-model results with student sample size on the horizontal axis and separate panels for total risk, squared bias, and variance, following the organization of Figure 4. Colors distinguish the teacher dimensions pw=100,500,900p_w=100,500,900, and all three panels share the same horizontal and vertical axis ranges. This makes it easier to compare how the components change with student data size across the two settings.

All the additional figures about Question1 and 2 are provided in 'rebuttal_cWCN' folder in the updated supplementary material.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.

However, despite the clarifications, my concerns regarding the novelty and contribution of the paper remain. As bias–variance decomposition is a standard analytical tool in machine learning theory, I find it difficult to identify a sufficiently novel insight or a clear, coherent takeaway from the proposed analysis. In particular, I remain unconvinced that the theoretical and empirical analyses together provide a consistent and meaningful explanation of the observed teacher-preference crossover.

Therefore, I have decided to maintain my original rating.

Reviewer_rFxtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies weak-to-strong generalization across student data sizes. Experiments show that students can surpass weak teachers with limited data, while stronger teachers and ground-truth labels become preferable with more data. A two-stage linear regression model examines the associated bias–variance tradeoff.

Critical strengths

The phenomenon is interesting and empirically validated across multiple tasks.

Critical weaknesses

The theoretical analysis under linear model and the Gaussian assumption is too narrow, and it lacks an explanation of the theoretical intuition.

Questions for authors
  1. The empirical findings are similar to those of Baruch et al. [1]. Beyond the theoretical analysis, what additional information can this paper provide compared with [1]? In my view, W2SG can also be understood as a generalized form of distillation.
  2. The paper states that "the variance increase outweighs the bias improvement." Dong et al. (2025) [2] also mainly focus on a variance-dominated scenario. However, Xu et al. [3] and Chen et al. (2024) [4] observe that bias reduction is the main source of W2SG gains. Combined with your Fig. 3(c), I guess there may be some condition that drives this transition. Please discuss your understanding, and it would be better if you could analyze it within your theoretical.
  3. I suggest moving some of the results on language tasks from the appendix into the main text, rather than including only the results on vision tasks.

[1] Distilling the Knowledge in Data Pruning.

[2] Discrepancies Are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension.

[3] Weak-to-Strong Generalization via Bregman Bias–Variance Decomposition.

[4] On Bias-Variance Alignment in Deep Models.

2 comments

We thank the reviewer for their time and feedback.

Questions for author

Question 1. The empirical findings are similar to those of Baruch et al. [1]. Beyond the theoretical analysis, what additional information can this paper provide compared with [1]? In my view, W2SG can also be understood as a generalized form of distillation.

We agree that our weak-to-strong setting falls within the broader framework of knowledge distillation, and that Ben-Baruch et al. already observed that smaller teachers can be preferable when student data are limited. Our additional empirical contribution is to characterize how student bias and variance change with teacher strength and student data size.

Ben-Baruch et al. interpret the preference for smaller teachers in terms of the teacher–student capacity gap. We examine how student bias and variance change across teacher strengths and student data sizes through an empirical cross-entropy decomposition on ImageNet.

Question 2. The paper states that "the variance increase outweighs the bias improvement." Dong et al. (2025) [2] also mainly focus on a variance-dominated scenario. However, Xu et al. [3] and Chen et al. (2024) [4] observe that bias reduction is the main source of W2SG gains. Combined with your Fig. 3(c), I guess there may be some condition that drives this transition. Please discuss your understanding, and it would be better if you could analyze it within your theoretical.

The quoted sentence, “the variance increase outweighs the bias improvement,” concerns why the preferred teacher changes with student data size. Figure 3(c) compares students trained with different teacher dimensions. With limited student data, increasing the teacher dimension incurs a variance cost that outweighs its bias benefit. With more student data, the bias benefit becomes more substantial relative to the variance cost, making larger teachers preferable.

The gain of a W2S student over its own weak teacher is a separate comparison. In the Gaussian-sequence model we analyze, the W2S student does not have lower squared bias than its teacher as known in Figure 13. Therefore, whenever the student outperforms its teacher, the improvement comes from variance reduction that outweighs the additional bias.

Thus, student data size changes the preferred teacher in our evaluated setting, while gains over the weak teacher arise from variance reduction. Our current linear analysis does not explain the bias-driven W2SG reported in other settings.

Question 3. I suggest moving some of the results on language tasks from the appendix into the main text, rather than including only the results on vision tasks.

The main text includes language-classification results on CLINC150 in Figure 2(b), using a pretrained BERT-Large student supervised by smaller BERT teachers. Additional results on SciQ and mathematical reasoning are provided in the appendix.

Thank you for the response. It addresses some of my concerns, but the additional contribution beyond Ben-Baruch et al. remains limited. I will maintain my original rating.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion deadline (Nov 18, AoE) approaches, please review the authors' rebuttal if you have not done so, and post a follow-up indicating whether your main concerns have been addressed or whether any issues remain.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.