acceptodds
Under review as a conference paper at ICLR 2027

When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation

10 views

Abstract

Data pruning reduces the training cost of knowledge distillation (KD). However, the preferred teacher capacity changes with the data budget: smaller teachers can outperform larger ones when limited training data are available. Understanding what drives this shift is important not only for teacher choice but also for identifying which samples are useful for distillation. We analyze teacher supervision by decomposing it into *relational ordering*—the ranking of classes—and *score geometry*—the magnitudes and margins of class probabilities—and show that the small-teacher advantage in the low-data regime arises not only from score geometry but also from relational ordering. Beyond understanding teacher capacity, our analysis reveals two properties of effective subsets: samples should match the difficulty appropriate for the available budget, and their relational signals should be diverse rather than redundant. Based on these findings, we propose **DVA** (**D**ifficulty- and **V**olume-**A**ware data selection for KD), a training-dynamics-free method, which uses a small teacher as a proxy for budget-aware difficulty filtering and class-conditional relational volume maximization. Despite requiring no training dynamics statistics, our method remains competitive with training-dynamics-based methods while consistently outperforming training-dynamics-free baselines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Knowledge Distillation, Data Pruning, Data Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GZ2ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies knowledge distillation with data pruning. It observes smaller teachers can beat larger ones under reduced data budgets. The authors split teacher supervision into relational ordering and score geometry. Hybrid-logit experiments verify both components impact low-data distillation. Drawing on sample difficulty and teacher output diversity analysis, they propose DVA. A small proxy teacher first selects samples of suitable difficulty within the budget. Log-determinant volume maximization then chooses a subset with complementary relational signals. Evaluations run on CIFAR-100, Tiny-ImageNet and ImageNet. DVA exceeds most baselines without student training dynamics across most setups and matches or outperforms some dynamics-based methods at high pruning rates.

Critical strengths
  1. The capacity-budget trend holds across multiple datasets and teacher sizes. Teacher selection thus cannot depend only on teacher-student capacity gaps.
  2. DVA requires no student training dynamics. It consistently outperforms other dynamics-free baselines at heavy pruning and is competitive with several dynamics-based methods.
Critical weaknesses
  1. The rationale for using small models as selection proxies is underdeveloped. Small capacity may not explain proxy effectiveness. Table 18 shows early large-teacher checkpoints work nearly as well as small proxies.
  2. DVA’s main gain originates from difficulty filtering. At a 0.9 drop rate, difficulty filtering contributes 6.09 pp while volume maximization adds just 0.74 pp. Table 7 lacks precise percentiles for Easy, Medium and Hard buckets and omits schedule validation and tuning details.
  3. Data and optimization budgets are coupled. Equal epochs across pruning rates reduce total optimizer steps for heavy pruning. The teacher-capacity crossover may partly come from inadequate optimization rather than data scarcity alone.
Questions for authors
  1. What are the exact percentile bounds for each difficulty bucket in Table 7? Were these thresholds set with a held-out validation set?
  2. Can early large-teacher checkpoints replicate small-teacher low-data advantages when used directly as KD teachers instead of only selection proxies?
  3. Does the teacher-capacity crossover persist if total optimizer steps are fixed instead of training epochs?
4 comments

We thank the reviewer for the constructive feedback and for noting the practical advantages of DVA over training-dynamics-free baselines. We address each concern below.


W1. & Q2. Teacher Capacity vs. Training-Set Fit.

The rationale for using small models as selection proxies is underdeveloped. Small capacity may not explain proxy effectiveness. Table 18 shows early large-teacher checkpoints work nearly as well as small proxies.

Can early large-teacher checkpoints replicate small-teacher low-data advantages when used directly as KD teachers instead of only selection proxies?

A1. A1. In short, none of the evaluated early large-teacher checkpoints matches the fully trained small teacher when used for KD. To investigate this, we evaluated different large-teacher (width-64) checkpoints on CIFAR-100 at a drop rate of 0.9, using each model as both the selection proxy and the KD teacher (10 seeds). The fully trained small teacher achieves substantially higher accuracy (62.58%) than the train/test-accuracy-matched large teachers (52.44%/52.60%), as well as the epoch-20 large teacher (56.90%), which previously showed competitive performance only as a selection proxy. This table has been added to Appendix J.

Teacher (Proxy and KD)T. Train Acc. (%)T. Test Acc. (%)DVA Student Acc. (%)
Large ep1053.8549.5453.97 ± 0.14
Large ep2063.9356.7656.90 ± 0.13
Large ep3064.5356.5355.64 ± 0.32
Large ep4069.3460.0457.14 ± 0.26
Large ep8073.5760.5055.88 ± 0.29
Large ep170 (test-matched)88.5366.7652.60 ± 0.73
Large ep180 (train-matched)92.8468.4252.44 ± 0.88
Small91.8667.4462.58 ± 0.72

W2-1. Contribution of Difficulty Filtering vs. Volume Maximization.

DVA’s main gain originates from difficulty filtering. At a 0.9 drop rate, difficulty filtering contributes 6.09 pp while volume maximization adds just 0.74 pp.

A2. While difficulty filtering contributes more at a drop rate of 0.9, volume maximization consistently provides additional gains across all six data budgets. As shown in Table 16 (Table 19 in the revised manuscript), volume maximization improves accuracy over random selection within the same difficulty-filtered pool by 0.35–1.05 pp (10 seeds), with gains reaching approximately 1 pp at drop rates of 0.7 and 0.8. Furthermore, highest-volume selection outperforms lowest-volume selection by 3.95, 4.06, and 3.18 pp at drop rates of 0.7, 0.8, and 0.9, respectively. These results support the complementary contributions of both components.

W2-2 & Q1. Difficulty Percentile Bounds and Held-Out Validation.

Table 7 lacks precise percentiles for Easy, Medium and Hard buckets and omits schedule validation and tuning details.

What are the exact percentile bounds for each difficulty bucket in Table 7? Were these thresholds set with a held-out validation set?

A3. The original difficulty thresholds were empirically selected based on student test accuracy, without a separate held-out validation set. However, our additional held-out validation experiments confirm a similar budget-dependent difficulty preference. We also clarify the exact percentile bounds below, where the 0th percentile corresponds to the easiest samples. We have updated Table 7 accordingly.

Drop RateCIFAR-100 / Tiny-ImageNetImageNetPool Size
0.3Medium [10%, 90%]Hard [20%, 100%]80%
0.5Medium [20%, 80%]Medium [20%, 80%]60%
0.7Medium [20%, 80%]Medium [20%, 80%]60%
0.8Easy [0%, 40%]Medium [30%, 70%]40%
0.9Easy [0%, 20%]Medium-Easy [15%, 35%]20%

For validation, we retrained both teachers using a class-stratified 45K/5K train/validation split of CIFAR-100 and evaluated different difficulty windows at drop rates of 0.5 and 0.9 (ResNet-18 student, T=4, 3 seeds). The validation results support the original trend: the preferred difficulty region shifts toward easier samples as the data budget decreases. The original window performs best at a drop rate of 0.5, while a nearby window achieves comparable performance at 0.9. We have included this table in Appendix K.

Drop RateDifficulty WindowDVA Val. Acc. (%)DVA Test Acc. (%)
0.5[0%, 60%]70.68 ± 0.1271.65 ± 0.07
[10%, 70%]71.98 ± 0.4572.63 ± 0.11
[20%, 80%] (Original)72.12 ± 0.4772.89 ± 0.40
[30%, 90%]71.91 ± 0.2272.43 ± 0.31
[40%, 100%]70.49 ± 0.4870.58 ± 0.00
0.9[0%, 20%] (Original)49.82 ± 0.7051.25 ± 0.77
[10%, 30%]50.63 ± 0.4851.45 ± 0.88
[20%, 40%]49.33 ± 0.9250.31 ± 0.89
[40%, 60%]42.71 ± 0.3543.38 ± 0.79
[60%, 80%]37.61 ± 0.5738.37 ± 0.43
[80%, 100%]29.61 ± 0.4629.12 ± 0.49

W3 & Q3. Teacher-Capacity Crossover Under Fixed Optimization Budgets.

Data and optimization budgets are coupled. Equal epochs across pruning rates reduce total optimizer steps for heavy pruning. The teacher-capacity crossover may partly come from inadequate optimization rather than data scarcity alone.

Does the teacher-capacity crossover persist if total optimizer steps are fixed instead of training epochs?

A4. We thank the reviewer for raising this important concern. We agree that fixing the number of training epochs across different pruning rates results in unequal optimization budgets, which may affect the observed teacher-capacity crossover.

The teacher-capacity crossover persists even under a fixed optimization budget. To verify this, we fixed the number of optimizer steps across pruning rates to 187,680 steps, equivalent to 240 epochs on the full CIFAR-100 training set, using a cosine learning rate scheduler (T = 4, 3 seeds). The small teacher (ResNet-34 width-8) outperforms the large teacher (ResNet-34 width-64) by 9.88 pp at a drop rate of 0.9, while the large teacher leads by 6.49 pp at a drop rate of 0.1. We have included this result in Appendix N.

Drop RateSmall T.Large T.
0.172.99 ± 0.1279.48 ± 0.27
0.966.69 ± 0.1856.81 ± 1.00
AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer GZ2Z,

Thank you once again for your careful review and constructive feedback. We greatly appreciate your time and effort in helping us improve our work.

As the discussion period draws to a close, we would like to ask whether you have any remaining concerns or questions regarding our responses. We would be happy to address them.

Sincerely,
The Authors

Reviewer_GZ2ZOpenReview ↗

reply to author

I appreciate the authors’ response and their answers to my questions. However, I still think my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it unchanged.

Reviewer_oci3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper tackles the question of why smaller teachers can produce better students than larger ones when the training data is heavily pruned. The authors decompose the teacher’s outputs into relational ordering (class ranking) and score geometry (sorted logit values), and swap these components between a large and a small teacher to show the contribution of each one individually. They also show that the preferred sample difficulty shifts toward easy samples at small budgets, and that subsets with diverse teacher outputs beat redundant ones. Based on this they propose DVA, which keeps the large teacher for distillation but uses a small proxy teacher in two stages. It first keeps only samples whose difficulty fits the data budget, and then greedily selects within each class samples whose proxy outputs are as diverse as possible. Experiments on CIFAR100, Tiny ImageNet, ImageNet, and ViT show the proposed method outperforms training-dynamics-free methods and is competitive with training-dynamics-based ones.

Critical strengths
  1. The paper is generally easy to read and follow, and is written well. The experiments where the authors swap only one factor and keep other ones constant are nice. The hybrid-logit swaps, the temperature sweep, and especially the early-checkpoint ablation are the right kind of experiments for the question being asked, and generally the experiments are extensive, with results averaged over many seeds.

  2. the proposed method is cheap and practical. It needs no student training dynamics, data selection is significantly cheaper compared to other methods, and it consistently beats the other training-dynamics-free baselines.

Critical weaknesses
  1. The main claim of the paper is that the preferred teacher capacity changes with the data budget, and the whole analysis and method are framed around teacher size. The main issue is that capacity is mixed with how strongly the teacher fits the training set. The most relevant line of work (see weakness 2) is barely mentioned and not discussed, and the capacity framing the authors lead with may be an artifact of this confusion rather than a real capacity effect. KD transfers the teacher’s outputs on training samples, and a teacher that has memorized the training set outputs near one-hot distributions there, so its soft labels carry almost nothing beyond the ground truth label (which the student receives anyway, since the loss the authors used to my understanding is 0.5 CE + 0.5 KD, i.e. 0.5 mixing parameter for most of the results in the paper). The results in the paper are consistent with this view. In Table 11 all teachers with accuracy over 99% perform worst at drop rate 0.9, and student performance jumps exactly where train accuracy leaves saturation, i.e. kind of a U shaped behavior in training-set fit rather than a monotone capacity effect (as the authors suggest). Moreover, in App G4 an early checkpoint large teacher of 64 width (same capacity, less fit) recovers nearly all of the small proxy advantage at drop rate 0.9 compared to the converged one, and the authors themselves write that “the training stage of the proxy explains the performance gap", yet the title, abstract, and Sections 1 and 2 keep attributing the behavior to capacity. Table 19 also shows the same behavior. Generally, experiments that would seperate a fit effect from a capacity effect are missing. There are no results with an early checkpoint of the large model used as the distillation teacher at 0.9 drop rate (App G4 only swaps the proxy). Also, for the ImageNet experiments (the only setting where the teachers do not memorize, and there the crossover is much smaller), the authors run at T=1 with no temperature sweep, while the temperature sweep in App F3 is only run in the memorized CIFAR100 regime, where no temperature can recover structure that is no longer in the outputs. So even though the authors show a temperature sweep to potentially fix calibration, it is only done on a single teacher-student pair memorized CIFAR100 setting and not on the other settings. The fix should be no memorization + temperature sweep for calibration.

  2. The paper does not discuss the closest existing line of work, which explains teacher quality in KD through how well the teacher’s probabilities approximate the true class probabilities rather than through its accuracy or size [1,2,3,4]. This perspective directly predicts the paper’s findings, A converged over parameterized teacher is overconfident on its training data [5,6], so its train-set outputs are poor estimates of the true class probabilities, while a smaller or less trained teacher provides softer, better calibrated targets, which are most important exactly when data is scarce. This view should be presented in the introduction, discussed as a paragraph in the related works section (currently in the appendix), and taken into account when interpreting Sections 2 and 3, since under it the analysis largely decomposes into known parts, and it is unclear what the capacity-budget framing adds beyond them. In addition, the practical significance of DVA itself is limited: most of its gain comes from the hand-specified difficulty schedule (most of the advantage over random at drop rate 0.9 in Table 3, while the volume-maximization stage barely adds any gains), and since DVA already requires a small teacher as its proxy, simply distilling from that small teacher at a 0.9 drop rate outperforms DVA with the large teacher by a large margin (Table 22 vs Table 2), a comparison the main text never makes. Finally, although the evaluation looks extensive, it is lacking and thin exactly where the claims need support. Each dataset uses essentially a single teacher-student pair, and teacher capacity is varied only within the same architecture family. Also, the VGG/MobileNetV2 students only reuse subsets selected in the Resnet setup, so every analysis claim in Sections 2 and 3 are supported by one or two pairs. Standard KD papers (and generally to supported claims) evaluate multiple teacher-student pairs on CIFAR100 alone, including cross-architecture pairs. General claims about teacher capacity and data budgets therefore need, on top of the corrected experiments above, support from multiple teacher-student pairs per dataset, or from additional datasets or modalities (small audio, NLP, or vision tasks such as CIFAR10, could potentially be fitting with many teacher-student pairs to support claims).

Questions for authors
  1. The most important set of necessary additional experiments - i'd ask for multiple settings in which the results are shown for teachers that don't overfit on their training data + are calibrated (temperature sweep), with mainly KD loss (i.e. KD weight of 0.9 or 1). I'd specifically be interested in this for the claims of Sections 2.1 and 2.2, and for the main results in Table 2. The drop rate sweep is not necessary. I'd recommend showing this on an additional modality (all experiments are vision), but the main request is generally showing this for multiple teacher-student architectures and datasets. Currently, the strong benefits are shown on ~2 teacher-student architectures in overfitting or uncalibrated regimes.

  2. Can you distill at a 0.9 drop rate from an early checkpoint of the width 64 model used as the distillation teacher, not only as the proxy as in App G4?

  3. Can you repeat the temperature sweep of App F3 on ImageNet at drop 0.9?

  4. Can you report (a) a one-hot training only baseline and (b) pure KD from each teacher on the same subsets, at least at drop 0.9 on CIFAR100? It would be nice if some of the main results (no memorization + temperature scaling) would could be performed for full distillation or 0.9 KD weight, as usually done in KD papers with CIFAR100.

  5. For the Section 3 analysis (specifically Figure 3(a)): which dataset and architecture are these on? how many seeds? and most importantly, was class balance enforced within the difficulty regions? Can you report per-class sample counts of the easy/medium/hard subsets at drop rates 0.8-0.9? At these rates there are very few samples per class, so class coverage alone could potentially explain the “easy wins” effect. Could you also repeat this experiment for a dataset with less classes (say 5-20 classes)? The effect might also be connected to the number of classes and it is currently shown for one dataset and one teacher-student pair.

  6. How were the difficulty regions and interval sizes in Table 7 chosen, and on what data?

  7. Given that the proposed method already requires the small teacher, why should a practitioner use it with a large teacher over distilling directly from the small teacher? Can you please add this comparison?

[1] A statistical perspective on distillation, ICML 2021.

[2] The role of teacher calibration in knowledge distillation, IEEE Access 2025.

[3] Revisit the essence of distilling knowledge through calibration, ICML 2024.

[4] SGD-based knowledge distillation with Bayesian teachers: Theory and guidelines, ICLR 2026.

[5] On calibration of modern neural networks, ICML 2017.

[6] Understanding deep learning requires rethinking generalization, ICLR 2017.

13 comments

We thank the reviewer for the careful review and constructive suggestions. Below we address each concern and report the additional experiments conducted in response.


W1-1. & Q1-1. Teacher Capacity vs. Training-Set Fit.

The main claim of the paper is that the preferred teacher capacity changes with the data budget, and the whole analysis and method are framed around teacher size. The main issue is that capacity is mixed with how strongly the teacher fits the training set. KD transfers the teacher’s outputs on training samples, and a teacher that has memorized the training set outputs near one-hot distributions there, so its soft labels carry almost nothing beyond the ground truth label (which the student receives anyway, since the loss the authors used to my understanding is 0.5 CE + 0.5 KD, i.e. 0.5 mixing parameter for most of the results in the paper). The results in the paper are consistent with this view. In Table 11 all teachers with accuracy over 99% perform worst at drop rate 0.9, and student performance jumps exactly where train accuracy leaves saturation, i.e. kind of a U shaped behavior in training-set fit rather than a monotone capacity effect (as the authors suggest).

The most important set of necessary additional experiments - i'd ask for multiple settings in which the results are shown for teachers that don't overfit on their training data + are calibrated (temperature sweep), with mainly KD loss (i.e. KD weight of 0.9 or 1).

A1. We thank the reviewer for the valuable suggestion to further disentangle the effects of teacher capacity, training-set fit, and temperature calibration.

Our additional experiments below confirm two key findings: (1) the observed small-teacher advantage cannot be explained by differences in training-set fit, and (2) the capacity–budget crossover persists after temperature scaling.

(1) Capacity vs. training-set fit. We compared a fully trained small teacher (width-8, 67.44% test accuracy) with large, early-stopped teacher (width-64) checkpoints, including an approximately accuracy-matched checkpoint (epoch 180, 68.42% test accuracy). At a drop rate of 0.9, the small teacher outperforms the accuracy-matched large teacher by 4.21 pp (66.25% vs. 62.04%), indicating that training-set fit alone does not explain the performance gap.

(2) Temperature scaling. We swept T ∈ {1, 4, 16, 128} with a KD weight of 0.9 on CIFAR-100 using class-balanced random pruning (ResNet-18 student, 3 seeds). Even after temperature tuning, the large teacher performs better at a drop rate of 0.1 (79.50% vs. 76.49%), while the small teacher performs better at 0.9 (66.25% vs. 53.44%), confirming that the capacity–budget crossover persists after temperature scaling.

We have also included this table in Appendix J.

Temperature sensitivity across teacher configurations. Student accuracy (%) on CIFAR-100 using a ResNet-18 student and ResNet-34 teachers at drop rates of 0.1 and 0.9.

TSmall (w8)Large (ep20)Large (ep180)Large (final)
0.10.90.10.90.10.90.10.9
176.49±0.1053.68±1.3362.97±0.1955.34±0.1974.89±0.2346.87±1.2677.11±0.1539.24±1.25
473.09±0.2063.64±0.1863.46±0.1957.89±0.2773.11±0.1454.30±0.8778.58±0.1348.60±1.29
1671.96±0.1865.61±0.1867.99±0.1460.33±0.2073.64±0.2161.20±0.5479.50±0.2053.44±1.01
12871.74±0.1666.25±0.0969.24±0.1560.75±0.1574.09±0.0762.04±0.1779.47±0.2553.08±1.88

Q4. Teacher Memorization and Soft-Label Information.

Can you report (a) a one-hot training only baseline and (b) pure KD from each teacher on the same subsets, at least at drop 0.9 on CIFAR100? It would be nice if some of the main results (no memorization + temperature scaling) could be performed for full distillation or 0.9 KD weight, as usually done in KD papers with CIFAR100.

A2. We thank the reviewer for this suggestion. We additionally evaluated (1) a one-hot training only (CE-only) baseline and (2) pure KD (full distillation) on CIFAR-100.

(1) One-hot training only baseline. We evaluated CE-only training across all six drop rates using randomly selected subsets over 10 seeds on CIFAR-100. The KD-trained student outperforms the CE-only baseline reported in Table 2.

Method0.10.30.50.70.80.9
CE-only77.10 ± 0.1874.22 ± 0.1970.49 ± 0.3462.98 ± 0.3155.92 ± 0.7338.20 ± 1.02

(2) Pure KD from each teacher on the same subsets. We additionally evaluate pure KD using the same randomly selected subsets. The capacity–budget crossover persists under pure KD, and we have included these results in Table 12.

Teacher0.10.30.50.70.80.9
Small (w8)72.76 ± 0.1672.62 ± 0.2072.11 ± 0.0770.90 ± 0.1569.46 ± 0.1163.85 ± 0.10
Large (w64 final)78.74 ± 0.0676.93 ± 0.3173.71 ± 0.1867.62 ± 0.2862.93 ± 0.3549.05 ± 1.35

W1-2. & Q2 Early Large-Teacher Checkpoints as KD Teachers.

Moreover, in App G4 an early checkpoint large teacher of 64 width (same capacity, less fit) recovers nearly all of the small proxy advantage at drop rate 0.9 compared to the converged one, and the authors themselves write that “the training stage of the proxy explains the performance gap", yet the title, abstract, and Sections 1 and 2 keep attributing the behavior to capacity. Table 19 also shows the same behavior. Generally, experiments that would seperate a fit effect from a capacity effect are missing. There are no results with an early checkpoint of the large model used as the distillation teacher at 0.9 drop rate (App G4 only swaps the proxy).

Can you distill at a 0.9 drop rate from an early checkpoint of the width 64 model used as the distillation teacher, not only as the proxy as in App G4?

A3. We thank the reviewer for this valuable suggestion. Building on the analysis in A1, we extend the evaluation to additional early-stopped checkpoints of the large (width-64) teacher and apply DVA using each checkpoint as both the selection proxy and KD teacher. As shown in the table below, none of the evaluated large-teacher checkpoints matches the fully trained small teacher, even when their training and test accuracies are comparable. We evaluated different checkpoints of the large (width-64) teachers on CIFAR-100 at a drop rate of 0.9, using each model as both the selection proxy and the KD teacher (10 seeds). The small (width-8) teacher outperforms the train-accuracy-matched (ep180) and test-accuracy-matched (ep170) large teachers by 10.14 pp and 9.98 pp, respectively, suggesting that training-set fit alone does not explain the small-teacher advantage. This table has also been added to Appendix J.

Teacher (Proxy and KD)T. Train Acc. (%)T. Test Acc. (%)DVA Student Acc. (%)
Large ep1053.8549.5453.97 ± 0.14
Large ep2063.9356.7656.90 ± 0.13
Large ep3064.5356.5355.64 ± 0.32
Large ep4069.3460.0457.14 ± 0.26
Large ep8073.5760.5055.88 ± 0.29
Large ep170 (test-matched)88.5366.7652.60 ± 0.73
Large ep180 (train-matched)92.8468.4252.44 ± 0.88
Small91.8667.4462.58 ± 0.72

W1-3. & Q3. Temperature Calibration on ImageNet.

Also, for the ImageNet experiments (the only setting where the teachers do not memorize, and there the crossover is much smaller), the authors run at T=1 with no temperature sweep, while the temperature sweep in App F3 is only run in the memorized CIFAR100 regime, where no temperature can recover structure that is no longer in the outputs. So even though the authors show a temperature sweep to potentially fix calibration, it is only done on a single teacher-student pair memorized CIFAR100 setting and not on the other settings. The fix should be no memorization + temperature sweep for calibration.

Can you repeat the temperature sweep of App F3 on ImageNet at drop 0.9?

A4. We thank the reviewer for this valuable suggestion. Our additional experiments confirm that the small-teacher advantage persists on ImageNet even after temperature tuning in a non-memorizing setting. We evaluated ResNet-10 (small) and ResNet-34 (large) teachers at a drop rate of 0.9 using randomly selected subsets over 3 seeds. The small teacher outperforms the large teacher across all tested temperatures, achieving 55.11% vs. 51.88% (+3.23 pp) at their best-performing temperature (T = 2). We include this table in Appendix F.3.

TeacherT=0.5T=1T=2T=4T=8T=16T=32
Large (ResNet-34)49.50 ± 0.0749.16 ± 0.1551.88 ± 0.0850.61 ± 0.1749.87 ± 0.1249.49 ± 0.2249.41 ± 0.32
Small (ResNet-10)50.95 ± 0.0453.85 ± 0.2555.11 ± 0.1253.92 ± 0.1553.00 ± 0.2452.58 ± 0.1052.58 ± 0.09

W2-1. Missing Related Work and Capacity-Based Framing.

The paper does not discuss the closest existing line of work, which explains teacher quality in KD through how well the teacher’s probabilities approximate the true class probabilities rather than through its accuracy or size [1,2,3,4]. This perspective directly predicts the paper’s findings, A converged over parameterized teacher is overconfident on its training data [5,6], so its train-set outputs are poor estimates of the true class probabilities, while a smaller or less trained teacher provides softer, better calibrated targets, which are most important exactly when data is scarce. This view should be presented in the introduction, discussed as a paragraph in the related works section (currently in the appendix), and taken into account when interpreting Sections 2 and 3, since under it the analysis largely decomposes into known parts, and it is unclear what the capacity-budget framing adds beyond them.

A5. We thank the reviewer for pointing out these relevant studies. We agree that teacher calibration and training-set fit are important factors in KD.

  • Key difference from [1–4]. While these studies emphasize teacher probability quality and calibration, our analysis additionally distinguishes relational ordering and score geometry. Temperature scaling preserves class rankings and therefore cannot explain the improvements obtained by changing relational ordering alone (Section 2.2). Thus, we believe our findings provide insights beyond the teacher-calibration perspective.

  • Regarding [5,6]. Our additional experiments show that the small-teacher advantage persists even when compared with large-teacher checkpoints with similar training and test accuracy, suggesting that training-set fit alone does not fully explain the observed capacity-dependent behavior.

  • Manuscript revisions. Following the reviewer's suggestion, we have expanded the discussion of these studies in the Introduction, Related Work, and Sections 2–3.


[1] Menon et al., "A Statistical Perspective on Distillation," ICML 2021.

[2] Kim et al., "The Role of Teacher Calibration in Knowledge Distillation," IEEE Access 2025.

[3] Fan et al., "Revisit the Essence of Distilling Knowledge through Calibration," ICML 2024.

[4] Morad et al., "SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines," ICLR 2026.

[5] Guo et al., "On Calibration of Modern Neural Networks," ICML 2017.

[6] Zhang et al., "Understanding Deep Learning Requires Rethinking Generalization," ICLR 2017.

Reviewer_suHgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies an interesting problem about the teacher capacity with limited data budget. The authors perform thorough analysis and investigation to further drive the potential reasons and implications behind the problem. A practical training-dynamics-free framework DVA is proposed to perform difficulty and volume aware data pruning in knowledge distillation.

Critical strengths
  1. The raised research questions are important for KD, and they are well stated and investigated with thorough analysis.
  2. The practical data pruning solution DVA is proposed, by involving the budget-aware difficulty filtering, relational projection, and volume constraints-based optimization.
  3. The presentation of this paper is well organized, and easy to follow.
Critical weaknesses
  1. The used dataset for the main experiments is CIFAR-100. It is better to include more datasets to further validate the performance.
  2. The main experimental conclusion are mainly with the ResNet, e,g. Table2. Different model architectures should be included to demonstrate the effectiveness of the proposed solution, especially the large vision model like ViT.
  3. The efficiency analysis should be further discussed.
Questions for authors

refer to the Weaknesses

4 comments

We thank the reviewer for the constructive feedback and the positive assessment of our analyses and the practicality of the proposed framework. We address each concern below.


W1. Additional Datasets.

The used dataset for the main experiments is CIFAR-100. It is better to include more datasets to further validate the performance.

A1. Thank you for this suggestion. We would like to clarify that our evaluation already includes experiments on Tiny-ImageNet and ImageNet-1K, in addition to CIFAR-100. The corresponding results are presented in Figures 4(c) and 4(d) and Table 5 of the main paper, with more detailed results provided in Tables 21 and 24 (Tables 24 and 27 in the revised manuscript) of the appendix. These results further demonstrate DVA's effectiveness across datasets of different scales, particularly under limited data budgets.


W2. Additional Experiments.

The main experimental conclusions are mainly with the ResNet, e,g. Table2. Different model architectures should be included to demonstrate the effectiveness of the proposed solution, especially the large vision model like ViT.

A2. Thank you for your suggestion. In addition to ResNet-based experiments, we have evaluated DVA using Vision Transformer (ViT) architectures. The corresponding results are presented in Figures 4(a) and 4(b) of the main paper, with detailed results reported in Tables 20 and 23 (Tables 23 and 26 in the revised manuscript) of the appendix. These results further demonstrate DVA's effectiveness across different model architectures, including both ResNet and ViT.


W3. Efficiency Analysis.

The efficiency analysis should be further discussed.

A3. Thank you for this valuable suggestion. We agree that a more detailed efficiency analysis would help clarify the computational advantages of our proposed method.

To address this concern, we provide an additional efficiency comparison table below, extending the results reported in Table 27 (Table 30 in the revised manuscript) of the appendix with a more detailed breakdown across different datasets and model architectures. We added this result in Table 31 (Appendix I) in the revised manuscript.

Computational cost (PFLOPs) of coreset selection across datasets. Values represent the total FLOPs required to generate coresets for all six drop rates (0.1–0.9), including computations shared across drop rates.

MethodCIFAR-100 (ResNet)CIFAR-100 (ViT)Tiny-ImageNetImageNet
Training-Dynamics-Based
D²33.522.281.31710
DUAL4.993.3179.91670
Training-Dynamics-Free
MDSLR0.1160.110.9310.5
IF-Beta3.183.1018.3140.68
DVA (Ours)0.00310.00400.06873.02
AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer suHg,

Thank you once again for your careful review and constructive feedback. We greatly appreciate your time and effort in helping us improve our work.

As the discussion period draws to a close, we would like to ask whether you have any remaining concerns or questions regarding our responses. We would be happy to address them.

Sincerely,
The Authors

Reviewer_suHgOpenReview ↗

Response to the rebuttal

Thanks for the rebuttal responses. But for W1 and W2, my previous suggestions and comments are because the additional dataset (W1) and different model setting (W2) are not included in the corresponding places(tables or figures) with the specified research questions, resulting in the “unconvincing & insufficient” results.

For example, the main experimental results in table 2 are only reported with CIFAR-100. If the additional datasets have been conducted for the main experiments (Section 5.2) with all compared baselines, why the results are placed in other places, e.g. figures 4(c) and 4(d) and Table 5 of the main paper as replied in the rebuttal response? Why not include those results in Table2 to well support the main results?

The replies in the rebuttal response for W2 have the same concerns.

If the experimental results have been conducted with more datasets and more different setting for the same studied questions, it is better to present the results in the corresponding tables or figures, which can make the studied research questions more convincing and clear.

We sincerely thank the reviewer for the detailed suggestions on improving the presentation of our experimental results. We agree that consolidating the results across datasets and model architectures would make the main comparisons clearer and more convincing.

Accordingly, we have added Table 3 in Section 5.2, which brings together the single-teacher KD results on CIFAR-100 (ViT), Tiny-ImageNet (ResNet), and ImageNet (ResNet), alongside the existing CIFAR-100 (ResNet) results in Table 2. We have also reorganized the corresponding discussion and removed redundant presentations, including the former Appendix H.1. We also added references in the corresponding sections of the main text to the related appendix results (e.g., Sections 2.2, 3, 5.2, and 5.4).

Comments on the paper

2 comments

Dear reviewers and AC,

We sincerely thank the reviewers for their valuable comments and constructive suggestions.

We have carefully addressed the reviewers' concerns and questions through our individual responses and revisions to the manuscript. For convenience, we summarize the key updates below, following the order of the revised manuscript.


Main Text

  • Sections 1, 2.1, 2.2, and 3: Refined interpretations, novelty, and connections to prior work. (oci3)
  • Figure 3(a): Clarified experimental settings. (oci3)

Appendix

  • Appendix B: Expanded related work on teacher quality, calibration, and training-set fit. (oci3)
  • Appendix D.5: Specified difficulty percentile bounds. (oci3, GZ2Z)
  • Appendix F.1: Additional teacher-supervision decomposition experiments. (oci3)
  • Appendix F.2: Teacher-capacity crossover across architectures. (oci3)
  • Appendix F.3: ImageNet temperature sensitivity. (oci3)
  • Appendix H.2: Direct small-teacher KD comparison and practical implications. (oci3)
  • Appendix I: Refined computational cost measurements. (suHg)
  • Appendix J: Teacher capacity, training-set fit, and temperature scaling. (oci3, GZ2Z)
  • Appendix K: Held-out difficulty-window validation. (oci3, GZ2Z)
  • Appendix L: DVA generalization across architectures. (oci3)
  • Appendix M: CIFAR-10 capacity crossover and difficulty analysis. (oci3)
  • Appendix N: Fixed optimization budget experiments. (GZ2Z)

All revisions and additions are highlighted in red in the updated manuscript for ease of reference.

We believe these updates have strengthened the manuscript and addressed the reviewers' concerns.

Sincerely,
The Authors

AuthorsOpenReview ↗

Additional Revision

We have further revised the manuscript in response to the follow-up discussion. For convenience, we summarize the additional updates below, following the order of the revised manuscript.

Main Text

  • Section 2.1: Stated the role of training-set fit explicitly and incorporated the controlled teacher comparisons (KD weight 0.9, tuned temperatures, accuracy-matched checkpoint, and ImageNet). (oci3)
  • Sections 2.2, 3, 5, 5.2, and 5.4: Added explicit references from each analysis to the corresponding additional results in the appendix. (suHg)
  • Section 5.2 and Table 3: Moved the single-teacher results on CIFAR-100 with ViT, Tiny-ImageNet, and ImageNet into the main experiment section, with all results reported as absolute accuracy. (suHg)
  • Section 5.5 and Figure 4: Reorganized to present scheduled multi-teacher KD results. (suHg)

Appendix

  • Appendix F.3: Corrected a transcription error in Table 18. (oci3)
  • Appendix H: Renumbered after moving the single-teacher results to the main text (multi-teacher KD is now Appendix H.1, cross-architecture transfer is now Appendix H.2).
  • Appendix J: Added the teacher-supervision decomposition under the controlled setting, using an accuracy-matched large-teacher checkpoint. (oci3)
  • Appendix O: Additional ImageNet results with a ResNet-50 teacher and scheduled multi-teacher KD. (oci3)

Due to these changes, some table numbers differ from the previous revision; our latest responses refer to the numbering of the current manuscript.

All revisions and additions are highlighted in red in the updated manuscript for ease of reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.