- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies knowledge distillation with data pruning. It observes smaller teachers can beat larger ones under reduced data budgets. The authors split teacher supervision into relational ordering and score geometry. Hybrid-logit experiments verify both components impact low-data distillation. Drawing on sample difficulty and teacher output diversity analysis, they propose DVA. A small proxy teacher first selects samples of suitable difficulty within the budget. Log-determinant volume maximization then chooses a subset with complementary relational signals. Evaluations run on CIFAR-100, Tiny-ImageNet and ImageNet. DVA exceeds most baselines without student training dynamics across most setups and matches or outperforms some dynamics-based methods at high pruning rates.
Critical strengths
- The capacity-budget trend holds across multiple datasets and teacher sizes. Teacher selection thus cannot depend only on teacher-student capacity gaps.
- DVA requires no student training dynamics. It consistently outperforms other dynamics-free baselines at heavy pruning and is competitive with several dynamics-based methods.
Critical weaknesses
- The rationale for using small models as selection proxies is underdeveloped. Small capacity may not explain proxy effectiveness. Table 18 shows early large-teacher checkpoints work nearly as well as small proxies.
- DVA’s main gain originates from difficulty filtering. At a 0.9 drop rate, difficulty filtering contributes 6.09 pp while volume maximization adds just 0.74 pp. Table 7 lacks precise percentiles for Easy, Medium and Hard buckets and omits schedule validation and tuning details.
- Data and optimization budgets are coupled. Equal epochs across pruning rates reduce total optimizer steps for heavy pruning. The teacher-capacity crossover may partly come from inadequate optimization rather than data scarcity alone.
Questions for authors
- What are the exact percentile bounds for each difficulty bucket in Table 7? Were these thresholds set with a held-out validation set?
- Can early large-teacher checkpoints replicate small-teacher low-data advantages when used directly as KD teachers instead of only selection proxies?
- Does the teacher-capacity crossover persist if total optimizer steps are fixed instead of training epochs?
4 comments
W2-2 & Q1. Difficulty Percentile Bounds and Held-Out Validation.
Table 7 lacks precise percentiles for Easy, Medium and Hard buckets and omits schedule validation and tuning details.
What are the exact percentile bounds for each difficulty bucket in Table 7? Were these thresholds set with a held-out validation set?
A3. The original difficulty thresholds were empirically selected based on student test accuracy, without a separate held-out validation set. However, our additional held-out validation experiments confirm a similar budget-dependent difficulty preference. We also clarify the exact percentile bounds below, where the 0th percentile corresponds to the easiest samples. We have updated Table 7 accordingly.
| Drop Rate | CIFAR-100 / Tiny-ImageNet | ImageNet | Pool Size |
|---|---|---|---|
| 0.3 | Medium [10%, 90%] | Hard [20%, 100%] | 80% |
| 0.5 | Medium [20%, 80%] | Medium [20%, 80%] | 60% |
| 0.7 | Medium [20%, 80%] | Medium [20%, 80%] | 60% |
| 0.8 | Easy [0%, 40%] | Medium [30%, 70%] | 40% |
| 0.9 | Easy [0%, 20%] | Medium-Easy [15%, 35%] | 20% |
For validation, we retrained both teachers using a class-stratified 45K/5K train/validation split of CIFAR-100 and evaluated different difficulty windows at drop rates of 0.5 and 0.9 (ResNet-18 student, T=4, 3 seeds). The validation results support the original trend: the preferred difficulty region shifts toward easier samples as the data budget decreases. The original window performs best at a drop rate of 0.5, while a nearby window achieves comparable performance at 0.9. We have included this table in Appendix K.
| Drop Rate | Difficulty Window | DVA Val. Acc. (%) | DVA Test Acc. (%) |
|---|---|---|---|
| 0.5 | [0%, 60%] | 70.68 ± 0.12 | 71.65 ± 0.07 |
| [10%, 70%] | 71.98 ± 0.45 | 72.63 ± 0.11 | |
| [20%, 80%] (Original) | 72.12 ± 0.47 | 72.89 ± 0.40 | |
| [30%, 90%] | 71.91 ± 0.22 | 72.43 ± 0.31 | |
| [40%, 100%] | 70.49 ± 0.48 | 70.58 ± 0.00 | |
| 0.9 | [0%, 20%] (Original) | 49.82 ± 0.70 | 51.25 ± 0.77 |
| [10%, 30%] | 50.63 ± 0.48 | 51.45 ± 0.88 | |
| [20%, 40%] | 49.33 ± 0.92 | 50.31 ± 0.89 | |
| [40%, 60%] | 42.71 ± 0.35 | 43.38 ± 0.79 | |
| [60%, 80%] | 37.61 ± 0.57 | 38.37 ± 0.43 | |
| [80%, 100%] | 29.61 ± 0.46 | 29.12 ± 0.49 |
W3 & Q3. Teacher-Capacity Crossover Under Fixed Optimization Budgets.
Data and optimization budgets are coupled. Equal epochs across pruning rates reduce total optimizer steps for heavy pruning. The teacher-capacity crossover may partly come from inadequate optimization rather than data scarcity alone.
Does the teacher-capacity crossover persist if total optimizer steps are fixed instead of training epochs?
A4. We thank the reviewer for raising this important concern. We agree that fixing the number of training epochs across different pruning rates results in unequal optimization budgets, which may affect the observed teacher-capacity crossover.
The teacher-capacity crossover persists even under a fixed optimization budget. To verify this, we fixed the number of optimizer steps across pruning rates to 187,680 steps, equivalent to 240 epochs on the full CIFAR-100 training set, using a cosine learning rate scheduler (T = 4, 3 seeds). The small teacher (ResNet-34 width-8) outperforms the large teacher (ResNet-34 width-64) by 9.88 pp at a drop rate of 0.9, while the large teacher leads by 6.49 pp at a drop rate of 0.1. We have included this result in Appendix N.
| Drop Rate | Small T. | Large T. |
|---|---|---|
| 0.1 | 72.99 ± 0.12 | 79.48 ± 0.27 |
| 0.9 | 66.69 ± 0.18 | 56.81 ± 1.00 |
A Gentle Reminder
Dear Reviewer GZ2Z,
Thank you once again for your careful review and constructive feedback. We greatly appreciate your time and effort in helping us improve our work.
As the discussion period draws to a close, we would like to ask whether you have any remaining concerns or questions regarding our responses. We would be happy to address them.
Sincerely,
The Authors
reply to author
I appreciate the authors’ response and their answers to my questions. However, I still think my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it unchanged.
We thank the reviewer for the constructive feedback and for noting the practical advantages of DVA over training-dynamics-free baselines. We address each concern below.
W1. & Q2. Teacher Capacity vs. Training-Set Fit.
A1. A1. In short, none of the evaluated early large-teacher checkpoints matches the fully trained small teacher when used for KD. To investigate this, we evaluated different large-teacher (width-64) checkpoints on CIFAR-100 at a drop rate of 0.9, using each model as both the selection proxy and the KD teacher (10 seeds). The fully trained small teacher achieves substantially higher accuracy (62.58%) than the train/test-accuracy-matched large teachers (52.44%/52.60%), as well as the epoch-20 large teacher (56.90%), which previously showed competitive performance only as a selection proxy. This table has been added to Appendix J.
W2-1. Contribution of Difficulty Filtering vs. Volume Maximization.
A2. While difficulty filtering contributes more at a drop rate of 0.9, volume maximization consistently provides additional gains across all six data budgets. As shown in Table 16 (Table 19 in the revised manuscript), volume maximization improves accuracy over random selection within the same difficulty-filtered pool by 0.35–1.05 pp (10 seeds), with gains reaching approximately 1 pp at drop rates of 0.7 and 0.8. Furthermore, highest-volume selection outperforms lowest-volume selection by 3.95, 4.06, and 3.18 pp at drop rates of 0.7, 0.8, and 0.9, respectively. These results support the complementary contributions of both components.