- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The papers tries to answer why immature teachers (early stop) can produce better students than fully trained ones. The authors decompose the student training loss into a penalty on the student’s logit norm (which they call the regularization part) and an alignment term whose gradient scales with the teacher’s logit norm (which they call the knowledge transfer part). The decomposition holds only for temperatures that approach inf. They then claim that the teacher’s logit norm sets the balance between the two components and that immature teachers achieve the best balance, yielding the best students. Experiments show that students distilled from teachers of different maturities follow a the same U-shaped curve of test loss against the student’s logit norm. They also perform an experiment where the mature teacher’s logit direction is kept while they use the immature teacher’s norm, achieving a lower student test loss.
Critical strengths
-
The paper is written very well and is easy to follow. The question that is asked is clear, and the decomposition of the loss into a regularization term and an alignment term is nice since it connects two views of why KD works through a single quantity, the teacher’s logit norm (only at high temperatures though).
-
The experiment where the norm is replaced cleanly separates the effect of the teacher’s logit norm from its direction while keeping the teacher’s predicted classes fixed, and the finding that direction matters at equal norm is interesting.
Critical weaknesses
-
The main claim of the paper is that the teacher’s logit norm explains why immature teachers produce better students. Throughout the paper, a “better student” is shown mostly through the test loss of the student. The issue is that test loss depends on both whether the student’s predictions are correct, and how confident the student is. A student that is less overconfident gets a lower test loss even if its predictions are no better, and test loss and accuracy are known to move in different directions because of this [1]. This decomposition (calibration + quality) is a standard property of proper scoring rules [2]. Overall there is a much simpler explanation for the results in the paper. A teacher with a smaller logit norm produces softer targets, so the student trained to match the teacher also ends up with smaller logits. The student is therefore less overconfident, and its test loss drops, without necessarily having learned anything better. The paper’s own results are consistent with this: under norm replacement, test loss improves on all datasets, but accuracy, which ignores confidence, barely improves on CIFAR-10. This also applies to the U-shaped curves which are also shown w.r.t. test loss. In addition, the paper does not discuss the closest existing line of related work, which explains teacher quality via calibration (apart from one sentence on page 3 line 142). This should be presented and hinted towards in the abstract and introduction, discussed as a paragraph in the related works section, and discussed and taken into account throughout the paper when displaying results. For example, the abstract says “difficult to explain from the knowledge-transfer view that more accurate teachers are more beneficial”, but recent works attribute teacher quality in KD to how well its predicted probabilities match the true class probabilities, not to its accuracy [3,4,5]. This perspective directly predicts that a less overconfident, immature teacher can be a better teacher, yet the paper mentions it only in one sentence (page 3 line 142) and instead frames the question as “more accurate teachers should be better”. This is also discussed in recent reviews of theoretical perspectives on KD [6]. Directly linked to the main issue, the introduction and related works section is mainly outdated with nearly all citations from 2022 or earlier, and no citations from 2025 or 2026. Continuing the chain of thought, if teacher calibration explains student quality (known) as well as the logit norm does (paper’s contribution), the paper’s finding would decompose into two known parts: that teachers closer to the true class probabilities make better students [3,4,5], and that a network’s logit norm controls its confidence and grows during training [7], which is a general property of neural networks rather than of distillation. It is therefore unclear what the norm-balance mechanism adds beyond this.
-
Results are obtained in a non-standard training setup that seems designed to make the effect visible, and it is unclear whether they hold in standard KD. Students are trained without data augmentation, with temperature T=1 (Standard KD usually uses T=4 [9]), and with a small constant LR with no scheduler. The paper states that this choice is made because otherwise “teachers converge within a few epochs and differences between their outputs become small”. This setting makes the students heavily untrained and is a setting no practician would use. From fig 3 and 4 (there are no result tables in the paper so i can't 100% verify the numbers), the students reach about 75% acc on CIFAR10, 45% on CIFAR100, and 40% on Tiny ImageNet, far below what the same architectures reach with standard training even without distillation (~91% [8], ~69% [9], ~63% [10]). Moreover, for a different setup with WRN-16-2 and Tiny ImageNet, and AG News, the effect weakens or disappears (also, only 1 seed is used). Even though the authors write in the limitation section "To observe differences in teacher outputs, we disabled student data augmentation and assigned a large weight to the distillation term (α = 0.9). Whether the same relationship holds under standard settings remains to be verified", α = 0.9 is actually the standard value [9], and the authors do not mention the small LR with no scheduler, which is the main reason the students are undertrained, and while temperature is varied in Section 4.4, the other non-standard choices are kept. These choices create unrealistic settings that make the paper’s central claim untested in standard scenarios. Also, the theoretical analysis holds only for T that goes to inf, while the main experiments use T=1, and the paper does not examine how well it describes the actual distillation gradient at the temperatures used.
Questions for authors
-
Can you use standard temperature scaling for each trained student (i.e. fit a single temperature per student on a val set) and then show the U-shaped curves and the norm-replacement results before and after scaling? This would show whether the improvements remain once the student’s confidence is unfactored.
-
Can you report test accuracy (with standard deviations) in tables for the maturity sweep and the norm replacement experiment?
-
Norm replacement is equivalent to changing the teacher’s temperature separately for each input, an interpretation that has also been made for per input logit normalization [7]. How does the effect compare to distilling from the mature teacher with well-tuned teacher temperature?
-
The teacher’s test loss should measure how closely its predictions approximate the true class probabilities. Does it predict student quality across teacher maturities as well as the teacher’s logit norm does? What does the teacher’s logit norm view explain that the calibration view doesn't?
-
Can you report results for standard KD settings such as with student data augmentation, T=4, and a standard LR scheduler? Or a different setting where the students reach standard accuracies?
-
Since the analysis is done for T that goes to inf, how well does it describe the actual distillation gradient at the temperatures used in the experiments (T=1,4)?
[1] On calibration of modern neural networks, ICML 2017.
[2] Reliability, sufficiency, and the decomposition of proper scores, Quarterly Journal of the Royal Meteorological Society 2009.
[3] Revisit the essence of distilling knowledge through calibration, ICML 2024.
[4] The role of teacher calibration in knowledge distillation, IEEE Access 2025.
[5] SGD-based knowledge distillation with Bayesian teachers: Theory and guidelines, ICLR 2026.
[6] Theoretical perspectives on knowledge distillation: A review. Wiley Interdisciplinary Reviews: Computational Statistics, 2025.
[7] Mitigating neural network overconfidence with logit normalization, ICML 2022.
[8] Deep residual learning for image recognition, CVPR 2016.
[9] Contrastive representation distillation, ICLR 2020.
[10] Revisiting knowledge distillation via label smoothing regularization, CVPR 2020.
6 comments
Response to Reviewer oqYW (2/4): W2 and Q1
For Tables A–B see Common Response 3/6 (comment 3/13); for Tables D–E see 4/6–5/6 (comments 4–5/13).
W2. Non-standard training settings and scope of the high-temperature analysis
At T=1 without student data augmentation, immature teachers achieved higher mean test accuracy under the alternative learning-rate schedules and optimizers tested. Changing distillation temperature or introducing data augmentation reduced or reversed the advantage in some settings (Tables D and E; oqYW Q5 (response 4/4; comment 13/13)). On CIFAR-10, gains with Adam (+1.1 points) and AdamW (+0.6 points) were modest relative to the variation across three student seeds.
On CIFAR-10, students distilled from the mature teacher achieved approximately 72% accuracy under the original settings, 81.5% with Adam, and 85.8% with data augmentation. The student distilled from the mature teacher under the original settings reached 99.9% training accuracy on CIFAR-10. Thus, its low test accuracy cannot be attributed to a failure to fit the training set. We will correct the description of α=0.9 as nonstandard; a distillation weight of 0.9 was also used by Tian et al. (2020; [9] in your review) and Kim et al. (2025, on CIFAR-100). The small constant learning rate, the absence of student data augmentation, and the single student seed (seed 0) for WRN-16-2 in Appendix H are stated in Section 4.1 and Appendix C; we will make them more prominent in the main text. We address the high-temperature approximation in oqYW Q6 (response 4/4; comment 13/13).
Q1. Student temperature scaling
Under direct distillation at T=1, the minimum calibrated student test loss for immature teachers remained below that for the mature teacher. Relative to the mature teacher, the reduction in minimum test loss before/after calibration was 0.412/0.097 on CIFAR-10, 0.584/0.312 on CIFAR-100, and 0.365/0.256 on Tiny ImageNet (Table A).
For norm replacement, the corresponding reductions changed from 0.226 to 0.003, from 0.337 to 0.094, and from 0.302 to 0.194. No clear improvement remained on CIFAR-10 (Table B). The best teacher checkpoint or reference maturity was selected separately before and after calibration. Temperatures were fitted on 20% of the test set, and losses were evaluated on the remaining 80%.
At T=4 (Section 5 teachers, cosine decay), the mature teacher yielded the lowest calibrated student test loss. At T=64 (Section 4 settings), the test-loss reduction relative to the mature teacher retained 1.9% and 24.0% of its pre-calibration magnitude on CIFAR-10 and CIFAR-100, respectively. We have not evaluated the intermediate student-training trajectories in Figures 3 and 4 after calibration.
Response to Reviewer oqYW (3/4): Q2–Q3
Q2. Test accuracy with standard deviations
We report student test accuracy and standard deviations for each teacher maturity and norm-replacement condition. Values are mean ± standard deviation over three student seeds (%, T=1, full test set).
Direct distillation: CIFAR-10/CIFAR-100 (Section 4)
| Teacher epoch | CIFAR-10 | CIFAR-100 |
|---|---|---|
| 1 | 42.52 ± 0.39 | 12.74 ± 0.34 |
| 2 | 46.69 ± 0.02 | 11.79 ± 0.28 |
| 5 | 57.81 ± 0.23 | 16.35 ± 0.13 |
| 10 | 67.66 ± 0.26 | 21.07 ± 0.23 |
| 25 | 74.06 ± 0.48 | 33.61 ± 0.11 |
| 30 | 73.50 ± 0.36 | 38.12 ± 0.08 |
| 35 | 75.26 ± 0.64 | 40.24 ± 0.13 |
| 40 | 71.30 ± 0.91 | 40.90 ± 0.33 |
| 45 | 74.87 ± 0.53 | 42.66 ± 0.33 |
| 50 | 75.16 ± 0.40 | 43.83 ± 0.29 |
| 100 | 73.68 ± 0.82 | 43.53 ± 0.56 |
| 150 | 73.56 ± 0.85 | 41.81 ± 0.46 |
| 200 | 73.03 ± 1.65 | 40.23 ± 0.68 |
| 400 (mature teacher) | 72.49 ± 0.90 | 38.61 ± 0.56 |
On CIFAR-100, the epoch-50 teacher yielded 5.22 percentage points higher student accuracy than the mature teacher. On CIFAR-10, the epoch-35 and epoch-50 teachers yielded gains of +2.77 and +2.67 points, respectively, whereas the epoch-40 teacher performed worse than the mature teacher.
Direct distillation: Tiny ImageNet (teacher-maturity comparison)
| Teacher epoch | Student test accuracy (%) |
|---|---|
| 1 | 7.23 ± 0.05 |
| 5 | 16.69 ± 0.35 |
| 10 | 27.19 ± 0.14 |
| 25 | 38.59 ± 0.18 |
| 50 | 41.72 ± 0.09 |
| 100 (mature teacher) | 37.81 ± 0.26 |
The epoch-50 teacher outperformed the mature teacher by 3.91 percentage points.
Norm replacement: CIFAR-10/CIFAR-100/Tiny ImageNet (Section 5, Figure 6)
| Reference maturity (epochs) | CIFAR-10 | CIFAR-100 | Tiny ImageNet |
|---|---|---|---|
| 1 | 66.40 ± 1.21 | 26.44 ± 0.45 | 30.27 ± 0.10 |
| 5 | — | — | 40.96 ± 0.41 |
| 10 | 70.99 ± 1.09 | 40.79 ± 0.40 | 40.21 ± 0.39 |
| 25 | 72.01 ± 0.80 | 40.20 ± 0.10 | 39.01 ± 0.41 |
| 50 | 72.24 ± 1.26 | 40.03 ± 0.57 | 38.27 ± 0.08 |
| 100 | 71.81 ± 0.88 | 40.08 ± 0.44 | — |
| No replacement | 72.30 ± 1.36 | 39.05 ± 1.04 | 37.81 ± 0.26 |
Relative to no replacement, norm replacement did not improve accuracy on CIFAR-10 but yielded gains of up to +1.74 points on CIFAR-100 and +3.15 points on Tiny ImageNet. We will add test accuracies and standard deviations to the relevant figures and tables in the revised paper.
Q3. Teacher temperature scaling vs. norm replacement
At T=1, single-temperature scaling of the mature teacher and the best norm replacement yielded similar student accuracies. Accuracy was 71.6% vs. 72.3% on CIFAR-10, 41.0% vs. 40.9% on CIFAR-100, and 41.0% vs. 40.8% on Tiny ImageNet (80% test subset; Table C). On CIFAR-10, neither clearly exceeded the 72.2% accuracy obtained from the unmodified mature teacher.
Calibrated student test losses for temperature scaling / norm replacement were 0.863/0.841 on CIFAR-10, 2.273/2.284 on CIFAR-100, and 2.496/2.492 on Tiny ImageNet. The best reference maturity for norm replacement was selected separately for accuracy and test loss. Neither method matched direct distillation from the epoch-50 teacher (74.6%, 44.0%, and 41.8%, respectively).
The temperature was selected by minimizing teacher test loss; we did not search for a temperature that maximized student accuracy. As an example showing that the temperature minimizing teacher test loss need not improve student accuracy, a T=4 experiment under the Section 5 setting (cf. Q5) found that temperature scaling reduced accuracy from 81.5% to 77.7% on CIFAR-10, from 49.3% to 47.5% on CIFAR-100, and from 47.1% to 43.4% on Tiny ImageNet (80% test subset).
Response to Reviewer oqYW (1/4): W1
See Tables A–B in Common Response 3/6 (comment 3/13), Tables C–D in 4/6 (comment 4/13), and Table E in 5/6 (comment 5/13).
We thank the reviewer for the detailed comments. In additional experiments, we examined the effects of post-hoc student temperature scaling, changes in training settings, and the validity of the high-temperature approximation.
W1. Calibration-based explanation and related work
At T=1, students directly distilled from immature teachers still had lower test loss than those distilled from mature teachers after post-hoc temperature scaling (differences of 0.097/0.312/0.256 on CIFAR-10/CIFAR-100/Tiny ImageNet; Table A). By contrast, the improvement in test loss from norm replacement decreased after calibration, reaching only 0.003 on CIFAR-10 (Table B). On CIFAR-100 and Tiny ImageNet, students obtained similar accuracy from single-temperature scaling of the mature teacher and from norm replacement (Table C). The shrinkage of the norm-replacement test-loss reduction after calibration and the accuracy gains under Condition D are consistent with calibration-related findings by Fan et al. (2024; [3] in your review) and Kim et al. (2025; [4] in your review). Condition D likewise applies a single temperature to the mature teacher's outputs; it improved accuracy on CIFAR-100 and Tiny ImageNet, but did not reach the accuracy obtained from the immature teacher. The selected temperatures (1.64–2.29) lie within the 1.5–3 range reported by Kim et al. (2025) to give robust gains under their experimental settings.
Lower teacher test loss does not necessarily lead to higher accuracy of the distilled student. On CIFAR-100 in Section 5, teacher test loss decreased from 1.969 to 1.742, while student accuracy declined from 44.0% to 39.2%. Among checkpoints from epoch 50 onward in Section 4, Spearman rank correlation between teacher test loss and student accuracy on CIFAR-100 was +0.20 (oqYW Q4 in response 4/4; comment 13/13).
When teacher logit norms were matched for each input (Conditions B and C), student accuracy still differed by 2.3, 4.0, and 3.5 percentage points (Table C). Direct distillation from the immature teacher also exceeded distillation from the single-temperature-scaled mature teacher by 3.0, 3.0, and 0.8 points (Table C). These comparisons show that matching the per-input teacher logit norm does not reproduce the accuracy achieved by direct distillation from the immature teacher. We will discuss the calibration findings of Fan et al. (2024) and Kim et al. (2025), the SGD-based theory of Morad et al. (2026; [5] in your review), and the theoretical survey of Liu et al. (2025; [6] in your review) in the revision.