acceptodds
Under review as a conference paper at ICLR 2027

Understanding the Benefits of Immature Teachers in Knowledge Distillation through Alignment and Regularization

Abstract

Knowledge distillation (KD) trains a small student model on the outputs of a large teacher model.Its effect has mainly been explained from two perspectives: the transfer of inter-class similarities contained in teacher predictions and regularization by soft targets.However, because both effects arise from the same distillation loss, what determines their balance remains unclear.Moreover, reports that immature teachers taken during training can yield better students than mature teachers are difficult to explain from the knowledge-transfer view that more accurate teachers are more beneficial.We show that, in the high-temperature limit, this balance is determined by the teacher logit norm, and we examine the resulting hypothesis experimentally.In this limit, the distillation loss separates into a teacher-independent penalty on the student logit norm and an alignment term whose gradient is proportional to the teacher logit norm.On four image and text classification datasets, the test loss of students distilled from teachers of different maturities followed a nearly common U-shaped curve against the student logit norm, and teacher maturity changed the point that students reached on this curve.Fixing the direction of the mature teacher's logits and replacing only their norm with that of an immature teacher reduced the student test loss below that obtained with the mature teacher, without changing the teacher's predicted classes.At the same teacher logit norm, however, student performance differed with the direction.These results suggest that the balance between knowledge transfer and regularization shifts with the teacher logit norm, and that this balance partly underlies the effectiveness of immature teachers.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
knowledge distillation, implicit regularization, generalization, undertrained teachers, immature teachers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oqYWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The papers tries to answer why immature teachers (early stop) can produce better students than fully trained ones. The authors decompose the student training loss into a penalty on the student’s logit norm (which they call the regularization part) and an alignment term whose gradient scales with the teacher’s logit norm (which they call the knowledge transfer part). The decomposition holds only for temperatures that approach inf. They then claim that the teacher’s logit norm sets the balance between the two components and that immature teachers achieve the best balance, yielding the best students. Experiments show that students distilled from teachers of different maturities follow a the same U-shaped curve of test loss against the student’s logit norm. They also perform an experiment where the mature teacher’s logit direction is kept while they use the immature teacher’s norm, achieving a lower student test loss.

Critical strengths
  1. The paper is written very well and is easy to follow. The question that is asked is clear, and the decomposition of the loss into a regularization term and an alignment term is nice since it connects two views of why KD works through a single quantity, the teacher’s logit norm (only at high temperatures though).

  2. The experiment where the norm is replaced cleanly separates the effect of the teacher’s logit norm from its direction while keeping the teacher’s predicted classes fixed, and the finding that direction matters at equal norm is interesting.

Critical weaknesses
  1. The main claim of the paper is that the teacher’s logit norm explains why immature teachers produce better students. Throughout the paper, a “better student” is shown mostly through the test loss of the student. The issue is that test loss depends on both whether the student’s predictions are correct, and how confident the student is. A student that is less overconfident gets a lower test loss even if its predictions are no better, and test loss and accuracy are known to move in different directions because of this [1]. This decomposition (calibration + quality) is a standard property of proper scoring rules [2]. Overall there is a much simpler explanation for the results in the paper. A teacher with a smaller logit norm produces softer targets, so the student trained to match the teacher also ends up with smaller logits. The student is therefore less overconfident, and its test loss drops, without necessarily having learned anything better. The paper’s own results are consistent with this: under norm replacement, test loss improves on all datasets, but accuracy, which ignores confidence, barely improves on CIFAR-10. This also applies to the U-shaped curves which are also shown w.r.t. test loss. In addition, the paper does not discuss the closest existing line of related work, which explains teacher quality via calibration (apart from one sentence on page 3 line 142). This should be presented and hinted towards in the abstract and introduction, discussed as a paragraph in the related works section, and discussed and taken into account throughout the paper when displaying results. For example, the abstract says “difficult to explain from the knowledge-transfer view that more accurate teachers are more beneficial”, but recent works attribute teacher quality in KD to how well its predicted probabilities match the true class probabilities, not to its accuracy [3,4,5]. This perspective directly predicts that a less overconfident, immature teacher can be a better teacher, yet the paper mentions it only in one sentence (page 3 line 142) and instead frames the question as “more accurate teachers should be better”. This is also discussed in recent reviews of theoretical perspectives on KD [6]. Directly linked to the main issue, the introduction and related works section is mainly outdated with nearly all citations from 2022 or earlier, and no citations from 2025 or 2026. Continuing the chain of thought, if teacher calibration explains student quality (known) as well as the logit norm does (paper’s contribution), the paper’s finding would decompose into two known parts: that teachers closer to the true class probabilities make better students [3,4,5], and that a network’s logit norm controls its confidence and grows during training [7], which is a general property of neural networks rather than of distillation. It is therefore unclear what the norm-balance mechanism adds beyond this.

  2. Results are obtained in a non-standard training setup that seems designed to make the effect visible, and it is unclear whether they hold in standard KD. Students are trained without data augmentation, with temperature T=1 (Standard KD usually uses T=4 [9]), and with a small constant LR with no scheduler. The paper states that this choice is made because otherwise “teachers converge within a few epochs and differences between their outputs become small”. This setting makes the students heavily untrained and is a setting no practician would use. From fig 3 and 4 (there are no result tables in the paper so i can't 100% verify the numbers), the students reach about 75% acc on CIFAR10, 45% on CIFAR100, and 40% on Tiny ImageNet, far below what the same architectures reach with standard training even without distillation (~91% [8], ~69% [9], ~63% [10]). Moreover, for a different setup with WRN-16-2 and Tiny ImageNet, and AG News, the effect weakens or disappears (also, only 1 seed is used). Even though the authors write in the limitation section "To observe differences in teacher outputs, we disabled student data augmentation and assigned a large weight to the distillation term (α = 0.9). Whether the same relationship holds under standard settings remains to be verified", α = 0.9 is actually the standard value [9], and the authors do not mention the small LR with no scheduler, which is the main reason the students are undertrained, and while temperature is varied in Section 4.4, the other non-standard choices are kept. These choices create unrealistic settings that make the paper’s central claim untested in standard scenarios. Also, the theoretical analysis holds only for T that goes to inf, while the main experiments use T=1, and the paper does not examine how well it describes the actual distillation gradient at the temperatures used.

Questions for authors
  1. Can you use standard temperature scaling for each trained student (i.e. fit a single temperature per student on a val set) and then show the U-shaped curves and the norm-replacement results before and after scaling? This would show whether the improvements remain once the student’s confidence is unfactored.

  2. Can you report test accuracy (with standard deviations) in tables for the maturity sweep and the norm replacement experiment?

  3. Norm replacement is equivalent to changing the teacher’s temperature separately for each input, an interpretation that has also been made for per input logit normalization [7]. How does the effect compare to distilling from the mature teacher with well-tuned teacher temperature?

  4. The teacher’s test loss should measure how closely its predictions approximate the true class probabilities. Does it predict student quality across teacher maturities as well as the teacher’s logit norm does? What does the teacher’s logit norm view explain that the calibration view doesn't?

  5. Can you report results for standard KD settings such as with student data augmentation, T=4, and a standard LR scheduler? Or a different setting where the students reach standard accuracies?

  6. Since the analysis is done for T that goes to inf, how well does it describe the actual distillation gradient at the temperatures used in the experiments (T=1,4)?

[1] On calibration of modern neural networks, ICML 2017.

[2] Reliability, sufficiency, and the decomposition of proper scores, Quarterly Journal of the Royal Meteorological Society 2009.

[3] Revisit the essence of distilling knowledge through calibration, ICML 2024.

[4] The role of teacher calibration in knowledge distillation, IEEE Access 2025.

[5] SGD-based knowledge distillation with Bayesian teachers: Theory and guidelines, ICLR 2026.

[6] Theoretical perspectives on knowledge distillation: A review. Wiley Interdisciplinary Reviews: Computational Statistics, 2025.

[7] Mitigating neural network overconfidence with logit normalization, ICML 2022.

[8] Deep residual learning for image recognition, CVPR 2016.

[9] Contrastive representation distillation, ICLR 2020.

[10] Revisiting knowledge distillation via label smoothing regularization, CVPR 2020.

6 comments
AuthorsOpenReview ↗

Response to Reviewer oqYW (1/4): W1

See Tables A–B in Common Response 3/6 (comment 3/13), Tables C–D in 4/6 (comment 4/13), and Table E in 5/6 (comment 5/13).

We thank the reviewer for the detailed comments. In additional experiments, we examined the effects of post-hoc student temperature scaling, changes in training settings, and the validity of the high-temperature approximation.

W1. Calibration-based explanation and related work

At T=1, students directly distilled from immature teachers still had lower test loss than those distilled from mature teachers after post-hoc temperature scaling (differences of 0.097/0.312/0.256 on CIFAR-10/CIFAR-100/Tiny ImageNet; Table A). By contrast, the improvement in test loss from norm replacement decreased after calibration, reaching only 0.003 on CIFAR-10 (Table B). On CIFAR-100 and Tiny ImageNet, students obtained similar accuracy from single-temperature scaling of the mature teacher and from norm replacement (Table C). The shrinkage of the norm-replacement test-loss reduction after calibration and the accuracy gains under Condition D are consistent with calibration-related findings by Fan et al. (2024; [3] in your review) and Kim et al. (2025; [4] in your review). Condition D likewise applies a single temperature to the mature teacher's outputs; it improved accuracy on CIFAR-100 and Tiny ImageNet, but did not reach the accuracy obtained from the immature teacher. The selected temperatures (1.64–2.29) lie within the 1.5–3 range reported by Kim et al. (2025) to give robust gains under their experimental settings.

Lower teacher test loss does not necessarily lead to higher accuracy of the distilled student. On CIFAR-100 in Section 5, teacher test loss decreased from 1.969 to 1.742, while student accuracy declined from 44.0% to 39.2%. Among checkpoints from epoch 50 onward in Section 4, Spearman rank correlation between teacher test loss and student accuracy on CIFAR-100 was +0.20 (oqYW Q4 in response 4/4; comment 13/13).

When teacher logit norms were matched for each input (Conditions B and C), student accuracy still differed by 2.3, 4.0, and 3.5 percentage points (Table C). Direct distillation from the immature teacher also exceeded distillation from the single-temperature-scaled mature teacher by 3.0, 3.0, and 0.8 points (Table C). These comparisons show that matching the per-input teacher logit norm does not reproduce the accuracy achieved by direct distillation from the immature teacher. We will discuss the calibration findings of Fan et al. (2024) and Kim et al. (2025), the SGD-based theory of Morad et al. (2026; [5] in your review), and the theoretical survey of Liu et al. (2025; [6] in your review) in the revision.

AuthorsOpenReview ↗

Response to Reviewer oqYW (2/4): W2 and Q1

For Tables A–B see Common Response 3/6 (comment 3/13); for Tables D–E see 4/6–5/6 (comments 4–5/13).

W2. Non-standard training settings and scope of the high-temperature analysis

At T=1 without student data augmentation, immature teachers achieved higher mean test accuracy under the alternative learning-rate schedules and optimizers tested. Changing distillation temperature or introducing data augmentation reduced or reversed the advantage in some settings (Tables D and E; oqYW Q5 (response 4/4; comment 13/13)). On CIFAR-10, gains with Adam (+1.1 points) and AdamW (+0.6 points) were modest relative to the variation across three student seeds.

On CIFAR-10, students distilled from the mature teacher achieved approximately 72% accuracy under the original settings, 81.5% with Adam, and 85.8% with data augmentation. The student distilled from the mature teacher under the original settings reached 99.9% training accuracy on CIFAR-10. Thus, its low test accuracy cannot be attributed to a failure to fit the training set. We will correct the description of α=0.9 as nonstandard; a distillation weight of 0.9 was also used by Tian et al. (2020; [9] in your review) and Kim et al. (2025, on CIFAR-100). The small constant learning rate, the absence of student data augmentation, and the single student seed (seed 0) for WRN-16-2 in Appendix H are stated in Section 4.1 and Appendix C; we will make them more prominent in the main text. We address the high-temperature approximation in oqYW Q6 (response 4/4; comment 13/13).

Q1. Student temperature scaling

Under direct distillation at T=1, the minimum calibrated student test loss for immature teachers remained below that for the mature teacher. Relative to the mature teacher, the reduction in minimum test loss before/after calibration was 0.412/0.097 on CIFAR-10, 0.584/0.312 on CIFAR-100, and 0.365/0.256 on Tiny ImageNet (Table A).

For norm replacement, the corresponding reductions changed from 0.226 to 0.003, from 0.337 to 0.094, and from 0.302 to 0.194. No clear improvement remained on CIFAR-10 (Table B). The best teacher checkpoint or reference maturity was selected separately before and after calibration. Temperatures were fitted on 20% of the test set, and losses were evaluated on the remaining 80%.

At T=4 (Section 5 teachers, cosine decay), the mature teacher yielded the lowest calibrated student test loss. At T=64 (Section 4 settings), the test-loss reduction relative to the mature teacher retained 1.9% and 24.0% of its pre-calibration magnitude on CIFAR-10 and CIFAR-100, respectively. We have not evaluated the intermediate student-training trajectories in Figures 3 and 4 after calibration.

AuthorsOpenReview ↗

Response to Reviewer oqYW (3/4): Q2–Q3

Q2. Test accuracy with standard deviations

We report student test accuracy and standard deviations for each teacher maturity and norm-replacement condition. Values are mean ± standard deviation over three student seeds (%, T=1, full test set).

Direct distillation: CIFAR-10/CIFAR-100 (Section 4)

Teacher epochCIFAR-10CIFAR-100
142.52 ± 0.3912.74 ± 0.34
246.69 ± 0.0211.79 ± 0.28
557.81 ± 0.2316.35 ± 0.13
1067.66 ± 0.2621.07 ± 0.23
2574.06 ± 0.4833.61 ± 0.11
3073.50 ± 0.3638.12 ± 0.08
3575.26 ± 0.6440.24 ± 0.13
4071.30 ± 0.9140.90 ± 0.33
4574.87 ± 0.5342.66 ± 0.33
5075.16 ± 0.4043.83 ± 0.29
10073.68 ± 0.8243.53 ± 0.56
15073.56 ± 0.8541.81 ± 0.46
20073.03 ± 1.6540.23 ± 0.68
400 (mature teacher)72.49 ± 0.9038.61 ± 0.56

On CIFAR-100, the epoch-50 teacher yielded 5.22 percentage points higher student accuracy than the mature teacher. On CIFAR-10, the epoch-35 and epoch-50 teachers yielded gains of +2.77 and +2.67 points, respectively, whereas the epoch-40 teacher performed worse than the mature teacher.

Direct distillation: Tiny ImageNet (teacher-maturity comparison)

Teacher epochStudent test accuracy (%)
17.23 ± 0.05
516.69 ± 0.35
1027.19 ± 0.14
2538.59 ± 0.18
5041.72 ± 0.09
100 (mature teacher)37.81 ± 0.26

The epoch-50 teacher outperformed the mature teacher by 3.91 percentage points.

Norm replacement: CIFAR-10/CIFAR-100/Tiny ImageNet (Section 5, Figure 6)

Reference maturity (epochs)CIFAR-10CIFAR-100Tiny ImageNet
166.40 ± 1.2126.44 ± 0.4530.27 ± 0.10
5——40.96 ± 0.41
1070.99 ± 1.0940.79 ± 0.4040.21 ± 0.39
2572.01 ± 0.8040.20 ± 0.1039.01 ± 0.41
5072.24 ± 1.2640.03 ± 0.5738.27 ± 0.08
10071.81 ± 0.8840.08 ± 0.44—
No replacement72.30 ± 1.3639.05 ± 1.0437.81 ± 0.26

Relative to no replacement, norm replacement did not improve accuracy on CIFAR-10 but yielded gains of up to +1.74 points on CIFAR-100 and +3.15 points on Tiny ImageNet. We will add test accuracies and standard deviations to the relevant figures and tables in the revised paper.

Q3. Teacher temperature scaling vs. norm replacement

At T=1, single-temperature scaling of the mature teacher and the best norm replacement yielded similar student accuracies. Accuracy was 71.6% vs. 72.3% on CIFAR-10, 41.0% vs. 40.9% on CIFAR-100, and 41.0% vs. 40.8% on Tiny ImageNet (80% test subset; Table C). On CIFAR-10, neither clearly exceeded the 72.2% accuracy obtained from the unmodified mature teacher.

Calibrated student test losses for temperature scaling / norm replacement were 0.863/0.841 on CIFAR-10, 2.273/2.284 on CIFAR-100, and 2.496/2.492 on Tiny ImageNet. The best reference maturity for norm replacement was selected separately for accuracy and test loss. Neither method matched direct distillation from the epoch-50 teacher (74.6%, 44.0%, and 41.8%, respectively).

The temperature was selected by minimizing teacher test loss; we did not search for a temperature that maximized student accuracy. As an example showing that the temperature minimizing teacher test loss need not improve student accuracy, a T=4 experiment under the Section 5 setting (cf. Q5) found that temperature scaling reduced accuracy from 81.5% to 77.7% on CIFAR-10, from 49.3% to 47.5% on CIFAR-100, and from 47.1% to 43.4% on Tiny ImageNet (80% test subset).

Reviewer_P9hZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates why immature teachers can sometimes outperform mature teachers in knowledge distillation. The authors propose that the teacher logits norm controls the balance between knowledge transfer and regularization in a high-temperature environment of the distillation loss. Experiments on four classification datasets reveal a U-shaped relationship between student test loss and student logit norm. Furthermore, replacing only the logit norm of a mature teacher with that of an immature teacher improves student performance, which suggests that teacher logit magnitude plays an important role in distillation effectiveness.

Critical strengths
  1. This paper addresses an important question in knowledge distillation: why immature teachers can sometimes outperform mature teachers. The invetiation of this phenomenon provides valuable insights into the mechanisms underlying knowledge distillation.

  2. The norm-replacement experiments demonstrate that adjusting the teacher logit norm improves student performance without changing the teacher's predicted classes. These results provide important evidence for the importance of logit magnitude in knowledge distillation.

Critical weaknesses
  1. The U-shaped relationship between student test loss and logit norm may simply result from the sensitivity of cross-entropy to prediction confidence, rather than the proposed alignment–regularization mechanism.

  2. The theoretical decomposition relies on a high-temperature approximation, while the main experiments use T=1( in Section 3). Although experiments at higher temperatures are provided, they do not directly validate the proposed mechanism at T=1.

  3. The paper lacks a clear and central argument. The experiments mainly demonstrate that both mature teacher logit norm and direction affect student performance, without sufficiently explaining how their interaction determines the optimal teacher maturity or providing direct evidence for the proposed mechanism.

  4. In the main expeiment, it is unclear whether the teacher models on CIFAR-10/100 and Tiny ImageNet were independently trained with multiple random seeds or whether all checkpoints were obtained from a single training run.

  5. The main point is that this is a findings to explain why immature teachers can sometimes outperform mature teachers in knowledge distillation, what is the main argument of this paper?

Questions for authors
  1. Have the authors evaluated students trained without knowledge distillation (α=0)? Including this baseline in the main experiments is needed clarify the actual benefits of KD.

  2. Why is teacher test loss not reported in Figures 3–4? Does it exhibit a similar U-shaped trend across training epoches. If so, how can the authors distinguish the proposed mechanism from the teacher's own calibration effects?

  3. Why is a cosine learning rate schedule used in Section 5, while Section 4 uses a constant learning rate on CIFAR-10/100? Does this change affect the experimental conclusions?

  4. Have the authors tried other methods such as temperature scaling or logit normalization, to achieve the same effect as norm-replacement?

4 comments
AuthorsOpenReview ↗

Response to Reviewer P9hZ (1/2): W1–W5

The additional results appear in Common Response 3/6 (Tables A–B; comment 3/13), 4/6 (Tables C–D; comment 4/13), and 5/6 (Table E; comment 5/13).

We thank the reviewer for the detailed questions.

W1. U shape and cross-entropy sensitivity to confidence

After post-hoc temperature scaling of the students, the reduction in test loss from norm replacement decreased from 0.226 to 0.003 on CIFAR-10, from 0.337 to 0.094 on CIFAR-100, and from 0.302 to 0.194 on Tiny ImageNet (Table B). No clear improvement remained on CIFAR-10.

For direct distillation at T=1, however, the minimum calibrated test loss for immature teachers remained below that for mature teachers by 0.097, 0.312, and 0.256, respectively (Table A; the best teacher epoch was selected separately before and after calibration). Post-hoc temperature scaling also leaves accuracy unchanged. Thus, the 2.4-, 4.8-, and 4.0-point accuracy advantages of direct distillation from the epoch-50 teacher over the mature teacher remain unchanged (Table C).

At T=4 (using the Section 5 teachers and cosine learning-rate decay), the mature teacher yielded the lowest calibrated test loss. At T=64 (under the Section 4 teacher-training settings), the test-loss reduction relative to the mature teacher retained 1.9% and 24.0% of its pre-calibration magnitude on CIFAR-10 and CIFAR-100, respectively. We will revise the interpretation in Section 4.4 that observing a U-shaped relationship at high temperatures supports our hypothesis, distinguishing the original observations from the post-calibration results. We have not evaluated the intermediate student-training trajectories in Figures 3 and 4 after calibration.

W2. High-temperature approximation vs. experiments at T=1

At T=1, the approximate gradient from Equation (7) was not sufficiently aligned with the exact distillation gradient to explain the observed performance differences directly. We measured cosine similarity by changing only the temperature at which gradients were evaluated, using students trained at T=1. We used teachers retrained for the additional experiments and report means over three student seeds on training inputs without augmentation (CIFAR-10 / CIFAR-100).

Teacher epochT=1T=4T=64
10.74 / 0.870.97 / 0.981.00 / 1.00
500.45 / 0.210.83 / 0.761.00 / 1.00
Mature teacher0.64 / 0.180.81 / 0.581.00 / 1.00

At T=64, similarity reached 1.00 when rounded to two decimal places. We will distinguish the high-temperature analysis in Equations (6) and (7) from the interventions on logit norm and direction performed at T=1.

W3 and W5. Central claim, norm–direction interaction, and direct evidence

Changing only the teacher logit norm altered student accuracy on CIFAR-100 and Tiny ImageNet. Moreover, even when the logit norm was matched for each input, teacher logit direction affected student accuracy. Conditions B and C differed by 2.3, 4.0, and 3.5 percentage points on CIFAR-10, CIFAR-100, and Tiny ImageNet, respectively (Table C). The additional comparison using independently trained teachers under another training configuration gave 76.0% vs. 73.1% on CIFAR-10 and 42.4% vs. 38.4% on CIFAR-100.

Regarding their interaction, the teacher logit norm and the usefulness of its direction as an alignment target both increase with maturity, while a large norm weakens the relative contribution of regularization (Section 6). The best maturity therefore reflects both factors: with the norm matched, the test-loss-minimizing reference maturity was lower with the mature teacher's direction than with direct distillation (Section 5.3, Figure 6). We do not have a quantitative model that predicts the optimal maturity.

The high-temperature approximation describes how the norm and direction affect the alignment gradient, but does not establish that this mechanism produces the T=1 performance differences (W2). The revision will distinguish the theoretical regime from the empirical matched-norm results.

W4. Teacher random seeds

In the main CIFAR-10/CIFAR-100 experiments at T=1, we did not repeat teacher training with only the teacher random seed changed under otherwise identical settings. The teachers used in Sections 4 and 5 and the additional experiments were trained independently, but their learning-rate schedules and other conditions differ.

Nevertheless, the immature teacher outperformed the mature teacher in all three experiments without student data augmentation. On CIFAR-100, student accuracy for the immature/mature teachers was 43.8%/38.6% in Section 4, 44.0%/39.2% in Section 5, and 42.2%/36.9% in the additional experiment (best immature teacher at epoch 100). The comparisons in Sections 4 and 5 use the epoch-50 teacher. Reported standard deviations reflect variation across three student seeds with each teacher fixed.

AuthorsOpenReview ↗

Response to Reviewer P9hZ (2/2): Q1–Q4

Q1. Baseline without distillation (α=0)

We evaluated students trained without distillation (α=0). On both CIFAR-10 and CIFAR-100, students distilled from the epoch-50 teacher outperformed them. The table reports test accuracy (mean ± standard deviation, %, three student seeds) under the settings of Section 4 (T=1).

DatasetNo distillationEpoch-50 teacherMature teacher
CIFAR-1073.1 ± 1.175.2 ± 0.472.5 ± 0.9
CIFAR-10033.5 ± 0.243.8 ± 0.338.6 ± 0.6

On CIFAR-10, the student distilled from the mature teacher performed similarly to the student trained without distillation. We will add the α=0 baseline to the main results.

Q2. Teacher test loss and teacher calibration effects

Teacher test loss initially decreased and then increased, but its minimum did not coincide with the teacher yielding the most accurate student. In Section 4, teacher test loss was lowest at epoch 63 on CIFAR-10 and epoch 139 on CIFAR-100, whereas the most accurate students among the evaluated teacher checkpoints came from the epoch-35 teacher on CIFAR-10 (75.3%) and the epoch-50 teacher on CIFAR-100 (43.8%). Teacher test loss is already reported in Appendix E; we will also include it in the main text.

Across 14 checkpoints, Spearman rank correlation between teacher test loss and student accuracy was −0.81/−0.83 on CIFAR-10/CIFAR-100. In Section 5 on CIFAR-100, teacher test loss decreased from 1.969 to 1.742, while student accuracy fell from 44.0% to 39.2% (80% test subset). Lower teacher test loss therefore does not always yield higher student accuracy.

In response to the reviewer's concern about teacher confidence, we also distilled students after scaling the mature teacher's logits by a single temperature selected to minimize teacher test loss. Their accuracy remained below that obtained by direct distillation from the immature teacher (Table C; Common Response 4/6, comment 4/13).

Q3. Constant vs. cosine learning-rate schedules

In Section 4 on CIFAR-10/CIFAR-100, we used a small constant learning rate to observe how teacher outputs change with maturity; in Section 5, we used cosine decay to reduce the teacher learning rate late in training and stabilize the final checkpoint. We retrained both teachers and students for Section 5; Section 4 also uses cosine decay on Tiny ImageNet. The small learning rate in Section 4 was chosen because, with a larger rate, the teachers converge within a few epochs and output differences across checkpoints become small (Appendix C).

In additional T=1 experiments varying only the student's learning-rate schedule (constant/cosine/linear), immature teachers outperformed mature teachers under all three schedules. The accuracy differences were 3.6/3.0/2.5 percentage points on CIFAR-10 and 5.3/6.1/6.1 points on CIFAR-100 (Table E). These experiments did not vary the teacher's learning-rate schedule.

Q4. Temperature scaling and logit normalization

We tested a single-temperature adjustment of the mature teacher, but not logit normalization as a separate baseline. The temperature was selected by teacher test loss rather than student performance (Condition D). As noted in Section 2.3 of our paper, norm replacement is equivalent to an input-dependent temperature applied to the teacher logits. A related connection between logit norms and temperature appears in the analysis of LogitNorm by Wei et al. (2022), although LogitNorm is a different training method.

On CIFAR-100 and Tiny ImageNet, the student accuracies obtained with temperature scaling and the best norm replacement were similar (Table C). Their calibrated test losses were 2.273/2.284 and 2.496/2.492, respectively (temperature scaling / norm replacement). Neither method gave a clear improvement over the mature teacher on CIFAR-10. In the comparisons conducted here, we found no advantage of norm replacement over single-temperature scaling.

Thank you for your detailed responses and the additional experiments. I have read all of the replies. Your responses strengthen the paper’s experimental evidence and clarify its argument. But my main concern about whether the high-temperature analysis explains the results at T=1 remains, so I will keep my score. Thank you again for your replies.

Thank you very much for carefully reviewing our responses and additional experimental results. We also appreciate your recognition that these results strengthened our experimental evidence and clarified our argument.

In future versions of this work, we will clarify the scope of the theoretical analysis based on the high-temperature approximation and further investigate the mechanisms underlying the effectiveness of immature teachers at T=1.

Thank you again for your time and thoughtful consideration.

Reviewer_YW4DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reason why immature teacher can produce better students than the final checkpoint in knowledge distillation. It tried to understand this phenomenon by decomposing the distillation loss into student logit-norm regularization and teacher-student alignment. Four image and text classification datasets are used to examine the influence of teacher maturity. Changing the scale of a mature teacher’s logits can lower the student’s test loss. Even when teachers have the same logit norms, their students can perform differently, suggesting that logit direction matters too.

Critical strengths
  1. The experiment in Section 5 is the most convincing part of the paper. Keeping the mature teacher’s logit direction fixed and changing only its norm helps isolate what output scale contributes. This gives us more information than simply observing that one checkpoint teaches better than another.
  2. The authors also show where their explanation stops working. They report cases in Appendix H where the U-shaped curve disappears. I appreciate this openness: it makes the findings more credible and gives a clearer picture of what the explanation can account for.
Critical weaknesses

1 My main concern is what we learn beyond Yuan et al. (2020), who already showed that poorly trained teachers can help through soft-target regularization. The norm-replacement experiment is interesting, but a lower test loss could simply mean that the student’s confidence is better calibrated. I would find the explanation more convincing if the benefit remained after calibration. Otherwise, it is hard to tell how much the paper adds to the earlier account.

2 I am also unsure how far these findings extend beyond the chosen setup. Students are trained without augmentation and with a large distillation weight, and the mature teacher is the final checkpoint from that training recipe. Appendix H already shows cases where mature teachers cause no degradation. The paper demonstrates that intermediate checkpoints can work better in some settings, but gives limited guidance on when to expect this advantage under other training conditions.

Questions for authors

These are clarification requests about the current claims and any results already available.

  1. Have you checked whether the norm-replacement benefit remains after calibrating the student’s predictions? If so, how much of the test-loss improvement remains?

  2. What do you see as the main finding beyond Yuan et al. (2020)? Which result best distinguishes your explanation from the existing account that softer targets provide useful regularization?

  3. How was the “mature” teacher checkpoint chosen? Were its training curves sufficiently stable to establish it as a meaningful reference, rather than simply the endpoint of the training budget?

  4. Given the restricted training setup and the exceptions in Appendix H, when should we expect immature teachers to help? Do any existing results support the explanation with student augmentation or a smaller distillation weight?

Yuan, L., Tay, F. E. H., Li, G., Wang, T., & Feng, J. (2020). Revisiting Knowledge Distillation via Label Smoothing Regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3903–3911.

1 comment
AuthorsOpenReview ↗

Response to Reviewer YW4D

Please also see Tables A–B in Common Response 3/6 (comment 3/13), Tables C–D in 4/6 (comment 4/13), and Table E in 5/6 (comment 5/13).

We thank the reviewer for the constructive questions.

Q1. Post-hoc calibration of norm replacement

After post-hoc temperature scaling, the reduction in student test loss due to norm replacement was 0.003 on CIFAR-10, 0.094 on CIFAR-100, and 0.194 on Tiny ImageNet, compared with 0.226, 0.337, and 0.302 before calibration (Table B). No clear improvement remained on CIFAR-10. The reference maturity minimizing test loss was selected separately before and after calibration.

On CIFAR-10, when we fixed the norm source to epoch 10, which was optimal before calibration, the calibrated test loss was 0.884, higher than the 0.844 without replacement. The best reference maturity after calibration was epoch 100, yielding a test loss of 0.841 (a reduction of only 0.003).

Post-hoc temperature scaling does not change the predicted class and therefore leaves test accuracy unchanged. Relative to no replacement, the accuracy differences after norm replacement were +0.1, +1.7, and +3.0 percentage points, respectively (80% test subset; Table C).

Q2. Main contribution beyond Yuan et al. (2020)

Yuan et al. (2020) explained the benefits of KD in terms of regularization by soft targets. Under the high-temperature approximation, we decompose the distillation loss into a logit norm penalty and a teacher–student alignment term, and show that the teacher logit norm scales the magnitude of the alignment gradient.

The most direct evidence is the matched-norm comparison between Conditions B and C. When the teacher logit norm was identical for each input, students trained with the immature teacher's direction achieved 2.3, 4.0, and 3.5 percentage points higher accuracy than those trained with the mature teacher's direction on CIFAR-10, CIFAR-100, and Tiny ImageNet, respectively (Table C). Thus, differences in student performance across teacher maturities cannot be explained by differences in per-input teacher logit norms alone. Beyond Yuan et al. (2020), our finding is that student performance differences persist even when the teacher logit norm is matched for each input.

Q3. Selection and stability of the mature teacher

We defined the mature teacher as the final checkpoint of the predetermined training schedule. In the first and second halves of the last 100 epochs (50 epochs each), teacher test accuracy was 87.1%/87.2% and test loss was 0.631/0.632 on CIFAR-10; the corresponding values on CIFAR-100 were 59.0%/59.1% and 1.728/1.731. These values were stable near the end of training. The minimum teacher test loss, however, occurred at epochs 63 and 139, respectively, rather than at the final checkpoint. On CIFAR-100, students distilled from teachers at epochs 100 and 150, near the teacher-loss minimum, achieved 43.53% and 41.81% accuracy, respectively, compared with 43.83% for the epoch-50 teacher (oqYW Q2; comment 12/13).

Q4. Applicability under different training settings and Appendix H

At T=1 without student data augmentation, immature teachers outperformed mature teachers. Increasing distillation temperature or introducing data augmentation reduced the gap, and mature teachers performed better in some settings (Tables D and E). The higher mean accuracy also persisted under the alternative learning-rate schedules and optimizers tested at T=1 without augmentation, although the CIFAR-10 gains with Adam (+1.1 points) and AdamW (+0.6 points) were modest relative to variation over three student seeds (Table E).

Using the same teachers as in Table E, we varied the distillation weight α (T=1, no data augmentation; three student seeds; full test set). Each cell reports test accuracy for the best immature teacher / mature teacher (mean ± standard deviation, %). Parentheses indicate the immature teacher's epoch.

αCIFAR-10CIFAR-100
0.173.5 ± 0.5 (40) / 72.8 ± 1.435.9 ± 0.4 (40) / 34.2 ± 1.1
0.575.8 ± 1.0 (45) / 72.0 ± 0.441.1 ± 0.1 (50) / 37.1 ± 0.3
0.976.1 ± 0.8 (50) / 72.5 ± 0.542.2 ± 1.0 (100) / 36.9 ± 0.1

At α=0.1, the difference was +0.7 percentage points on CIFAR-10 (small relative to the variation across student seeds) and +1.7 points on CIFAR-100. At α=0.5, the differences were +3.8 and +4.0 points, respectively. With data augmentation (α=0.9), accuracy was 85.9% vs. 85.8% on CIFAR-10 and 58.3% vs. 59.2% on CIFAR-100 (Table E).

The U shape was absent with TextCNN on AG News and WRN-16-2 on Tiny ImageNet (Appendix H), so it depends on the dataset and student. As discussed there, we hypothesize that immature teachers help when a mature teacher's strong alignment pushes the student past the minimum of the U, which may depend on task difficulty for the student.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Common Response (1/6): Main Claims

We thank the reviewers for their careful comments. In additional experiments, we examined how student performance depends on the norm and direction of the teacher logits, temperature calibration, distillation temperature, and student training settings. We summarize the results and planned revisions below.

Main claims of the paper

We studied the effectiveness of immature teachers by examining the relationship between regularization and teacher–student alignment in the distillation loss. Our main claims are as follows.

  1. Empirical claim. Among the changes in teacher logits that occur as the teacher matures, changing only the logit norm alters student test accuracy on CIFAR-100 and Tiny ImageNet. However, the dependence of student performance on teacher maturity cannot be explained by the teacher logit norm alone; it also depends on the direction of the teacher logits.

  2. Theory-based interpretation. Under the high-temperature approximation, decomposing the distillation loss shows that the teacher logit norm scales the gradient magnitude of the teacher–student alignment term, changing its strength relative to the logit norm penalty. The teacher logit direction determines the direction of the alignment gradient. These results suggest that explaining the effectiveness of immature teachers requires considering not only the magnitude of the teacher logits but also differences in their direction, which determines the target of teacher–student alignment.

AuthorsOpenReview ↗

Common Response (2/6): Conditions A–D and Empirical Comparisons

Comparison conditions and results

We trained students under four conditions that modify the norm or direction of the teacher logits. Condition D was added in response to the reviews. Teacher temperature scaling has been studied as a calibration-based KD intervention by Kim et al. (2025); our experiment tests a single temperature selected by teacher NLL, rather than searching for the temperature that maximizes student accuracy.

  • Condition A (mature teacher): Direct distillation from the final teacher checkpoint.
  • Condition B (immature teacher): Direct distillation from the teacher at epoch 50.
  • Condition C (norm replacement): Preserve the mature teacher's logit direction, but replace the norm for each input with that of the teacher in Condition B.
  • Condition D (teacher temperature scaling): Divide the mature teacher's logits by a single temperature shared across all inputs, without changing their direction.

The table reports student test accuracy (mean ± standard deviation, %, over three student seeds) for distillation at T=1. Evaluation used 80% of the test set. For Condition D, we followed the single-temperature NLL-fitting procedure of Guo et al. (2017): the temperature was selected to minimize the teacher's negative log-likelihood on 20% of the test set, disjoint from the 80% used for evaluation (CIFAR-10: 2.29; CIFAR-100: 1.64; Tiny ImageNet: 1.64). Unlike the usual use of a separate validation set, the fitting split here was drawn from the test set.

ConditionDirectionNormCIFAR-10CIFAR-100Tiny ImageNet
AMatureMature72.2 ± 1.439.2 ± 1.037.8 ± 0.3
BImmatureImmature74.6 ± 0.544.0 ± 0.741.8 ± 0.2
CMatureImmature (epoch 50)72.3 ± 1.540.0 ± 0.638.3 ± 0.1
DMatureScaled by a single temperature71.6 ± 1.441.0 ± 0.841.0 ± 0.4

1. Matched-norm comparison (B vs. C). The teacher logit norms are identical for every input; only their directions differ. Students in Condition B outperformed those in Condition C by 2.3, 4.0, and 3.5 percentage points on CIFAR-10, CIFAR-100, and Tiny ImageNet, respectively. The same comparison using teachers trained independently under a different training configuration also favored the immature teacher's direction (76.0% vs. 73.1% on CIFAR-10; 42.4% vs. 38.4% on CIFAR-100; three student seeds per condition). The observed performance differences cannot be explained by logit-norm differences alone. These comparisons are not repetitions in which only the teacher's random seed was changed while holding all training settings fixed.

2. Temperature scaling of the mature teacher (A vs. D). Condition D improved accuracy over Condition A by 1.8 percentage points on CIFAR-100 and 3.2 points on Tiny ImageNet, but decreased it by 0.6 points on CIFAR-10. This partially agrees with the teacher-scaling improvements reported by Kim et al. (2025), although the architectures, distillation settings (T=4 for CIFAR-100 and T=1 for ImageNet in Kim et al.), and teacher-temperature selection procedures differ.

3. Immature teacher vs. temperature-scaled mature teacher (B vs. D). Condition B outperformed Condition D by 3.0, 3.0, and 0.8 percentage points, respectively. The single-temperature adjustment examined here did not close the gap to direct distillation from the immature teacher.

Our additional experiments establish that student accuracy can differ at identical teacher logit norms, that single-temperature scaling of a mature teacher did not match the accuracy obtained from the immature teacher, and that the advantage of immature teachers varies with distillation temperature and data augmentation. We will state these empirical findings explicitly in the revised paper. We address the scope of the high-temperature approximation and its implications for the mechanism at T=1 in our responses to P9hZ (W2; comment 8/13) and oqYW (Q6; comment 13/13).

AuthorsOpenReview ↗

Common Response (3/6): Tables A–B (Post-hoc Calibration)

Table A: Student test loss after direct distillation from teachers at different maturities (before / after calibration)

We applied post-hoc temperature scaling to students trained under the settings of Section 5 and Figure 6 of the submission. Following the post-hoc temperature-scaling procedure of Guo et al. (2017), we fitted one temperature per student by minimizing NLL on 20% of the test set; test loss before and after calibration was evaluated on the remaining, disjoint 80%. Entries are means over three student seeds at T=1. Because calibration temperatures were fitted on data disjoint from the evaluation subset, post-calibration test loss can occasionally increase slightly.

Teacher epochCIFAR-10CIFAR-100Tiny ImageNet
11.600 / 1.5704.142 / 4.0254.691 / 4.661
100.986 / 0.9853.190 / 3.1633.172 / 3.173
250.761 / 0.7482.538 / 2.5372.531 / 2.529
500.790 / 0.7472.066 / 2.0662.457 / 2.430
1000.898 / 0.7872.191 / 2.161—
Mature teacher1.173 / 0.8442.650 / 2.3782.822 / 2.686

Table B: Student test loss after distillation from norm-replaced teachers (before / after calibration)

We fixed the direction of the mature teacher's logits and replaced their norm for each input with the norm of a teacher at an earlier checkpoint. Students trained under each condition were calibrated post hoc using the same procedure as in Table A. The settings follow Section 5 and Figure 6 of the submission (T=1; mean over three student seeds).

Reference maturity (epochs)CIFAR-10CIFAR-100Tiny ImageNet
11.093 / 1.0843.687 / 3.1033.730 / 3.065
5——2.530 / 2.492
100.947 / 0.8842.334 / 2.2842.520 / 2.520
251.017 / 0.8622.313 / 2.2972.643 / 2.601
501.029 / 0.8492.382 / 2.3072.733 / 2.645
1001.087 / 0.8412.475 / 2.333—
No replacement1.173 / 0.8442.650 / 2.3782.822 / 2.686

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.