acceptodds
Withdrawn from ICLR 2027

The Same Target Can Help or Hurt: Regime-Dependent Sharpness Utility in Knowledge Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Should teacher targets be softened or sharpened in knowledge distillation? We study this question with a fixed prediction-preserving intervention, , that changes target sharpness while preserving logit ordering and the teacher's top-1 prediction. In a controlled ImageNet-100 DeiT setting, we fix the dataset, architecture pair, teacher, KD objective, and continuation protocol, and vary only the student starting state. The preferred direction reverses between the predeclared endpoints: softening is favored from the untrained state, whereas sharpening is favored from the trained state, with a planned endpoint contrast of pp. Broader regimes exhibit the same directional non-invariance: controlled CIFAR is softening-favored, matched scratch DeiT shows no measurable directional preference, and pretrained DeiT is sharpening-favored. Gradient, difficulty, target-component, and functional-geometry diagnostics constrain simple explanations but yield no regime-independent directional rule. Target-sharpness utility is therefore relational rather than target-intrinsic: the same intervention can favor opposite directions under different supervision regimes.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Knowledge Distillation, Teacher–Student Learning, Target Sharpness, Logit Scaling, Supervision Regimes, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nYKQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper empirically investigates when softer versus sharper teacher targets are preferable under temperature scaling, with a particular focus on different student training states. The experiments demonstrate a controlled reversal in the preferred direction on ImageNet-100 with the DeiT family and identify distinct directional preferences across broader experimental settings.

Critical strengths

None

Critical weaknesses
  1. Poor readability and unclear logical flow.

The paper is difficult to follow, particularly in terms of its main contributions, interpretation of experimental results, and overall logical structure.

First, the contributions listed in the Introduction are not clearly articulated. It is difficult to understand how these individual contributions connect to form a coherent scientific insight.

Second, the presentation of experimental results is inconsistent across sections. Section 4 reports results on ImageNet-100 with DeiT models in Table 1, whereas Section 5 introduces controlled CIFAR experiments with different student states (rather than S0S_0 and S4S_4) and additional teacher states. Furthermore, the results for scratch-trained and pretrained DeiT models are presented as Equations (6)–(9), rather than in tables or figures. These differences in experimental configurations and presentation formats make it difficult to compare results across sections. Table 2 also uses inconsistent reporting formats across columns (e.g., "28/28 positive" versus "-0.060 [-0.38, +0.25]"), further reducing clarity.

Third, the logical transition between Sections 5 and 6 is insufficiently explained. Section 5 investigates directional preferences across various experimental settings, while Section 6 abruptly introduces three "natural reductions of the softening-favored response" in the controlled CIFAR setting. However, the paper does not adequately explain why these analyses follow from the preceding observations or why the three approaches are considered natural reductions. In particular, concepts such as "CE–KD gradient magnitude," "example correctness and difficulty," and "isolated components of the teacher target" are introduced without sufficient motivation or explanation in the main text. Relevant details appear to be relegated to Appendix C without an explicit reference, and experiment details. Moreover, the terminology used to describe these analyses is inconsistent across the Introduction, Section 6, and Figure 3.

Overall, these issues substantially hinder the reader's ability to understand the paper's central message and how the experimental findings support its conclusions.

  1. Insufficient justification of experimental settings.

The paper draws conclusions about the preferred direction of temperature scaling based on student models trained for 20 or 30 epochs. However, the authors do not sufficiently justify these training budgets. In particular, it is unclear whether the selected training durations are sufficient for convergence or whether the observed directional preferences remain consistent under longer training schedules. Without such evidence, it is difficult to determine whether the conclusions reflect general properties of temperature scaling or are specific to the chosen training budgets.

Questions for authors

See the weaknesses above.

Reviewer_zHbbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether softening or sharpening teacher predictions is preferable in knowledge distillation (KD). Using fixed logit rescaling, the authors show that the preferred direction reverses between untrained and trained student states (Sec. 4 and Table 1). Additional experiments and diagnostics suggest that sharpness utility depends on the supervision regime rather than the target transformation alone.

Critical strengths
  1. Controlled experimental design. The authors isolate student starting state while keeping the teacher, dataset, and optimization protocol fixed, providing evidence for a directional reversal (Table 1).

  2. Careful statistical analysis. The endpoint reversal is replicated across three trajectories, with confidence intervals and additional diagnostic analyses (Tables 3, 6).

Critical weaknesses
  1. Missing important related work. Chandrasegaran et al. (2022) study how label smoothing during teacher training interacts with KD temperature, identifying systematic diffusion as a mechanism that can degrade distillation at higher temperatures. While their intervention differs from fixed-teacher logit scaling, this directly relevant study is neither discussed nor adequately positioned against the paper's findings.

  2. Insufficient experimental scale. All experiments are restricted to CIFAR and ImageNet-100 with relatively small ResNet and DeiT architectures (Tables 1–2, 5). Given the entirely empirical nature of the contribution, at least ImageNet-1K experiments are necessary to establish whether the observations generalize.

  3. Limited insights and practical utility. The central conclusion relies on comparing two selected sharpness values (α = 0.5 and α = 4). However, neither yields the highest observed gain across all evaluated states. Moreover, the paper does not explain the underlying mechanism or provide a predictive criterion for choosing target sharpness in unseen settings (Secs. 6–8). It is therefore unclear what actionable insight follows beyond empirical temperature tuning.

Questions for authors

Please see the weaknesses list.

Reviewer_nSy8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper provides an analysis on whether softening or sharpening teacher targets is preferable across different knowledge distillation settings. Using positive teacher-logit rescaling, the authors preserve the teacher's class predictions while changing target sharpness. The main experiment fixes the teacher, dataset, architecture pair, and continuation protocol, and finds opposite preferences for untrained and trained students. Additional experiments examine broader settings and offer several candidate explanations.

Critical strengths
  1. The controlled student-state comparison provides good a empirical analysis. Holding the teacher and training setup fixed makes the observed reversal more informative than a comparison across unrelated datasets or architectures.

  2. The endpoint reversal is consistent across all three student trajectories. Reporting both hierarchical intervals and trajectory-level Student-t intervals supports the authors' claims.

  3. The paper distinguishes relative sharpness preference from absolute distillation benefit and includes matched cross-entropy continuations.

Critical weaknesses
  1. I find there is limited conceptual advance over earlier temperature curricula. Annealing-KD (Jafari et al., EACL 2021) already scales teacher logits during training, starting with softened outputs and progressively restoring the original teacher outputs. CTKD (Li et al., AAAI 2023) learns a dynamic temperature to adapt distillation difficulty as the student develops. The fixed-teacher endpoint comparison provides a more focused empirical test, but the broader view that target sharpness should be considered in relation to the student's learning state is already well represented in this literature. A clearer account of what the observed reversal adds to that understanding, beyond documenting it under a particular controlled protocol would help decide if the novelty is solely in the specific experimental setup or in the overall analysis/insight.

  2. The diagnostic experiments do not directly explain the reversal that motivates the paper. The gradient-magnitude, difficulty, and target-component analyses are conducted primarily in controlled CIFAR, where the measured response remains softening-favored. These experiments help characterize that setting, but leave the change from softening-favored to sharpening-favored in the controlled ImageNet-100 experiment unexplained.

  3. The robustness of the controlled reversal to the continuation protocol is not clear to me. The main experiment uses a 20-epoch continuation with a freshly initialized optimizer and learning-rate schedule, under one fixed pretrained teacher. The other experiments use other continuation protocols while also changing the surrounding setup. Consequently, they do not isolate whether the reversal persists when the continuation horizon or optimizer treatment changes. I do not yet see how strongly the result depends on the particular short-horizon training protocol.

  4. In terms of improving performance, this introduces yet another KD hyperparameter that may need to be swept over and is non-obvious. The experiments give some results that help guide based on whether the teacher is strongly trained or not, though the lack of a principled approach may limit usefulness.

Questions for authors
  1. What insight does the controlled reversal establish beyond the student-dependent temperature view underlying Annealing-KD and CTKD? An explicit comparison of the claims and experimental evidence would help better understand the contribution.

  2. How do the gradient-matching and target-component diagnostics behave at the untrained and trained endpoints of the controlled ImageNet-100 experiment? Does either analysis identify a difference associated with the reversal?

  3. Does the endpoint reversal persist with longer continuations and when optimizer state and the original learning-rate schedule are resumed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.