- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper empirically investigates when softer versus sharper teacher targets are preferable under temperature scaling, with a particular focus on different student training states. The experiments demonstrate a controlled reversal in the preferred direction on ImageNet-100 with the DeiT family and identify distinct directional preferences across broader experimental settings.
Critical strengths
None
Critical weaknesses
- Poor readability and unclear logical flow.
The paper is difficult to follow, particularly in terms of its main contributions, interpretation of experimental results, and overall logical structure.
First, the contributions listed in the Introduction are not clearly articulated. It is difficult to understand how these individual contributions connect to form a coherent scientific insight.
Second, the presentation of experimental results is inconsistent across sections. Section 4 reports results on ImageNet-100 with DeiT models in Table 1, whereas Section 5 introduces controlled CIFAR experiments with different student states (rather than and ) and additional teacher states. Furthermore, the results for scratch-trained and pretrained DeiT models are presented as Equations (6)–(9), rather than in tables or figures. These differences in experimental configurations and presentation formats make it difficult to compare results across sections. Table 2 also uses inconsistent reporting formats across columns (e.g., "28/28 positive" versus "-0.060 [-0.38, +0.25]"), further reducing clarity.
Third, the logical transition between Sections 5 and 6 is insufficiently explained. Section 5 investigates directional preferences across various experimental settings, while Section 6 abruptly introduces three "natural reductions of the softening-favored response" in the controlled CIFAR setting. However, the paper does not adequately explain why these analyses follow from the preceding observations or why the three approaches are considered natural reductions. In particular, concepts such as "CE–KD gradient magnitude," "example correctness and difficulty," and "isolated components of the teacher target" are introduced without sufficient motivation or explanation in the main text. Relevant details appear to be relegated to Appendix C without an explicit reference, and experiment details. Moreover, the terminology used to describe these analyses is inconsistent across the Introduction, Section 6, and Figure 3.
Overall, these issues substantially hinder the reader's ability to understand the paper's central message and how the experimental findings support its conclusions.
- Insufficient justification of experimental settings.
The paper draws conclusions about the preferred direction of temperature scaling based on student models trained for 20 or 30 epochs. However, the authors do not sufficiently justify these training budgets. In particular, it is unclear whether the selected training durations are sufficient for convergence or whether the observed directional preferences remain consistent under longer training schedules. Without such evidence, it is difficult to determine whether the conclusions reflect general properties of temperature scaling or are specific to the chosen training budgets.
Questions for authors
See the weaknesses above.