- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the potential conflict between full teacher imitation and the student's acquired task discrimination in knowledge distillation. The authors theoretically characterize when full imitation becomes blocked and show that conditional knowledge among non-target classes can still be transferred while preserving classification margins. Based on these findings, they propose Task-Preserving Knowledge Distillation (TPKD), which retains the label gradient and projects the conditional distillation gradient onto a margin-preserving cone. Theoretical analysis and experiments on image and text classification tasks demonstrate the effectiveness of the proposed method.
Critical strengths
-
The paper studies an interesting and well-motivated problem. It highlights the potential conflict between full teacher imitation and the student's acquired task discrimination, providing a meaningful perspective on what knowledge should be transferred during distillation.
-
The theoretical analysis is rigorous and insightful. The paper establishes a necessary-and-sufficient condition for full-imitation blocking and derives the minimum confidence compensation required for task-preserving conditional knowledge transfer. These results provide a deeper understanding of the relationship between full imitation and conditional learning.
-
The proposed TPKD framework is simple and theoretically well-motivated. By preserving the label gradient and projecting the conditional distillation gradient onto a margin-preserving cone, the method allows the student to learn additional teacher knowledge without sacrificing the discrimination gains of the label update. The authors further provide theoretical guarantees on retained conditional learning and joint convergence, strengthening the justification for the proposed method.
Critical weaknesses
-
The experimental evaluation is relatively limited. The paper reports full-training results on only two datasets, with one teacher–student pair in each domain. Since the proposed method is motivated by differences in teacher and student discrimination, experiments with varying teacher quality and student capacity would help further validate its generality and effectiveness.
-
The advantage over simpler conditional distillation methods remains unclear. Although the paper includes a variant without projection, it lacks comparisons with a properly tuned CE-plus-conditional-KL baseline. Moreover, several existing methods, particularly DKD, perform substantially worse than CE on CIFAR-100. Further analysis of hyperparameter sensitivity and baseline configurations would help establish whether the reported improvements are consistent and reliable.
Questions for authors
Please see the Weaknesses