- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper considers the mismatch between KL-divergence minimization and classification decisions in logit-based knowledge distillation. It proposes CAMO-DKD, a plug-and-play objective that converts the teacher’s most confident class into a degenerate target distribution, aligns teacher and student probabilities using L1-based target and non-target terms, and introduces a class-aware margin penalty to encourage separation between the target and competing classes. The authors provide generalization analyses and evaluate the method with multiple logit-based distillation baselines on CIFAR-100, ImageNet, Tiny-ImageNet, and MS-COCO. The results generally show improvements across different teacher-student architecture pairs and object detection settings. Ablations are presented to examine the effects of the target, non-target, margin, and loss-design components.
Critical strengths
The proposed method achieves performance gains across multiple datasets and tasks as well as when integrated with different logit-based distillation methods. The idea is intuitive, and the method is straightforward and easy to implement.
Critical weaknesses
-
My main concern is the technical novelty. The method has some formulation-level novelty within logit-based knowledge distillation, since I am not aware of a mainstream KD method using exactly the same combination of a degenerate teacher target, probability-space L1 alignment, and a hinge-style target/non-target margin. However, most of this formulation closely parallels LDL-HR, including the objective-mismatch motivation, degenerate highest-label distribution, L1 loss, large-margin constraint, rest-label alignment, and the associated generalization arguments. Therefore, the main novelty appears to be largely the adaptation and integration of LDL-HR into the DKD setting, rather than a fundamentally new margin-based distillation principle.
-
Lemma 3.2 seems false. A counter example is a=1, b=0, c=1, d=0.5. The theoretical results relying on it may require correction or additional assumptions.
-
The paper motivates the method by noting the small margin between target and non-target logit values in fine-grained classification tasks, but no standard fine-grained classification benchmarks, such as CUB-200, Cars196, and Stanford Dogs, are used for evaluation (the authors may refer to RKD and SDD for relevant evaluations).
-
Minor weakness: The citation format is incorrect. The authors should distinguish between the uses of \citet and \citep. There are a number of inconsistencies in the tables. For example, in Table 1, the “-0.04”is not marked in blue. Table 1 shows degradations on DKD and TeKAP. Table 6 shows degradation on KD+LSKD. Hence the repeated claim of “consistent” performance gains should be toned down. More recent KD baselines should be discussed or compared with, .e.g, [1-4].
[1] Parchami-Araghi et al. Good teachers explain: Explanation-enhanced knowledge distillation. ECCV 2024.
[2] Shi et al. SelKD: Selective Knowledge Distillation via Optimal Transport Perspective. ICLR 2025.
[3] Zhang et al. VRM: Knowledge distillation via virtual relation matching. ICCV 2025.
[4] Huang et al. DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer NeurIPS 2025.
Questions for authors
-
Is Lemma 3.2 missing additional assumptions?
-
Why is the teacher-derived one-hot classifier identified with a Bayes classifier?
-
Is the target class the ground-truth label or teacher argmax? Does the margin compare student target vs student non-target, or teacher target vs student non-target?
-
Does the proposed method perform even better on fine-grained classification datasets?