acceptodds
Withdrawn from ICLR 2027

Class-Aware Margin Optimization: Resolving KL Divergence-Classification Inconsistency in Decoupled Knowledge Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Logit-based knowledge distillation (KD) for classification offers a cost-efficient alternative to feature-based and relationbased methods. However, logit-based knowledge distillation methods fail to explicitly model the discriminative margins between target and non-target categories. This limitation primarily stems from their overemphasis on globally minimizing KL divergence during distillation , which rigidly enforces distribution-level similarity while neglecting the inconsistency between KL divergence and classification. To address this limitation, we propose a Class- Aware Margin Optimization method for Decoupled Knowledge Distillation (CAMO-DKD). CAMO-DKD enforces the margin constraint by aligning the student’s decoupled predictions with the teacher’s decoupled distribution through a margin-sensitive loss. This enhances the separation between the target class and competing non-target categories, thereby preventing boundary collapse during distillation. The proposed CAMO-DKD method can enhance any logit-based KD approach in a plugand-play manner. Additionally, we provide a detailed theoretical analysis demonstrating the generalization capabilities of CAMO-DKD. Experiments on CIFAR-100, ImageNet and COCO show consistent improvements, while outperforming state-of-the-art KD methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Margin Optimization, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VQzXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper considers the mismatch between KL-divergence minimization and classification decisions in logit-based knowledge distillation. It proposes CAMO-DKD, a plug-and-play objective that converts the teacher’s most confident class into a degenerate target distribution, aligns teacher and student probabilities using L1-based target and non-target terms, and introduces a class-aware margin penalty to encourage separation between the target and competing classes. The authors provide generalization analyses and evaluate the method with multiple logit-based distillation baselines on CIFAR-100, ImageNet, Tiny-ImageNet, and MS-COCO. The results generally show improvements across different teacher-student architecture pairs and object detection settings. Ablations are presented to examine the effects of the target, non-target, margin, and loss-design components.

Critical strengths

The proposed method achieves performance gains across multiple datasets and tasks as well as when integrated with different logit-based distillation methods. The idea is intuitive, and the method is straightforward and easy to implement.

Critical weaknesses
  1. My main concern is the technical novelty. The method has some formulation-level novelty within logit-based knowledge distillation, since I am not aware of a mainstream KD method using exactly the same combination of a degenerate teacher target, probability-space L1 alignment, and a hinge-style target/non-target margin. However, most of this formulation closely parallels LDL-HR, including the objective-mismatch motivation, degenerate highest-label distribution, L1 loss, large-margin constraint, rest-label alignment, and the associated generalization arguments. Therefore, the main novelty appears to be largely the adaptation and integration of LDL-HR into the DKD setting, rather than a fundamentally new margin-based distillation principle.

  2. Lemma 3.2 seems false. A counter example is a=1, b=0, c=1, d=0.5. The theoretical results relying on it may require correction or additional assumptions.

  3. The paper motivates the method by noting the small margin between target and non-target logit values in fine-grained classification tasks, but no standard fine-grained classification benchmarks, such as CUB-200, Cars196, and Stanford Dogs, are used for evaluation (the authors may refer to RKD and SDD for relevant evaluations).

  4. Minor weakness: The citation format is incorrect. The authors should distinguish between the uses of \citet and \citep. There are a number of inconsistencies in the tables. For example, in Table 1, the “-0.04”is not marked in blue. Table 1 shows degradations on DKD and TeKAP. Table 6 shows degradation on KD+LSKD. Hence the repeated claim of “consistent” performance gains should be toned down. More recent KD baselines should be discussed or compared with, .e.g, [1-4].

[1] Parchami-Araghi et al. Good teachers explain: Explanation-enhanced knowledge distillation. ECCV 2024.

[2] Shi et al. SelKD: Selective Knowledge Distillation via Optimal Transport Perspective. ICLR 2025.

[3] Zhang et al. VRM: Knowledge distillation via virtual relation matching. ICCV 2025.

[4] Huang et al. DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer NeurIPS 2025.

Questions for authors
  1. Is Lemma 3.2 missing additional assumptions?

  2. Why is the teacher-derived one-hot classifier identified with a Bayes classifier?

  3. Is the target class the ground-truth label or teacher argmax? Does the margin compare student target vs student non-target, or teacher target vs student non-target?

  4. Does the proposed method perform even better on fine-grained classification datasets?

Reviewer_ggk8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Class-Aware Margin Optimization for Decoupled Knowledge Distillation (CAMO-DKD) to address the inconsistency between KL divergence minimization and classification decisions in logit-based knowledge distillation. The authors argue that a small KL divergence between teacher and student predictions does not necessarily guarantee consistent classification decisions, particularly when the predicted probabilities of competing classes are close. To address this issue, CAMO-DKD introduces a composite distillation objective consisting of target-class L1 matching, non-target-class L1 matching, and a margin-based regularization term. The proposed objective can be integrated into existing logit-based KD methods. Experiments on multiple image classification and object detection datasets demonstrate performance improvements across various teacher-student architectures and KD baselines.

Critical strengths
  1. The research problem is meaningful, and the proposed framework is relatively simple. This paper highlights an important distinction between distribution matching and classification decision consistency in logit-based knowledge distillation, which is worth investigating. Moreover, the proposed auxiliary objective can be integrated into existing KD methods without modifying the model architecture, offering good extensibility and ease of application.

  2. The experimental evaluation covers various knowledge distillation frameworks. The experiments are conducted on multiple datasets, different teacher-student architectures, and representative logit-based KD methods. The reported performance improvements across several settings suggest that the proposed objective may offer practical benefits.

Critical weaknesses
  1. The proposed optimization objective is not sufficiently aligned with the core motivation. The central motivation of this paper is to address the inconsistency between teacher and student classification decisions despite a small discrepancy between their predicted distributions. Compared with existing baselines, the paper mainly introduces two modifications: adopting L1 loss instead of KL divergence for probability matching and introducing an extra margin loss. However, neither loss, as formulated, directly guarantees that the student's predicted class agrees with the teacher's prediction. The paper does not sufficiently explain how these modifications address the identified classification inconsistency and lacks detailed qualitative and quantitative analyses of this mechanism.

  2. The reported performance improvements lack sufficient attribution analysis. The proposed method mainly introduces L1 loss and an additional margin loss compared with the baselines. However, although the paper includes some ablation studies, the main text does not clearly establish the individual contributions of these two modifications or which contributes more to the performance improvements. Furthermore, compared with the target-label distillation in DKD, the proposed method converts the teacher's target-label distribution into a one-hot form. The specific benefits of this modification also remain unclear.

Questions for authors
  1. Regarding the consistency between the optimization objective and the research motivation. The authors should further clarify how the two key modifications relate to the core motivation, supported by qualitative and quantitative analyses. For example, why can KL divergence-based distribution matching lead to inconsistent classification decisions between the teacher and student? How frequently does this phenomenon occur during standard training on CIFAR-100 and ImageNet? Why can L1 loss and margin loss mitigate this issue, and to what extent does each loss reduce classification inconsistency?

  2. Regarding the key ablation studies. More controlled ablation experiments are needed to disentangle the performance gains from adopting L1 loss, introducing margin loss, and modifying target-label distillation to use a one-hot distribution.

  3. Regarding the definition of the margin. Why is the margin defined as ptyt−psjp_t^{y_t}-p_s^j rather than the student-side margin psyt−psjp_s^{y_t}-p_s^j? How does the current constraint guarantee or encourage consistency between teacher and student classification decisions?

  4. Regarding the consistency between the experimental settings and the Introduction. In the Introduction, the authors emphasize the importance of the margin in fine-grained classification. However, the experiments mainly focus on general image classification and object detection tasks, including ImageNet, CIFAR-100, and MS-COCO. Would the proposed method yield more substantial improvements on fine-grained classification benchmarks, such as CUB?

  5. Other presentation and experimental reporting issues. a) What does "MMKD" refer to in Figure 2 of the main paper and Figure 1 of the supplementary material? This term does not appear to be defined in the methodology. b) Additionally, it might be more appropriate to include the ImageNet results in the main paper rather than the supplementary material. c) For the CIFAR-100 experiments, reporting the mean and standard deviation over multiple independent runs would provide stronger evidence of the stability of the reported improvements.

Reviewer_DSnaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a class-aware margin optimization method to address inconsistency in DKD. The key idea is to introduce a margin between the target and non-target classes, allowing the model to focus on learning the target class while exploiting correlations among non-target classes. However, the paper contains several potential correctness issues and inconsistencies among its equations, claims, and reported results. These concerns need to be resolved before the reported empirical improvements can be justified for acceptance.

Critical strengths

This paper provides theoretical analysis and relatively extensive experiments covering multiple baselines and experimental settings.

Critical weaknesses
  • My main concern is the lack of comparisons with recent work. Despite the extensive reference list, all cited studies, including the experimental baselines, were published in 2025 or earlier. This raises my concerns when assessing the paper’s technical novelty and empirical effectiveness. The authors should discuss recent SOTA methods and include relevant comparisons.

  • The reported results also contain internal inconsistencies. In Table 2, row #9 of the Logit-based section, both MLKD+LS_KD and MLKD+LS_KD+Ours achieve 69.70, yet the reported gain is +1.22. In Appendix Table 6, RKD’s Top-1 accuracy is reported as 91.32, exceeding its Top-5 accuracy of 90.62. The authors should correct these discrepancies and verify the underlying results. These inconsistencies raise concerns about the reliability of the experimental results.

  • A minor suggestion: the authors could use \citep{} where parenthetical citations are appropriate, rather than \cite{} or \citet{}, to improve readability.

Questions for authors
  1. Lemma 3.2 appears to be incorrect. If we set a=c=0.8a=c=0.8 and b=d=0.2b=d=0.2, then this lemma implies 0.6≤00.6\le0. The authors should correct the lemma and revisit the proof steps and subsequent theorems that depend on it.

  2. The margin equations do not enforce the stated student margin. Equations (11), (13), and (17) use ptyt−psjp_t^{y_t}-p_s^j, rather than the difference between two student probabilities. For teacher probabilities (0.9,0.06,0.04)(0.9,0.06,0.04), student probabilities (0.4,0.5,0.1)(0.4,0.5,0.1), and ρ=0.001\rho=0.001, the non-target margin penalties can be zero even though the student assigns a higher probability to a non-target class than to the target class. Although other loss terms may mitigate this issue, this penalty alone does not guarantee the claimed separation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.