- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a group-based knowledge distribution as an extension of DKD [Zhao+22]. The method is formulated by decomposing a primary KD loss into inter-/intra-KD terms by means of grouped logit representations. Through the analysis of those two terms in terms of temperature, the proposed group KD (GKD) is equipped with temperature-free KL-divergence while temperature-scaling is only applied for weighting intra-KD terms. In addition, logit grouping is adaptively determined based on prediction perplexity at each sample. In the experiments on KD tasks in image classification, the proposed GKD produces competitive performance to the other KD approaches.
Critical strengths
- The paper is well written and easy to follow. Preliminary experimental results in Tables 1&2 facilitate understanding motivation to design the proposed temperature-scaling strategy.
- To start with group-wise decomposition of KD loss in a similar way to DKD, the GKD is formulated in a theoretically-sound way with the adaptive grouping approach which is also well founded on the perplexity.
- Efficacy of the proposed method is empirically evaluated on image classification/detection tasks to exhibit favorable performance.
Critical weaknesses
-
Main concern is about novelty of the approach. Group-wise decomposed KD has been proposed in [R1,R2]. They report similar performances to the GKD in their respective papers while employing a simple two-grouping (). As to grouping, TDKD [R2] derives an adaptive strategy through temperature stimulus. The effect of temperature in the grouped formulation is also empirically analyzed in [R1]. Compared to those related works, the novelty of the proposed method is limited; the paper lacks both analysis about the connections to those related approaches and empirical performance comparison.
-
The grouping in Alg. 1 inherits the target-class partitioning from DKD without providing its rationale, while [R1] discusses effect of (teacher's) top-logit on KD training. According to the basic motivation that roughly equalize logits to be compared as shown in Fig.1, it is unclear how and why such a heuristic works.
-
The experimental evaluation is less sufficient.
- Sensitivity analysis regarding temperature is not shown. Other than and , the temperature is a key hyper-parameter in the GKD loss (8).
- The perplexity-based grouping in (6) is not so well analyzed. As it is built upon the student logits, the group cardinality is varied during the training, according to maturity of the learnt student network, implying that the number of groups is changed throughout the training. The analysis about over the training process is demanded for validating its effectiveness. Besides, the softmax probability with could provide rather smaller , which may require temperature-scaling as well. The paper lacks analysis regarding the temperature on .
[R1] Bowen Zheng and Ran Cheng, Rethinking Decoupled Knowledge Distillation: A Predictive Distribution Perspective, arXiv:2512.04625, 2025.
[R2] Yiqin Wang and Ying Chen, Temperature-driven category decoupled knowledge distillation with interpretability for model compression, Advanced Engineering Informatics, vol. 69, 2026.
Questions for authors
- Please clarify novelty of the proposed method over the prior group-based approaches [R1,R2], through qualitative and quantitative comparison.
We sincerely thank the reviewer for the constructive feedback. We have revised the manuscript to clarify GKD's novelty, grouping mechanism, and temperature sensitivity. Our responses follow.
W1: We have reconsidered and clarified the novelty of our work. Our central contribution is structured temperature scaling. The multi-group KL decomposition enables temperature to redistribute intra-KD supervision weights while preserving fine-grained structural knowledge. From this perspective, GKD should not be viewed merely as an extension of DKD or other grouping-based approaches [R1, R2]. Our grouping is designed to enable structured temperature regulation, rather than improving knowledge transfer through partitioning alone.
The revised Table 8 provides supporting evidence. Even random grouping (including the target class) achieves competitive performance, showing that global temperature scaling is unnecessary. Applying structured temperature scaling to DKD's binary partition also improves performance, while further non-target partitioning enables finer supervision reallocation and achieves the best results.
W2: Thank you for highlighting this issue. We have clarified the rationale in Appendix A.5. Unlike DKD, we isolate the target class because its sharply peaked probability can drive the initial Neff toward one. Separating it avoids an unnecessary grouping iteration and excessively small head groups, improving partition stability.
We have also added Fig. 3 to explain the effectiveness of structured temperature scaling. By moving temperature scaling from the global distribution to group-level supervision weights, GKD both preserves fine-grained intra-group structures and redistributes their supervision strengths. Fig. 3 shows lower teacher–student KL divergence than KD and DKD, indicating better matching of high-probability structural knowledge. Interestingly, Appendix A.9 shows that GKD achieves closer teacher matching despite higher prediction entropy and smaller logit gaps. This suggests that better recognition does not require strictly reproducing the teacher's global confidence differences.
W3.1: We have added temperature sensitivity experiments in Appendix Table 13, structured temperature scaling applies temperature only to the aggregated group masses used as intra-KD weights, leaving intra-group structures unchanged. As T increases, these weights approach a uniform distribution, making performance relatively stable at high temperatures. Since aggregated group masses may differ substantially, stronger smoothing helps rebalance their supervision strengths. This explains the benefits of relatively large temperatures.
W3.2: Since Neff is computed from student predictions, group composition adapts during training, while the total group count remains relatively stable with a slight increase. A small initial Neff at T=1 is intentional to separate comparable high-probability predictions. After extracting each group, we renormalize the remaining probabilities and recompute Neff, which increases rapidly across recursive steps due to the long-tailed distribution. The resulting group counts are approximately 8 for CIFAR-100, 14 for ImageNet-1K, and over 20 for GPT with 50K tokens. Thus, a small initial Neff does not imply a small total group count.
Q1: As clarified above, our method should not be regarded simply as an extension of DKD. We have added heterogeneous ViT experiments (Table 6) and GPT2-1.5B/GPT2-120M distillation (Table 7), achieving the best average ROUGE-L of 21.07 under reverse KL. Since next-token prediction admits multiple plausible outputs, strict target/non-target separation can be restrictive. GKD instead adaptively reallocates supervision across prediction ranks while preserving fine-grained relations among high-probability predictions. These results support the distinct contribution and broad applicability of structured temperature scaling across vision and language tasks.