acceptodds
Under review as a conference paper at ICLR 2027

Structured Temperature Scaling for Group Knowledge Distillation

Abstract

Knowledge distillation (KD) commonly uses temperature scaling to soften the teacher distribution and transfer dark knowledge. In this work, we revisit temperature from a structured prediction perspective. To this end, we derive an exact multi-group decomposition of vanilla KD into inter-group distillation (inter-KD) and probability-mass-weighted intra-group distillation (intra-KD). Inter-KD preserves the probability allocation across groups, while intra-KD transfers fine-grained conditional structures within each group. This decomposition reveals that temperature improves KD by adjusting the contribution of different probability regions, but conventional global temperature simultaneously alters inter-group allocation and flattens fine-grained intra-group structures.Based on this analysis, we propose Group Knowledge Distillation (GKD), which introduces structured temperature scaling by shifting temperature smoothing from the global category distribution to the group level. Specifically, GKD adaptively determines prediction groups based on prediction perplexity and applies temperature scaling at the group level, thereby rebalancing supervision across groups while preserving fine-grained intra-group structural knowledge. Without introducing additional grouping hyperparameters or complex structural designs, GKD only modifies the weighting strategy in the reconstructed KD formulation and achieves effective knowledge transfer.Experiments on CIFAR-100 and ImageNet demonstrate state-of-the-art Top-1 performance, providing a new perspective on temperature regulation in knowledge distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Distillation, Temperature Scaling, Multi-Group Decomposition, Prediction Perplexity

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_BkMyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a group-based knowledge distribution as an extension of DKD [Zhao+22]. The method is formulated by decomposing a primary KD loss into inter-/intra-KD terms by means of grouped logit representations. Through the analysis of those two terms in terms of temperature, the proposed group KD (GKD) is equipped with temperature-free KL-divergence while temperature-scaling is only applied for weighting intra-KD terms. In addition, logit grouping is adaptively determined based on prediction perplexity at each sample. In the experiments on KD tasks in image classification, the proposed GKD produces competitive performance to the other KD approaches.

Critical strengths
  • The paper is well written and easy to follow. Preliminary experimental results in Tables 1&2 facilitate understanding motivation to design the proposed temperature-scaling strategy.
  • To start with group-wise decomposition of KD loss in a similar way to DKD, the GKD is formulated in a theoretically-sound way with the adaptive grouping approach which is also well founded on the perplexity.
  • Efficacy of the proposed method is empirically evaluated on image classification/detection tasks to exhibit favorable performance.
Critical weaknesses
  • Main concern is about novelty of the approach. Group-wise decomposed KD has been proposed in [R1,R2]. They report similar performances to the GKD in their respective papers while employing a simple two-grouping (K=2K=2). As to grouping, TDKD [R2] derives an adaptive strategy through temperature stimulus. The effect of temperature in the grouped formulation is also empirically analyzed in [R1]. Compared to those related works, the novelty of the proposed method is limited; the paper lacks both analysis about the connections to those related approaches and empirical performance comparison.

  • The grouping in Alg. 1 inherits the target-class partitioning from DKD without providing its rationale, while [R1] discusses effect of (teacher's) top-logit on KD training. According to the basic motivation that roughly equalize logits to be compared as shown in Fig.1, it is unclear how and why such a heuristic works.

  • The experimental evaluation is less sufficient.

    • Sensitivity analysis regarding temperature TT is not shown. Other than α\alpha and β\beta, the temperature TT is a key hyper-parameter in the GKD loss (8).
    • The perplexity-based grouping in (6) is not so well analyzed. As it is built upon the student logits, the group cardinality neffn_{eff} is varied during the training, according to maturity of the learnt student network, implying that the number of groups is changed throughout the training. The analysis about neffn_{eff} over the training process is demanded for validating its effectiveness. Besides, the softmax probability with T=1T=1 could provide rather smaller neffn_{eff}, which may require temperature-scaling as well. The paper lacks analysis regarding the temperature on neffn_{eff}.

[R1] Bowen Zheng and Ran Cheng, Rethinking Decoupled Knowledge Distillation: A Predictive Distribution Perspective, arXiv:2512.04625, 2025.

[R2] Yiqin Wang and Ying Chen, Temperature-driven category decoupled knowledge distillation with interpretability for model compression, Advanced Engineering Informatics, vol. 69, 2026.

Questions for authors
  • Please clarify novelty of the proposed method over the prior group-based approaches [R1,R2], through qualitative and quantitative comparison.
1 comment

We sincerely thank the reviewer for the constructive feedback. We have revised the manuscript to clarify GKD's novelty, grouping mechanism, and temperature sensitivity. Our responses follow.

W1: We have reconsidered and clarified the novelty of our work. Our central contribution is structured temperature scaling. The multi-group KL decomposition enables temperature to redistribute intra-KD supervision weights while preserving fine-grained structural knowledge. From this perspective, GKD should not be viewed merely as an extension of DKD or other grouping-based approaches [R1, R2]. Our grouping is designed to enable structured temperature regulation, rather than improving knowledge transfer through partitioning alone.

The revised Table 8 provides supporting evidence. Even random grouping (including the target class) achieves competitive performance, showing that global temperature scaling is unnecessary. Applying structured temperature scaling to DKD's binary partition also improves performance, while further non-target partitioning enables finer supervision reallocation and achieves the best results.

T/SDKDBinaryRandom (K=5K=5)Random (K=10K=10)Perplexity-S
R32×4/R8×476.3276.7977.3177.2677.55
R32×4/SV277.0777.4677.3977.8678.24

W2: Thank you for highlighting this issue. We have clarified the rationale in Appendix A.5. Unlike DKD, we isolate the target class because its sharply peaked probability can drive the initial NeffN_{\mathrm{eff}} toward one. Separating it avoids an unnecessary grouping iteration and excessively small head groups, improving partition stability.

We have also added Fig. 3 to explain the effectiveness of structured temperature scaling. By moving temperature scaling from the global distribution to group-level supervision weights, GKD both preserves fine-grained intra-group structures and redistributes their supervision strengths. Fig. 3 shows lower teacher–student KL divergence than KD and DKD, indicating better matching of high-probability structural knowledge. Interestingly, Appendix A.9 shows that GKD achieves closer teacher matching despite higher prediction entropy and smaller logit gaps. This suggests that better recognition does not require strictly reproducing the teacher's global confidence differences.

W3.1: We have added temperature sensitivity experiments in Appendix Table 13, structured temperature scaling applies temperature only to the aggregated group masses used as intra-KD weights, leaving intra-group structures unchanged. As TT increases, these weights approach a uniform distribution, making performance relatively stable at high temperatures. Since aggregated group masses may differ substantially, stronger smoothing helps rebalance their supervision strengths. This explains the benefits of relatively large temperatures.

T/ST=1T=1T=2T=2T=4T=4T=10T=10T=20T=20T=100T=100
R32×4/R8×473.7675.2876.8877.6877.6477.66
R32×4/SV272.8575.9077.4978.2477.9877.63

W3.2: Since NeffN_{\mathrm{eff}} is computed from student predictions, group composition adapts during training, while the total group count remains relatively stable with a slight increase. A small initial NeffN_{\mathrm{eff}} at T=1T=1 is intentional to separate comparable high-probability predictions. After extracting each group, we renormalize the remaining probabilities and recompute NeffN_{\mathrm{eff}}, which increases rapidly across recursive steps due to the long-tailed distribution. The resulting group counts are approximately 8 for CIFAR-100, 14 for ImageNet-1K, and over 20 for GPT with 50K tokens. Thus, a small initial NeffN_{\mathrm{eff}} does not imply a small total group count.

Q1: As clarified above, our method should not be regarded simply as an extension of DKD. We have added heterogeneous ViT experiments (Table 6) and GPT2-1.5B/GPT2-120M distillation (Table 7), achieving the best average ROUGE-L of 21.07 under reverse KL. Since next-token prediction admits multiple plausible outputs, strict target/non-target separation can be restrictive. GKD instead adaptively reallocates supervision across prediction ranks while preserving fine-grained relations among high-probability predictions. These results support the distinct contribution and broad applicability of structured temperature scaling across vision and language tasks.

Reviewer_EGzyOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits temperature scaling in knowledge distillation. It splits the vanilla KD loss into an inter-group KL term and an intra-group KL term weighted by the teacher's group probability. DKD is the special case with two groups. Based on this split, the paper argues that temperature mainly helps by giving low-probability groups more weight, but a global temperature also flattens the relations inside each group and changes the allocation across groups. The proposed method groups classes adaptively by the perplexity of the student prediction. It applies temperature only to the group-level weights and keeps the intra-group distributions unchanged.

Critical strengths
  1. The analysis of temperature is clear and supported by experiments. The decomposition shows that much of the benefit of temperature comes from giving low-probability groups more weight. This helps explain how KD works.

  2. The method is simple. It only changes the weights in the reconstructed KD loss, adds no extra modules, and sets the number of groups automatically from the prediction.

  3. The experiments are well organized. Results are averaged over three runs, several grouping strategies are compared.

Critical weaknesses
  1. The novelty of the decomposition is overclaimed. Eq. (3) is the chain rule of KL divergence, which is a standard result, and DKD already gives the two-group case. Calling it "the first equivalent KL decomposition under general multi-group partitioning" is not accurate. The new part is applying temperature only to the group weights.

  2. It is unclear how much of the gain comes from grouping. The best results come from GKD*, which adds the prediction augmentation of MLKD. Without augmentation, GKD is below MLKD or LDRLD in several settings.

  3. The comparison with vanilla KD may be unfair. GKD uses large weights (α\alpha = 8, β\beta = 32), while vanilla KD uses its default setting. Part of the gain may come from stronger distillation weights rather than grouping.

  4. The tables may contain errors. In Tables 1 and 2, the second column is labeled VGG13/VGG8, but the listed accuracies 72.34/69.06 belong to ResNet56/ResNet20. The third column is labeled R32 ×\times 4/SV1 in both tables, but vanilla KD is 74.07 in Table 1 and 74.45 in Table 2.

Questions for authors
  1. Without prediction augmentation, how much does GKD improve over MLKD and LDRLD? Are these differences significant over the three runs?

  2. If KD and DKD use tuned loss weights, how large is the remaining gain of GKD?

  3. Please explain the column labels and the vanilla KD numbers in Tables 1 and 2.

3 comments

We sincerely thank the reviewer for the careful and constructive feedback. We have revised the manuscript to address the concerns and strengthen our experimental evidence. Our responses follow.

W1: We acknowledge that our original novelty claim regarding the KL decomposition was overstated and have revised it accordingly. We would like to clarify that our core contribution is structured temperature scaling, where the decomposition enables temperature to redistribute intra-group supervision weights while preserving fine-grained structural knowledge.

W2: We agree that further evidence is needed to demonstrate GKD's effectiveness. We have added extensive ViT-based and LLM distillation experiments (Tables 6–7). GKD achieves substantial improvements across heterogeneous architectures and the best average ROUGE-L score of 21.07 under reverse KL. Notably, next-token prediction admits multiple plausible outputs, making strict target/non-target separation potentially restrictive. GKD adaptively groups predictions by rank and reallocates supervision while preserving fine-grained relations among high-probability predictions. These results demonstrate the broad applicability of GKD across both vision and language tasks.

W3: Conventional KD and DKD apply global logit temperature scaling and use an additional T2T^2 gradient compensation (e.g., a factor of 16 at T=4T=4). GKD instead applies temperature to teacher group masses, without this T2T^2 factor. We therefore use larger coefficients to maintain comparable distillation strength. Importantly, stronger weights are not always better, as excessive distillation may conflict with CE supervision (Appendix A.8).

All baseline results are taken from the original papers, where hyperparameters are also tuned for different architectures. For example, DKD's reported 76.45% and 76.70% for R32×4/SV1 and WRN-40-2/SV1 correspond to β=8\beta=8 and β=6\beta=6 in its supplementary Table A.1. Thus, these baselines already reflect architecture-specific hyperparameter selection.

W4: Thank you for carefully identifying these errors. The second column was indeed intended for VGG13/VGG8. We have corrected the labels and corresponding results in both tables, along with the other inconsistencies.

Q1: We report mean ± std from repeated CIFAR-100 runs below. The main tables use the first run, and the reporting description has been corrected.

Homogeneous architectures (Table 4)

Teacher → StudentMain TableMean ± Std
R56 → R2072.0472.06 ± 0.18
R110 → R3274.2574.13 ± 0.10
R32×4 → R8×477.5577.57 ± 0.11
WRN-40-2 → WRN-16-276.1676.11 ± 0.06
WRN-40-2 → WRN-40-175.2975.17 ± 0.10
VGG13 → VGG875.1575.14 ± 0.05

Heterogeneous architectures (Table 3)

Teacher → StudentMain TableMean ± Std
R32×4 → SV177.2977.47 ± 0.17
WRN-40-2 → SV177.3177.19 ± 0.11
VGG13 → MV270.3970.35 ± 0.04
R50 → MV271.1571.08 ± 0.07
R32×4 → SV278.2478.18 ± 0.09

Q2: As noted in W3, the reported baselines already use architecture-specific hyperparameter selection. We have also added ViT and LLM experiments to demonstrate GKD's effectiveness across tasks. Fig. 3 further shows that structured supervision reallocation improves teacher–student prediction matching. When teacher and student have similar recognition abilities, less additional knowledge may be transferable, explaining the smaller gains in some settings (Table 4). Table 6 shows a similar trend across heterogeneous architectures, with particularly substantial gains for DeiT-T and ResMLP-S12 students.

StudentST−SGKDGKD−S
R1874.0115.52 ± 2.3881.43 ± 1.23+7.42 ± 1.24
MV273.6815.85 ± 2.3981.36 ± 1.05+7.68 ± 1.05
DeiT-T68.0019.85 ± 0.7979.80 ± 1.03+11.80 ± 1.03
Swin-P72.6315.22 ± 0.7981.79 ± 0.23+9.16 ± 0.23
ResMLP-S1266.5622.28 ± 0.6080.99 ± 0.23+14.43 ± 0.23

Q3: Tables 1 and 2 follow the standard KD configuration (T=4T=4, CE weight = 0.1, KD weight = 0.9). To ensure fair comparisons, we use 10 fixed sequential groups constructed according to teacher predictions, avoiding grouping variations caused by different student predictions. All other training settings remain unchanged, with only the designated temperature or intra-KD weighting component modified in each variant. Table 1 compares different intra-KD weighting strategies, while Table 2 examines intra-group temperature smoothing and group-level temperature regulation. Further details are provided in Appendix A.6.

Thanks for the authors' rebuttal. I agree with Reviewer Jbrm that a well-tuned DKD baseline is necessary. How much does GKD improve over DKD with the same training settings and tuning budget, and with prediction augmentation turned off for both? More fundamentally, random grouping nearly matches perplexity-based grouping on one of the two pairs in Table 8, so the added value of adaptive grouping remains unclear. I'm keeping my original rating of 2.

Thank you again for your feedback. I am particularly eager to clarify that none of our newly added experiments, including ViT and LLM distillation, uses prediction augmentation. Throughout the paper, prediction augmentation is used only when explicitly marked with an asterisk (*). I am not sure whether this may have led to your misunderstanding of our experimental settings.

Regarding random grouping, Table 8 shows that different students may favor different grouping configurations. The key significance of perplexity-based grouping is that it eliminates the tunable hyperparameter KK by adaptively determining the groups. This advantage may become even more evident in larger output spaces, such as ImageNet-1K or large-scale LLMs.

In any case, thank you for your follow-up. I remain convinced that the role of temperature smoothing and regulation in knowledge distillation has been underestimated, and I will continue to improve and refine this research. :P

Reviewer_JbrmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Group Knowledge Distillation (GKD), which applies temperature scaling at the group level instead of the entire class distribution. The method separates knowledge distillation into inter-group and intra-group components and uses prediction perplexity to form groups. It aims to preserve fine-grained knowledge within groups while adjusting the importance of each group. Experiments on CIFAR-100 and ImageNet show competitive performance.

Critical strengths
  1. The paper provides a clear motivation for preserving intra-group probability relationships during knowledge distillation.

  2. The proposed KL decomposition provides a useful perspective on the role of temperature scaling in knowledge distillation.

  3. The method achieves competitive results across different teacher-student architectures on CIFAR-100 and ImageNet.

  4. The ablation studies provide useful evidence for the effects of temperature scaling and grouping strategies.

Critical weaknesses
  1. Improving the transfer of dark knowledge has been an important research topic in knowledge distillation for many years. Although GKD introduces a new grouping strategy, its main idea is closely related to DKD, particularly the further division of non-target classes within NCKD. It is unclear whether similar improvements can be achieved by carefully tuning the temperature and loss weights (alpha and beta) of DKD without additional grouping. A comparison with a well-tuned DKD baseline is needed to show the actual benefit of grouping.

  2. The experimental results mainly focus on classification accuracy, but they do not sufficiently explain why GKD performs better than DKD. The authors should analyze how GKD changes the transferred dark knowledge within NCKD, such as the teacher-student distribution differences, entropy, or non-target class relationships. This would help show whether the improvement comes from the proposed grouping mechanism rather than hyperparameter optimization.

  3. In Table 4, GKD shows only small improvements over DKD for several homogeneous teacher-student pairs, although larger improvements are observed in some cases. Since DKD already achieves strong performance through NCKD, the remaining room for improvement may be limited. This raises the question of whether further grouping within NCKD provides meaningful benefits beyond a well-tuned DKD baseline.

I would like to clarify that my concern is not about DKD itself. Rather, I consider DKD to be the starting point of GKD. If comparable performance can be achieved by tuning the existing hyperparameters without the main components proposed in GKD, the significance of GKD's contributions may be limited. This is why I believe such a comparison is important.

Questions for authors
  1. Can the authors compare GKD with DKD after carefully tuning its temperature, alpha, and beta under the same training settings and a comparable hyperparameter search budget? How much additional improvement can be achieved by grouping the non-target classes within NCKD?

  2. Can the authors provide quantitative evidence that GKD transfers dark knowledge more effectively than DKD's NCKD? For example, how do the teacher-student distribution differences, entropy, or non-target class relationships change after grouping? How are these changes related to the performance improvement?

  3. In Table 4, why does GKD show only small improvements over DKD for several homogeneous teacher-student pairs? Could this indicate that DKD's NCKD already captures most of the useful non-target knowledge? Can the authors show whether further grouping still provides benefits after careful DKD tuning?

1 comment

We sincerely thank the reviewer for the detailed and constructive feedback. We have carefully reconsidered the concerns and revised the manuscript to better clarify the benefits of structured temperature scaling and its role in dark knowledge transfer. Our responses follow.

W1 & Q1: First, we clarify that the DKD baselines are taken from officially reported results with tuned hyperparameters. For example, DKD reports 76.45% and 76.70% for R32×4/SV1 and WRN-40-2/SV1, corresponding to β=8\beta=8 and β=6\beta=6 in its supplementary Table A.1. GKD uses larger coefficients than DKD because DKD retains the conventional T2T^2 compensation, whereas GKD does not.

More importantly, GKD is not merely a finer partition of DKD's NCKD. Our grouping aims to enable structured temperature scaling, which provides two main benefits: (1) redistributing supervision weights across prediction regions and (2) preserving richer fine-grained intra-group knowledge (Tables 1–2). To address your concern, we have extended Table 8 to compare structured temperature scaling under different grouping strategies, including the binary target/non-target partition used by DKD:

T/SDKDTarget/Non-targetRandom (K=5)Random (K=10)Perplexity S-guided
R32×4/R8×476.3276.7977.3177.2677.55
R32×4/SV277.0777.4677.3977.8678.24

Even random grouping (including the target class) achieves competitive performance, showing that global temperature scaling over all categories is unnecessary. Applying structured temperature scaling to DKD also consistently improves performance. Further partitioning the non-target classes, as in GKD, enables finer supervision reallocation and achieves the best results. Another important piece of evidence comes from ImageNet (Table 5). Both KD and DKD use a global temperature of T=1T=1, meaning that the original fine-grained class relationships are already preserved. Nevertheless, GKD achieves further improvements by redistributing supervision weights, demonstrating benefits beyond simply preserving local structures.

These benefits are particularly evident in LLM distillation with approximately 50K tokens (Table 7), where GKD achieves the best average ROUGE-L of 21.07 under reverse KL. Since next-token prediction admits multiple plausible outputs, strict target/non-target separation can be restrictive. GKD instead adaptively groups predictions by rank and reallocates supervision while preserving relations among high-probability predictions.

W2 & Q2: We agree that our previous version lacked sufficient analysis of structured temperature scaling. Its effectiveness comes from smoothing the supervision weights across groups, allowing capacity-limited students to more flexibly match the teacher's high-probability intra-group structures, while lower-probability groups retain smaller supervision weights.

As shown in the new Fig. 3, GKD consistently achieves the lowest teacher–student KL divergence compared with KD and DKD, particularly in the probability-mass-weighted intra-KD component. This indicates closer prediction matching, especially in high-probability regions. Appendix A.9 further reveals a counterintuitive result: GKD achieves closer teacher matching despite producing higher-entropy predictions and smaller Top-1/Top-2 logit gaps. This suggests that better distillation does not require reproducing the teacher's overall prediction confidence, especially when student capacity is limited.

W3 & Q3: As discussed above, GKD improves teacher–student prediction matching through structured supervision reallocation. When the teacher and student have similar recognition abilities, the additional transferable knowledge becomes limited, explaining the smaller gains in some settings in Table 4. This is related to the teacher–student performance gap rather than architectural homogeneity.

We have further added 12 heterogeneous ViT-based distillation settings (Table 6), consistently demonstrating the effectiveness of GKD. Similar trends can be observed: students with larger teacher–student gaps, particularly DeiT-T and ResMLP-S12, achieve substantial improvements.

StudentST−SGKDGKD−S
R1874.0115.52 ± 2.3881.43 ± 1.23+7.42 ± 1.24
MV273.6815.85 ± 2.3981.36 ± 1.05+7.68 ± 1.05
DeiT-T68.0019.85 ± 0.7979.80 ± 1.03+11.80 ± 1.03
Swin-P72.6315.22 ± 0.7981.79 ± 0.23+9.16 ± 0.23
ResMLP-S1266.5622.28 ± 0.6080.99 ± 0.23+14.43 ± 0.23

Thank you again for your insightful comments and suggestions. We welcome further discussion of any remaining questions or concerns.

Reviewer_poJoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper rewrites the KD loss over a partition of the classes into two parts: one term that matches the probability mass of each group, and other terms that match the distribution within each group, weighted by how much probability mass the teacher assigns to that group. Based on this, the authors argue that temperature mainly helps by giving more weight to low-probability groups, but at the same time, it also flattens the distribution within each group. GKD addresses this by computing the loss without temperature inside the groups and applying temperature only to the group weights. Groups are formed for each sample based on the student prediction using perplexity. The results are reported on CIFAR-100, ImageNet, and COCO detection.

Critical strengths

I think the question of what temperature actually does in KD is interesting. I found the analysis in Sec. 3.2 to be the most interesting part of the paper, especially the result showing that removing temperature inside the groups helps. The method itself is simple, and the paper is easy to follow. I also appreciate that the authors report GKD and GKD* separately.

Critical weaknesses
  1. Eq. 3 is the chain rule of KL divergence, and DKD is already its special case. I do not think this can be called the "first equivalent KL decomposition." Also, the best results (GKD*) use the prediction augmentation from MLKD. This is not new either, as augmentation is already known to help KD. The perplexity grouping is also very close to random grouping (77.26 vs. 77.55 in Table 7).
  2. For several pairs, GKD is only slightly better, with gains below 0.2% (R56/R20, WRN-40-2/WRN-16-2, and ImageNet R50/MV1). The authors average results over three runs, but do not show the std. Also, looking at Table 3, GKD is below MLKD on 3 out of 5 pairs. On ImageNet, it is also below MLKD and LDRLD. I find this quite different from the paper's statement that GKD "consistently achieves state-of-the-art performance."
  3. The difference between GKD and GKD* is quite large on R50/MV2 and R32×4/SV1. So the effect of the augmentation needs to be separated from the effect of the proposed loss. Another issue is hyperparameter tuning. The authors tune separately for each teacher-student pair, including on the CIFAR-100 test set. GKD also uses different settings from the baselines, which keep their default settings and seem to use results from earlier papers.
  4. Logit Standardization (CVPR 2024) is in the related work, but the authors do not compare GKD against it. I also noticed that KD-Zero, SimKD, and SFKD (2026) report higher numbers than GKD on some of the same teacher-student pairs. The comparison is mostly with older KD methods.
  5. The experiments use CNN teacher-student pairs and follow the 2022 DKD. For ImageNet, the authors only test two pairs. I was also expecting to see some ViT experiments. Since GKD works on logits, it should be possible to test it with ViT models too. For example, a ViT/Swin teacher and CNN student on CIFAR-100, or a DeiT setting on ImageNet.
Questions for authors
  1. Can you report the mean and standard dev. for Tables 3, 4, and 7?
  2. Could you add results with ViT models?
1 comment

We thank the reviewer for the detailed and constructive feedback, particularly for recognizing the value of our temperature analysis. We have carefully reconsidered the concerns and substantially revised the manuscript. Our point-by-point responses are provided below.

W1: We acknowledge that our original novelty claim regarding the KL decomposition was overstated and have revised it accordingly. We would like to clarify that our core contribution is structured temperature scaling, where the decomposition enables temperature to redistribute intra-group supervision weights while preserving fine-grained structural knowledge. Even random grouping achieves competitive performance, demonstrating the effectiveness of structured temperature scaling beyond a specific grouping strategy. Fig. 3 further shows that this simple modification reduces group-weight disparities and improves teacher–student prediction matching. We hope this work inspires further exploration of temperature scaling for more effective knowledge transfer in large-output-space settings.

W2: We acknowledge that our original performance claims were too strong and have revised them accordingly. To further validate GKD, we have added extensive ViT-based and LLM distillation experiments (Tables 6 and 7). By simply reallocating group-level supervision, GKD achieves substantial improvements across heterogeneous architectures. Notably, this modification also transfers effectively to LLM distillation, achieving the best average ROUGE-L score under reverse KL.

W3: We would like to clarify that GKD does not employ the conventional T2T^2 compensation, and larger coefficients are used to maintain an appropriate distillation strength. Appendix A.8 also provides guidance for different teacher–student settings. All baseline results are taken from their original papers, where hyperparameters are also adjusted for different architectures. For example, the DKD results of 76.45% and 76.70% for R32×4/SV1 and WRN-40-2/SV1 correspond exactly to β=8\beta=8 and β=6\beta=6 in its supplementary Table A.1. Thus, the reported results already reflect architecture-specific hyperparameter selection.

W4: Logit Standardization uses Z-score normalization to emphasize relative logit structures, whereas GKD preserves intra-group knowledge while reweighting its supervision. SimKD and SFKD explore feature-based supervision and selection, respectively. In contrast, GKD requires no feature alignment and readily extends to heterogeneous architectures and LLMs.

W5: We have added ViT and LLM experiments (Tables 6–7) to demonstrate generality. As discussed in Sec. 4.1 and Appendix A.8, smaller teacher–student gaps leave less transferable knowledge, explaining the limited gains in some homogeneous settings (Table 4).

Q1: We report mean ± std over three independent CIFAR-100 runs below. The main tables use the first run, and the reporting description has been corrected.

Homogeneous architectures (Table 4)

Teacher → StudentMain TableMean ± Std
R56 → R2072.0472.06 ± 0.18
R110 → R3274.2574.13 ± 0.10
R32×4 → R8×477.5577.57 ± 0.11
WRN-40-2 → WRN-16-276.1676.11 ± 0.06
WRN-40-2 → WRN-40-175.2975.17 ± 0.10
VGG13 → VGG875.1575.14 ± 0.05

Heterogeneous architectures (Table 3)

Teacher → StudentMain TableMean ± Std
R32×4 → SV177.2977.47 ± 0.17
WRN-40-2 → SV177.3177.19 ± 0.11
VGG13 → MV270.3970.35 ± 0.04
R50 → MV271.1571.08 ± 0.07
R32×4 → SV278.2478.18 ± 0.09

Grouping strategies (Table 8)

Grouping StrategyR32×4/R8×4R32×4/SV2
Random (K=5K=5)77.55 ± 0.1977.39 ± 0.01
Random (K=10K=10)77.35 ± 0.3177.64 ± 0.17
Sequential-S (K=5K=5)77.17 ± 0.1977.82 ± 0.11
Sequential-T (K=5K=5)76.97 ± 0.1676.62 ± 0.36
Sequential-S (K=10K=10)77.51 ± 0.1377.76 ± 0.23
Sequential-T (K=10K=10)76.66 ± 0.1675.90 ± 0.33
Perplexity-S77.57 ± 0.1178.18 ± 0.09
Perplexity-T76.96 ± 0.3576.68 ± 0.06

Q2: We have added 12 heterogeneous architecture settings, including ViT and Swin (Table 6). GKD achieves the highest average accuracy gain of 10.10 percentage points, compared with 8.72 for LDRLD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.