acceptodds
Under review as a conference paper at ICLR 2027

Dissecting Soft-Target Supervision in Knowledge Distillation

Abstract

Knowledge distillation trains a student to match a teacher's predictive distribution. The probabilities assigned to non-target classes provide supervision beyond the ground-truth label. Permuting those probabilities changes their class assignments but still supplies an individual target for each class, so it does not directly test the effect of omitting those targets. This study compares unmodified and permuted targets with grouped and residual-uniform supervision. Grouped supervision keeps the retained-class targets and supervises the residual classes only through their total probability. Residual-uniform supervision keeps the same targets and assigns the residual mass equally across the remaining classes. All four conditions preserve the same retained probabilities and residual mass for a given training example and retained set. For each ground-truth class, we define two fixed sets of K non-target classes, denoted Top and Bottom, using the largest and smallest mean teacher probabilities, respectively. These means are computed over training images with that ground-truth label, rather than separately for each image. On CIFAR-100 and Tiny-ImageNet, the grouped-minus-permuted effect differs between Top and Bottom by +1.49 and +0.85 macro-F1 percentage points. Grouped supervision underperforms permutation for Bottom in both settings. For Top, the grouped-minus-permuted interval is above zero on CIFAR-100 but includes zero on Tiny-ImageNet. Residual-uniform targets improve over Bottom-grouped supervision by +1.06 and +1.12 points. Three additional CIFAR training seeds show the same interaction and negative Bottom C−P effect. These comparisons evaluate changes in class assignment and omission of conditional residual supervision. The confidence intervals condition on the fitted models and mappings. The analysis reuses development sets from the preceding permutation analyses and does not match probability mass between Top and Bottom.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
knowledge distillation, soft targets, dark knowledge, non-target supervision, teacher-student learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nmEdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether shuffling the teacher's non-target class probabilities in knowledge distillation can tell us whether class-wise probability supervision is necessary. The authors point out that assigning probabilities to the wrong classes and removing individual probability targets for these classes are two different operations.

Critical strengths
  • The problem is clearly defined. Permutation still preserves class-wise targets, so it cannot directly replace an experiment that removes these targets. The proposed grouped control is therefore methodologically meaningful.
  • The grouped intervention is reasonably designed. It preserves the total residual probability mass, avoiding the confounding effects of changing both the total mass and normalization when probabilities are directly set to zero.
  • The mathematical treatment appears largely sound. I did not notice any obvious errors in the derivations, including those involving expectations.
Critical weaknesses
  • The paper feels somewhat rough in its presentation. There are no visualizations, which makes the proposed method and the overall paper difficult to follow.
  • The main novelty lies in the design of the control experiments. However, the paper still lacks sufficiently broad empirical findings to support its contribution as a full paper.
Questions for authors

nan

1 comment

Thank you for recognizing the control design. We agree with both weaknesses, and we revised the paper around what a reader can conclude from it. Section, table, and figure numbers refer to the revised main PDF; "App." refers to the supplementary PDF.

What the paper now shows

A drop under permutation shows only that the student is sensitive to where residual probability is placed. Whether the teacher's class-wise residual targets were needed is a separate question, and it requires an omission control. A uniform control UU separates the two. Eq. 1 is an algebraic identity; what the controls add is that each term becomes measurable:

O−P=(O−U)+(U−P),O−C=(O−U)+(U−C).O-P = (O-U) + (U-P), \qquad O-C = (O-U) + (U-C).

Across six settings (Table 3):

  • The teacher's ratios outperform a uniform target of the same mass: O−UO-U = +1.6 to +3.5 pp, positive for all 20 paired seeds.
  • Permutation overstates that benefit by the cost of misassignment: U−PU-P = +0.3 to +0.9 pp, or 9–35% of the permutation drop.
  • Omission adds a separate cost: U−CU-C = +0.9 to +1.7 pp.
  • U−CU-C is also positive for all 20 paired seeds (Table 7).

The consequence for existing ablations is that a permutation drop is an upper bound on this benefit, not an estimate of it. A small permutation drop also does not mean the targets are dispensable (Sec. 6). The abstract, contributions, and section order now put this result first; the Top–Bottom interaction and the mass controls follow as the conditions under which the two interventions compare differently.

W1. Visualizations and readability

  • Figure 1 (p. 4), new. Panel (a) shows the four conditions on one target. Panel (b) shows the measured Top and Bottom sets for the label "dolphin": Top covers aquatic mammals and fish, and Bottom covers fruit, flowers, and vehicles.
  • Figure 3. All settings in one plot.
  • Figure 4, a new result. Grouped and permuted students make different kinds of errors. Their difference in the share of same-superclass errors is +3.4 to +4.5 pp in all six CIFAR settings and positive for all 20 paired seeds.
  • Text. We added a contribution list (Sec. 1) and a guide to what each contrast identifies (Sec. 3).

We also compared how students order the residual classes on the CIFAR-100 test set (App. F, Table 11). A CE student trained without a teacher already reaches a Spearman correlation of 0.51–0.68 with the teacher's residual order. Uniform (0.37–0.47) and permuted (0.22–0.39) students fall below CE in every setting and seed, so substituted targets erase order that labels alone provide. Grouped students, which receive no target for that order, reach 0.62–0.69. Omission therefore largely preserves the order that reassignment destroys, without recovering the teacher's probability ratios.

W2. Empirical breadth

Added: CE and teacher baselines; 200-epoch training (5 seeds); 240-epoch training with ResNet-20, ResNet-32, and VGG8-BN students; three mass controls; a K/λ/τK/\lambda/\tau grid; teacher-reliability filtering; official Tiny-ImageNet validation; and official CIFAR-100 test evaluation.

Results:

  • The interaction recurs at 200 epochs (+1.43 pp [+0.93, +1.93]), with VGG8-BN (+0.27 pp [+0.10, +0.44]), and on official validation (+1.63 pp).
  • It does not survive approximate mass matching, and the negative Bottom C−PC-P effect does not survive any mass control.
  • All results, including null ones, are in Table 8.

We therefore claim neither a mass-independent class-identity effect nor an improvement over unmodified KD. The contribution is the decomposition above and its consequence: a permutation ablation alone cannot establish which teacher probabilities a student needs.

Reviewer_7nMZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a question about distillation for classification convolutional models: what should we do with the probabilities of the classes which are not the target one? These probabilities could stay unmodified, as we usually do, but we could also modify them in different ways. They computed the average values of these probabilities for the same target class. They used these averages to define two sets, Top and Bottom, based on their average rank and then applied three possible types of modifications: permuting the probabilities, distributing the mass uniformly and grouping them (forcing them to keep the total mass for them). The idea is good, and the writing and experiments are well done, but they do not achieve a real improvement with respect to the common way of applying distillation without any modification to the probabilities.

Critical strengths
  1. The question is useful and interesting. Studying what the probabilities for other classes contribute to distillation is a valid contribution, even without improving over the original targets.
  2. The experiments are solid within the tested settings. The conditions use the same teacher and student architectures, matched training settings, and paired seeds. The grouped and uniform controls are useful. Results on two datasets and additional CIFAR seeds support the main findings.
  3. The paper is clearly written and reports its limitations carefully. The authors distinguish null results from equivalence and explain what their confidence intervals measure.
Critical weaknesses
  1. The relation to earlier grouped distillation should be clearer, particularly Grouped Knowledge Distillation for Deep Face Recognition by Weisong Zhao et al. from AAAI 2023. The comparison may still contribute new evidence, but this connection should be discussed.

  2. The hyperparameters are specified, but their sensitivity is not studied. The conclusions are therefore limited to the selected retained set sizes, temperature, loss weight, and training duration.

  3. Top and Bottom modify different amounts of probability mass. This limits what the comparison tells us about class identity independently of probability mass. The authors acknowledge this limitation.

  4. The development images were reused after earlier analyses. Confirmation on fresh evaluation data would strengthen the results, although this limitation is disclosed.

Questions for authors
  1. How does this study relate to earlier grouped distillation, including Weisong Zhao et al. from AAAI 2023 and Gan et al. from ICASSP 2025?

  2. Could you add a comparison where Top and Bottom modify approximately the same probability mass, and report the number of modified classes?

  3. As a possible extension, would the results change when selecting only examples predicted correctly by the teacher, or examples where the teacher gives the correct class more probability than the student?

  4. How sensitive are the main findings to the retained set size and distillation weight?

  5. Could the main findings be confirmed on evaluation data that were not used in the earlier analyses?

1 comment

Thank you for the five questions. For each, we give what we added, the result, and the limitation that remains. Section, table, and figure numbers refer to the revised main PDF; "App." refers to the supplementary PDF.

Q1. Relation to grouped KD (Sec. 2; App. A, Eq. 9)

Grouped omission is not new. Eq. 9 shows that our grouped loss drops the same within-group term as Zhao et al. (AAAI 2023), and Sec. 2 also covers Gan et al. (ICASSP 2025).

The difference is in selection and weighting. Those methods choose groups per example from student probabilities: Zhao et al. use a cumulative threshold, and Gan et al. use the top-kk classes with adaptive logit softening. We fix sets per label from training-set teacher means and weight the residual term by the teacher's residual mass.

Our contribution is the matched O/P/C/UO/P/C/U comparison and the decomposition O−C=(O−U)+(U−C)O-C = (O-U) + (U-C), which separates what grouped methods remove.

Q2. Matched mass and number of modified classes (Sec. 5.5, Tables 8–9)

Added: three mass controls.

Result (CIFAR, 5 seeds):

  • Approximate mass matching: residual mass 0.358 vs. 0.359, with 74.0 vs. 25.0 modified classes. II = +0.09 pp [−1.76, +1.94].
  • Normalized coefficient: II = +1.43 pp [+0.13, +2.72].
  • Exact per-example rescaling: II = +0.58 pp [+0.05, +1.10].
  • On Tiny, the interaction is +0.45 and +0.04 pp, both with CIs including 0.

Limitation: Bottom C−PC-P turns positive under every control, so we claim no mass-independent class effect.

Q3. Teacher-reliable examples (Sec. 5.6, Table 8)

Added: KD restricted to (i) teacher-correct images and (ii) images where the teacher's true-class probability exceeds that of a fixed CE student. Each rule is compared with a class-count-matched random mask.

Result: The rules keep 99.6% and 92.6% of images. All reliability-minus-random effects have CIs including 0.

Limitation: Rule (i) removes too few images to be informative, and rule (ii) uses a single reference student. This extension does not identify a reliability mechanism.

Q4. Sensitivity to KK, λ\lambda, τ\tau and training length (Fig. 3, Table 8)

Added: a one-factor grid (KK = 10, 25, 50; λ\lambda = 0.5, 0.9; τ\tau = 2, 4, 8), a 5-seed 200-epoch schedule, and 240-epoch training with three students.

Result:

  • In the grid, II is positive in 4 of 5 non-baseline cells and largest at KK = 10 (+2.74 pp).
  • At 200 epochs, II = +1.43 pp [+0.93, +1.93].
  • At 240 epochs, II is positive for all three students, but only the VGG8-BN interval excludes 0.

Limitation: Bottom C−PC-P changes sign at λ\lambda = 0.5 and τ\tau = 8. We state this rather than claim robustness.

Q5. Fresh evaluation data (Sec. 4, Sec. 5.6, Table 8, App. D and F)

Tiny-ImageNet: We evaluated the frozen submitted students on the official validation set (9,993 images, after removing 7 exact training duplicates).

  • Bottom C−PC-P = −1.51 pp [−2.50, −0.52].
  • II = +1.63 pp. With the models fixed, its evaluation-bootstrap CI is [+0.88, +2.40]; its family-adjusted seed CI includes 0.

CIFAR-100: We evaluated all 140 saved 200- and 240-epoch checkpoints on the official test set, without retraining.

  • At 200 epochs, II = +1.47 pp [+0.50, +2.45] and Bottom C−PC-P = −0.92 pp, both close to the development results.
  • O−UO-U is positive in all four settings.

Limitation: This analysis was post hoc. Corrupted versions of the CIFAR-100 test images were examined in earlier analyses, so these images are new to the models but not entirely unexposed.

Overall

The interaction survives coefficient normalization and exact mass rescaling but not approximate mass matching, and the negative Bottom C−PC-P effect does not survive any mass control. We therefore narrowed the claim. A drop under permutation shows sensitivity to where residual probability is placed, not that the teacher's class-wise residual targets were needed. With the uniform control, O−P=(O−U)+(U−P)O-P = (O-U) + (U-P) and O−C=(O−U)+(U−C)O-C = (O-U) + (U-C) (Eq. 1). Across six settings, the benefit of the teacher's ratios over a uniform target of the same mass, O−UO-U, is +1.6 to +3.5 pp, and permutation overstates it by U−PU-P (+0.3 to +0.9 pp) (Table 3). All results, including null ones, are in Table 8.

Reviewer_M4CROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a comprehensive comparison of four supervision conditions: unmodified and permuted targets, and grouped and residual-uniform supervision. The unmodified condition achieves the best performance, while all other conditions degrade the effect of knowledge distillation. Under the two retained-set definitions, Top and Bottom, the conditions show different levels of degradation. With bottom retained sets, grouped supervision performs worse than permutation, whereas with Top retained sets it performs better (significantly only on CIFAR-100).

Critical strengths
  1. The four conditions are clearly defined. All four conditions share the same retained-class probabilities and residual mass.
  2. A Sattolo cycle is used to avoid fixed points in the permutation, so that every residual class receives another class's probability.
  3. The experimental tables are solid and provide guidance for designing and interpreting soft-target ablations.
Critical weaknesses
  1. The residual mass differs substantially between Top and Bottom (e.g., 0.36 vs. 0.64 on CIFAR-100), which makes the conclusion less solid. Normalizing the residual mass, for example by reweighting the residual term by 1/QR1/Q_R or by using retained sets with matched residual mass, could be considered to make the comparison more solid.
  2. For the experiments, the performance of a student trained only with ground-truth labels is not reported, so it is unclear whether the modified conditions still improve over training without distillation. The accuracy of the teacher model could also be added to make the comparison in the tables more complete.
  3. For the performance, whether the training is sufficient could be further explained. The unmodified-target student reaches only about 66% macro-F1 on CIFAR-100 after 60 epochs, which is lower than commonly reported results for ResNet-20. Whether the models are well trained with tuned hyperparameters to reach performance levels comparable to those reported in the literature can also be further explained.
  4. This paper presents analytical results, but the unmodified condition (O) still achieves the best performance, so the conclusions offer limited guidance for developing future methods.
Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.