- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether shuffling the teacher's non-target class probabilities in knowledge distillation can tell us whether class-wise probability supervision is necessary. The authors point out that assigning probabilities to the wrong classes and removing individual probability targets for these classes are two different operations.
Critical strengths
- The problem is clearly defined. Permutation still preserves class-wise targets, so it cannot directly replace an experiment that removes these targets. The proposed grouped control is therefore methodologically meaningful.
- The grouped intervention is reasonably designed. It preserves the total residual probability mass, avoiding the confounding effects of changing both the total mass and normalization when probabilities are directly set to zero.
- The mathematical treatment appears largely sound. I did not notice any obvious errors in the derivations, including those involving expectations.
Critical weaknesses
- The paper feels somewhat rough in its presentation. There are no visualizations, which makes the proposed method and the overall paper difficult to follow.
- The main novelty lies in the design of the control experiments. However, the paper still lacks sufficiently broad empirical findings to support its contribution as a full paper.
Questions for authors
nan
Thank you for recognizing the control design. We agree with both weaknesses, and we revised the paper around what a reader can conclude from it. Section, table, and figure numbers refer to the revised main PDF; "App." refers to the supplementary PDF.
What the paper now shows
A drop under permutation shows only that the student is sensitive to where residual probability is placed. Whether the teacher's class-wise residual targets were needed is a separate question, and it requires an omission control. A uniform control U separates the two. Eq. 1 is an algebraic identity; what the controls add is that each term becomes measurable:
O−P=(O−U)+(U−P),O−C=(O−U)+(U−C).
Across six settings (Table 3):
The consequence for existing ablations is that a permutation drop is an upper bound on this benefit, not an estimate of it. A small permutation drop also does not mean the targets are dispensable (Sec. 6). The abstract, contributions, and section order now put this result first; the Top–Bottom interaction and the mass controls follow as the conditions under which the two interventions compare differently.
W1. Visualizations and readability
We also compared how students order the residual classes on the CIFAR-100 test set (App. F, Table 11). A CE student trained without a teacher already reaches a Spearman correlation of 0.51–0.68 with the teacher's residual order. Uniform (0.37–0.47) and permuted (0.22–0.39) students fall below CE in every setting and seed, so substituted targets erase order that labels alone provide. Grouped students, which receive no target for that order, reach 0.62–0.69. Omission therefore largely preserves the order that reassignment destroys, without recovering the teacher's probability ratios.
W2. Empirical breadth
Added: CE and teacher baselines; 200-epoch training (5 seeds); 240-epoch training with ResNet-20, ResNet-32, and VGG8-BN students; three mass controls; a K/λ/τ grid; teacher-reliability filtering; official Tiny-ImageNet validation; and official CIFAR-100 test evaluation.
Results:
We therefore claim neither a mass-independent class-identity effect nor an improvement over unmodified KD. The contribution is the decomposition above and its consequence: a permutation ablation alone cannot establish which teacher probabilities a student needs.