acceptodds
Under review as a conference paper at ICLR 2027

Does Representation Similarity Rank Distillability? A Bounded Audit of Knowledge Distillation

Abstract

Limited training budgets require prioritizing students for knowledge distillation (KD). We evaluate early representation similarity through global gain ranking, selected-subset benefit, and gain-order repeatability using seed-paired improvements over task-only training. Five metrics at initialization and after task-only warm-up cover four 12-student Ped2 pools and a separate eight-student CIFAR-100 pool, with teacher and recipe fixed within each pool. Ped2 Spearman estimates range from -0.336 to +0.434 with wide intervals; CIFAR-100 estimates range from -0.333 to +0.238 despite positive gains in all 24 paired runs. A complete post-hoc evaluation on original Ped2 B0 assesses student selection across metrics, measurement phases, and subset sizes using three independent holdout seeds. Mean gains relative to uniform-random expectation range from -0.037 to +0.306 AUC percentage points, with 78/110 combinations above zero. In that pool, disjoint pilot and holdout blocks of three seeds each agree on 28/66 student-pair orders. Sample-size and sensitivity analyses characterize the resolution of the ranking measurements. Supporting Ped2 and ImageNet studies find identical selections for noise-aware and matched mean rules. The results support assessing early KD signals through selection benefit and gain-order repeatability alongside global ranking.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
knowledge distillation, representation similarity, student selection, paired evaluation, ranking stability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WqNmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tries to tackle the question whether early teacher–student representation similarity predicts a student’s improvement from knowledge distillation over its own task only baseline. The paper is mainly empirical in nature.

Critical strengths

I found the the statistical reporting to be useful. Student configurations are the ranking units and seeds are not counted as additional architectures. The paper reports all metric phase combinations and corrects multiple tests. Further, the authors acknowledges dependent comparisons and explicitly avoids interpreting nonsignificance as evidence of no association.

Independent outcome evaluation is also valuable. Original Ped2 B0 uses disjoint probe/pilot and holdout seeds. The other pools include cross seed selection evaluations.

Critical weaknesses

My main concern is that the evidence remains too inconclusive to establish either a reliable limitation of representation similarity or a useful student selection strategy. The experimental scope and measurement choices also limit the generality of the findings. Most evidence comes from one video anomaly-detection dataset, closely related PVTv2 students, one reconstruction-based KD objective, and a ten-epoch training horizon.

There are also some other uncertainties, i.e., The number of student configurations and the precision of each student’s expected gain. Going over subsets of the same three seeds does not add independent information about those.

Also with only 8 to12 students per pool and three outcome seeds, the study can reliably detect only very strong correlations. The wide uncertainty makes it difficult to distinguish weak similarity metrics from noisy estimations of gains. I believe that more student configurations and higher-precision gain estimates are needed to come to a conclusion.

Questions for authors

See above weaknesses

Reviewer_cZ2UOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a methodology for student selection when provided a teacher model for knowledge distillation. The selection is based on representation alignment at initialisation and after warm up. The results are primarily shown on the Ped2, CIFAR100 datasets with supporting results on ImageNet. The results suggest that assessing the KD via representation alignment can helpful in ascertaining KD benefits for student models.

Critical strengths

None

Critical weaknesses
  1. There is limited data to support the generality of the findings. For an exploration on knowledge distillation I would expect all hyperparameters to be explored in depth such as teacher weighting for values such as 0.1, 0.5 and 0.9 and a range of temperatures such as 1, 2 and 4. From what I can tell only a teacher weighting of 0.9 and temperature of 4 is explored. The lack of hyperparameter exploration and severely limits the generality of the findings as it unclear how changing these parameters would affect the results, undermining the utility of the work. This is especially important when the current representation alignment results give mixed correlation directions ("positive" and "negative") and the metric directions are inconsistent across datasets and models.

  2. The paper attempts to provide a mechanism of student selection. However, the distillation scaling laws [1] work provides a stronger mathematical and empirical account for the deciding when to distil or not within a given compute budget and provided compute-optimal distillation recipes. This paper falls far short of achieving or providing such insight. Instead, it presents somewhat unclear results, beyond what can be somewhat summarised as check how the student is performing during training. Therefore, it is not obvious what insight this paper is meant to give the reader or provide to the community beyond what is covered in existing literature.

[1] Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russell Webb 2025. Distillation Scaling Laws. In Forty-second International Conference on Machine Learning.

Questions for authors

Could you address the weaknesses above and more concretely provide:

  1. Provide results for a hyperparameter search for all models and datasets explored:
    • Teacher Weighting (0.1,0.5,0.9)
    • Teacher Temperature (1,2,4)
  2. Provide clarification and situate work with regard to scaling laws of distillation [1].
  3. Provide an explanation behind the incoherent representation alignment correlation.
  4. Provide full results at ImageNet Scale.

[1] Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russell Webb 2025. Distillation Scaling Laws. In Forty-second International Conference on Machine Learning.

Reviewer_Qe1DOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper empirically investigates whether early teacher–student representation similarity can predict and rank the gain obtained from knowledge distillation (KD). It evaluates five similarity metrics at initialization and after task-only warm-up across four Ped2 student pools and one CIFAR-100 pool, using seed-paired KD gains as the target. The study further examines subset-selection utility, gain-order repeatability across seeds, and a bounded pilot-based decision rule. Overall, the paper finds weak and highly uncertain global similarity–gain rankings, while some similarity-based subset selections still outperform random selection in specific settings.

Critical strengths
  1. The problem formulation is practically meaningful. The paper studies whether inexpensive early signals can prioritize students for costly KD evaluation, rather than merely analyzing representation similarity itself.
  2. The paper evaluates five similarity metrics, multiple measurement phases and student pools, and additionally studies seed sensitivity, probe sensitivity, subset selection, and gain-order repeatability.
Critical weaknesses
  1. The candidate pools are too small to draw strong conclusions about similarity–gain ranking. With only 12 students per Ped2 pool and 8 on CIFAR-100, the paper’s model-based calibration estimates that roughly 0.75–0.89 population Spearman correlation is needed for 80% power even before multiple-testing correction.
  2. The empirical scope remains relatively narrow. Most evidence comes from PVTv2 students on Ped2, with only one eight-student CIFAR-100 ViT pool, limiting conclusions across architectures, tasks, and KD recipes.
  3. The subset-selection evidence is largely post-hoc and descriptive. Results such as “78/110 combinations above random” aggregate many dependent metrics, phases, and subset sizes, making the practical robustness of this advantage difficult to assess.
  4. The evidence for the proposed noise-aware decision rule is limited. In the reported Ped2 and ImageNet studies, the noise-aware and matched mean rules ultimately select the same students, so the empirical benefit of explicitly modeling noise is not demonstrated.
Questions for authors

refer to the weakness list

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.