- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tries to tackle the question whether early teacher–student representation similarity predicts a student’s improvement from knowledge distillation over its own task only baseline. The paper is mainly empirical in nature.
Critical strengths
I found the the statistical reporting to be useful. Student configurations are the ranking units and seeds are not counted as additional architectures. The paper reports all metric phase combinations and corrects multiple tests. Further, the authors acknowledges dependent comparisons and explicitly avoids interpreting nonsignificance as evidence of no association.
Independent outcome evaluation is also valuable. Original Ped2 B0 uses disjoint probe/pilot and holdout seeds. The other pools include cross seed selection evaluations.
Critical weaknesses
My main concern is that the evidence remains too inconclusive to establish either a reliable limitation of representation similarity or a useful student selection strategy. The experimental scope and measurement choices also limit the generality of the findings. Most evidence comes from one video anomaly-detection dataset, closely related PVTv2 students, one reconstruction-based KD objective, and a ten-epoch training horizon.
There are also some other uncertainties, i.e., The number of student configurations and the precision of each student’s expected gain. Going over subsets of the same three seeds does not add independent information about those.
Also with only 8 to12 students per pool and three outcome seeds, the study can reliably detect only very strong correlations. The wide uncertainty makes it difficult to distinguish weak similarity metrics from noisy estimations of gains. I believe that more student configurations and higher-precision gain estimates are needed to come to a conclusion.
Questions for authors
See above weaknesses