- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses an important problem in large-scale visual encoder knowledge distillation. SpecMatch balances the contributions of principal components (PCs) and aims to prevent the loss of informative signals in non-PCs for downstream tasks. The authors demonstrate the effect of SpecMatch in several experiments; transfer to 14 classification benchmarks, reconstruction error analysis, -shot training.
Critical strengths
- The paper addresses a well-defined and potentially important problem in large-scale visual encoder distillation.
- The paper is clearly written and easy to follow.
- Section 3 provides a principled mathematical formulation and derivation of the proposed approach.
Critical weaknesses
- Weak support for the motivation. The left side of Figure 1 appears to provide a core motivation as an evidence for the importance of non-principal componenets. The intuition is reasonable, but the evaluated datasets (CIFAR, CUB, OrgainMNIST, and ViSA) provide limited evidence for this claim. Most of these benchmarks are small-scale and classification-oriented, and their semantic distributions may substantially overlap with those covered by large-scale visual pretraining. Evaluation on substantially different and larger-scale distributions (e.g., iNaturalist) and, more importantly, different task structures such as dense prediction (e.g., ADE20K) would provide stronger evidence.
- Core chain relies on intuition. The authors imply that a large spectral reconstruction error of the teacher indicates the loss of components that are informative for downstream tasks. None of the experiments provided in the manuscript supports this intuition, since reconstruction error does not reflect utility in downstream tasks. This chain remains unsupported.
- Limited novelty in observation. The proposition that dominant PCs in mean squared error dominate reconstruction and optimization is not surprising and somewhat trivial. The important question is why these dominant components are related to downstream tasks.
- Critical missing references. Even the latest baseline in Table 1 is DINO (Caron et al., 2021), which is a self-distillation for self-supervised learning, not a knowledge distillation for compression. Apart from DINO, RKD (Park et al., 2019) is the latest proper knowledge distillation baseline. However, the manuscript already cites the closest reference (, Miles et al., 2024) and never compared. Other contemporary distillation methods are also absent.
- Narrow focus on classification. The authors' claim of general representation distillation is supported only by evaluations on several classification benchmarks. Despite the large number of benchmarks, classification tasks do not cover the ability on spatially structured downstream tasks.
- Narrow student-teacher pool. The manuscript mentions "general applicability" as an evaluation criterion. The current architecture pool is too limited to support the claimed general utility of SpecMatch. The authors should cover a wider range of student-teacher architectures or narrow the claim.
- Overclaimed conclusion. Considering the above weaknesses and the limited causal evidence provided, the current conclusion section claims too much regarding the contributions to general representation distillation.
- Limitations are not acknowledged. Classification-centric evaluation, ViT-only teachers, limited distribution shift, outdated or missing baselines, and weak support for the motivation are clear limitations that have not been acknowledged. Since some of these limitations are critical, the authors should explicitly acknowledge them. The manuscript currently provides no discussion of its limitations.
Questions for authors
- Can the authors establish a principled connection between spectral reconstructoin error and downstream utility? If so, is it possible to validate the downstream relevance of each components?
- How do the authors justify the chosen baselines and architecture pool in terms of downstream applicability and generality?