acceptodds
Withdrawn from ICLR 2027

The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent large visual foundation models have demonstrated remarkable transferability across a wide range of downstream tasks. To deploy such models efficiently, feature matching has become a popular knowledge distillation approach that transfers teacher representations to smaller student models without requiring labeled data. However, we show that the conventional feature matching objective with L2-distance is inherently biased toward reconstructing dominant spectral directions of the teacher representation, while under-optimizing low-variance directions that often contain task-relevant information. To address this limitation, we propose Spectrum-Balanced Feature Matching (SpecMatch), a simple objective that balances optimization across spectral directions by adaptively emphasizing under-optimized components during training. SpecMatch is easy to implement, introduces negligible computational overhead. Extensive experiments on image recognition demonstrate that SpecMatch consistently improves downstream adaptation across diverse tasks, including image classification, anomaly detection, medical image analysis, and domain generalization. In particular, SpecMatch outperforms conventional feature matching in 40 of 42 teacher–student and training-setting combinations, while consistently improving over the original student model in all settings. We further demonstrate that the proposed objective generalizes beyond vision, improving downstream performance across six protein understanding tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
distillation, transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_z7TAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an important problem in large-scale visual encoder knowledge distillation. SpecMatch balances the contributions of principal components (PCs) and aims to prevent the loss of informative signals in non-PCs for downstream tasks. The authors demonstrate the effect of SpecMatch in several experiments; transfer to 14 classification benchmarks, reconstruction error analysis, nn-shot training.

Critical strengths
  1. The paper addresses a well-defined and potentially important problem in large-scale visual encoder distillation.
  2. The paper is clearly written and easy to follow.
  3. Section 3 provides a principled mathematical formulation and derivation of the proposed approach.
Critical weaknesses
  1. Weak support for the motivation. The left side of Figure 1 appears to provide a core motivation as an evidence for the importance of non-principal componenets. The intuition is reasonable, but the evaluated datasets (CIFAR, CUB, OrgainMNIST, and ViSA) provide limited evidence for this claim. Most of these benchmarks are small-scale and classification-oriented, and their semantic distributions may substantially overlap with those covered by large-scale visual pretraining. Evaluation on substantially different and larger-scale distributions (e.g., iNaturalist) and, more importantly, different task structures such as dense prediction (e.g., ADE20K) would provide stronger evidence.
  2. Core chain relies on intuition. The authors imply that a large spectral reconstruction error of the teacher indicates the loss of components that are informative for downstream tasks. None of the experiments provided in the manuscript supports this intuition, since reconstruction error does not reflect utility in downstream tasks. This chain remains unsupported.
  3. Limited novelty in observation. The proposition that dominant PCs in mean squared error dominate reconstruction and optimization is not surprising and somewhat trivial. The important question is why these dominant components are related to downstream tasks.
  4. Critical missing references. Even the latest baseline in Table 1 is DINO (Caron et al., 2021), which is a self-distillation for self-supervised learning, not a knowledge distillation for compression. Apart from DINO, RKD (Park et al., 2019) is the latest proper knowledge distillation baseline. However, the manuscript already cites the closest reference (VkDV_kD, Miles et al., 2024) and never compared. Other contemporary distillation methods are also absent.
  5. Narrow focus on classification. The authors' claim of general representation distillation is supported only by evaluations on several classification benchmarks. Despite the large number of benchmarks, classification tasks do not cover the ability on spatially structured downstream tasks.
  6. Narrow student-teacher pool. The manuscript mentions "general applicability" as an evaluation criterion. The current architecture pool is too limited to support the claimed general utility of SpecMatch. The authors should cover a wider range of student-teacher architectures or narrow the claim.
  7. Overclaimed conclusion. Considering the above weaknesses and the limited causal evidence provided, the current conclusion section claims too much regarding the contributions to general representation distillation.
  8. Limitations are not acknowledged. Classification-centric evaluation, ViT-only teachers, limited distribution shift, outdated or missing baselines, and weak support for the motivation are clear limitations that have not been acknowledged. Since some of these limitations are critical, the authors should explicitly acknowledge them. The manuscript currently provides no discussion of its limitations.
Questions for authors
  1. Can the authors establish a principled connection between spectral reconstructoin error and downstream utility? If so, is it possible to validate the downstream relevance of each components?
  2. How do the authors justify the chosen baselines and architecture pool in terms of downstream applicability and generality?
Reviewer_sAMQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies label-free feature distillation from a large teacher to a smaller student. It argues that standard L2 feature matching puts too much emphasis on high-variance directions and does not transfer low-variance directions well enough. SpecMatch applies a sublinear loss to reconstruction errors in the teacher PCA basis. The paper evaluates the method on several vision tasks and six protein tasks.

Critical strengths

The loss is simple and easy to implement (Eq. 4 and Algorithm 1). The paper also reports fairly consistent gains over vanilla feature matching across several downstream tasks (Tables 1 and 5). These results suggest that the idea could be useful in practice.

Critical weaknesses
  1. The novelty is not clear enough. The general problem of transferring VFM knowledge to small models is already well studied. Balancing feature statistics in distillation has also been explored before. The paper itself cites work on normalization and whitening in Section 2, including PHI-S. The potential contribution here is the adaptive weighting in Eq. 4. I would like a clearer explanation of what this adds beyond existing approaches. The whitening comparison in Table 3 and the static weighting comparison in Table 11 are useful, but they do not fully establish this distinction.

  2. The baseline set is too limited to make the experiments convincing. Table 1 compares against Logits-KD (2015), RKD (2019), DINO (2021), and vanilla feature matching. These are useful reference points, but recent work on VFM distillation is missing from the actual comparisons. For example, CustomKD (CVPR 2025) and Vemulapalli et al. (ICML 2024) are cited but not evaluated. Their settings are not identical, so the paper should explain which methods can be compared fairly and include suitable baselines with matched data, label access, and training budgets. Testing on many datasets does not resolve this gap. The results support an improvement over vanilla feature matching, but do not show clearly how SpecMatch compares with existing VFM distillation methods. This is the main reason I do not find the experimental evidence strong enough for acceptance.

Questions for authors
  1. What is the main technical difference between the adaptive weighting in Eq. 4 and earlier methods for balancing or normalizing teacher features? Which existing result best isolates the benefit of this difference?

  2. Can you compare with recent VFM distillation methods under a matched protocol? If methods such as CustomKD or the task-specific transfer approach of Vemulapalli et al. cannot be used in your setting, please explain the incompatibility and identify the closest suitable baselines. Stronger comparisons here could materially affect my assessment.

Reviewer_PtrAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that L2 feature matching (FM) transfers teacher representations unevenly. It derives that the per-direction learning signal scales with the teacher variance λk\lambda_k along the teacher's PCA directions, leaving low-variance directions under-optimized. To counter this, SpecMatch has the student predict the teacher's PCA coefficients and aggregates the per-direction mini-batch errors eke_k as ∑k(ek+ϵ)β\sum_k (e_k+\epsilon)^\beta, with β−1=3\beta^{-1}=3. Each target dataset is distilled using only its unlabeled images and evaluated by linear probe. Across three teacher-student pairs and 14 datasets, SpecMatch exceeds FM in 40 of 42 settings and the original student in all 42. ImageNet, iNaturalist2018 and varying VisA anomaly counts, six protein tasks, and whitening and static-weighting comparisons are also reported.

Critical strengths
  • The comparison against feature matching is clean. FM and SpecMatch appear to share the PCA-coefficient prediction structure, training budget, and hyperparameters, differing only in β\beta. This isolates the sub-linear aggregation, which is the main basis of my positive assessment.
  • SpecMatch beats FM in 40 of 42 settings and the original student in all 42, whereas FM sometimes falls below the student. Sign agreement in 40 of 41 decided settings makes average superiority unlikely to be chance.
Critical weaknesses
  • My main concern is the absence of variance or significance. Table 1 averages three runs without reporting std, and Tables 2, 3, 4, and 11 state no run counts. Eight of the 42 wins over FM are 0.1 to 0.4 points, so whether these per-setting wins exceed run-to-run variation cannot be judged.
  • A second critical weakness is that the partial-balance explanation conflicts with the paper's own gradient analysis. From Eq. 6, the per-direction squared gradient norm scales as (ek+ϵ)2β−2ek(e_k+\epsilon)^{2\beta-2}e_k, which is approximately ek2β−1e_k^{2\beta-1} when ek≫ϵe_k \gg \epsilon. At the default β=1/3\beta=1/3, lower-error directions receive the larger gradients.
Questions for authors
  • Please report std for Table 1's three runs, and run counts plus std for Tables 2, 3, 4, and 11. How many of the 42 differences over FM exceed std?
  • Why is the Table 1 average column wrong, and why do Table 3 left's FM row and Table 9's iWildCam value differ from Table 1?
  • As a clarification, on what data and normalization were MSE and λ\lambda computed in Figures 1 and 6? As I read them, MSE roughly equals λ\lambda up to rank about 800, so the FM student explains almost no teacher variance.
  • As an additional experiment, please measure per-direction gradient norms and explained variance during training for FM, SpecMatch, and whitening. Please also compare whitening, PHI-S, and static weighting in all Table 1 settings.
  • Were β−1=3\beta^{-1}=3, ϵ\epsilon, and the smaller MVTec and iWildCam learning rates selected on validation or test data, and how were DINO and RKD tuned? Test-based selection would lower my baseline-fairness judgment. For whitening, which ϵ\epsilon and how many PCs did you use, given eigenvalues down to 10−810^{-8} (Figure 5)?
  • Did test images enter the VisA and MVTec distillation sets? Their Table 6 train counts match the full public dataset sizes I checked, and all 1,200 VisA anomaly images are in the test split. If so, please state the transductive setting and reinterpret Table 4 left; the FM comparison still stands.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.