acceptodds
Under review as a conference paper at ICLR 2027

WFD: Teacher Feature Whitening and Asymmetric Student Decorrelation for Cross-Architecture Knowledge Distillation

Abstract

Cross-architecture knowledge distillation transfers knowledge from large teacher models to structurally different student models, but existing methods do not address the distributional anisotropy of teacher features across heterogeneous architectures. We show that teacher features from different architecture families exhibit substantially different distributional characteristics, and that this heterogeneity distorts the alignment signal received by the student, leading to inconsistent performance across teacher-student pairs. We propose WFD (Whitened Feature Distillation), which addresses this problem through two complementary components: (1) ZCA whitening applied to frozen teacher features, which flattens the teacher's eigenvalue spectrum toward an isotropic alignment target without modifying the teacher or introducing additional parameters; and (2) asymmetric student decorrelation, which suppresses representational redundancy in the student projector to improve the diversity of learned features. Evaluated on CIFAR-100 across a wide range of heterogeneous teacher-student pairs spanning Transformer, CNN, and MLP architectures, WFD yields consistent improvements over prior methods. On ImageNet-1K, WFD remains competitive across the evaluated teacher-student pairs, with the most consistent gains observed for MLP students. Code is available at https://anonymous.4open.science/r/WFD-7D80.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
knowledge distillation, cross-architecture distillation, feature whitening, redundancy reduction, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ax3eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-architecture knowledge distillation and argues that teacher features from different architecture families have different degrees of anisotropy, which distorts identity-target alignment such as RSD. The authors propose WFD, which applies batch-wise ZCA whitening to frozen teacher features and adds a student-side decorrelation loss on the projector output, on top of the RSD-style cross-correlation objective. WFD outperforms RSD on all 12 teacher-student pairs on CIFAR-100, while on ImageNet-1K it is comparable to RSD.

Critical strengths
  1. Simple and low-cost method with consistent CIFAR-100 gains.
  • Teacher whitening adds no learnable parameters, and an optimized eigendecomposition reduces its computational overhead to approximately 4.4% per step.
  • WFD outperforms RSD on all 12 heterogeneous teacher–student pairs on CIFAR-100.
  1. Transparent analysis and reporting.
  • The paper analyzes teacher anisotropy, the rank-induced floor of batch whitening, and the effect of batch size.
  • The mixed ImageNet-1K results and the gap to FBT are reported honestly, and the code and checkpoints are released.
Critical weaknesses
  1. The motivation and mechanism of teacher whitening are not clearly established.
  • Anisotropy and performance: There is no direct evidence that the different anisotropy of teachers causes inconsistent results of RSD. The largest gains appear with ViT-S, which has the lowest anisotropy (8×), and the authors note that anisotropy does not predict gains across teachers.
  • Explanation of the problem: After column-wise standardization, every teacher dimension has unit variance, so the statement that high-variance dimensions dominate the diagonal loss seems imprecise. The more direct issue may be that P = I is not achievable when teacher dimensions are correlated, but its practical effect is unclear with κ = 0.01.
  • Batch whitening with B < D: All CIFAR-100 experiments use B (128 or 256) smaller than D_T (384 or 768). In this regime, near-complete batch whitening can make inter-sample distances nearly uniform before the final column-wise normalization. It remains unclear how much of the teacher’s original relational structure is preserved in the final target. Also, W is re-estimated for every batch, which may make the target for the same sample less stable.
  • ImageNet results: The CNN runs on ImageNet-1K use batches large enough to avoid rank deficiency (B ≥ 1024), but WFD shows no gain over RSD (−0.04 pp on average). This raises questions about the generality of the whitening-quality explanation proposed in Appendix C.
  1. The experimental evidence could be strengthened.
  • Baseline parity: Baseline numbers are taken from RSD and OFA, while WFD uses its own training configuration, including model EMA and AMP. The RSD row in Table 3 (83.92) is the same as in Table 1, so it is unclear whether the ablation starts from the same configuration.
  • Variance: Results are reported without mean and standard deviation across independent runs. Several gains over RSD are below 0.5 pp (e.g., +0.35, +0.43, +0.48), making their statistical reliability unclear.
  • Hyperparameter tuning: Hyperparameters are tuned on Swin-T → ResNet-18, which is also used for the ablations, but the validation protocol and selection criteria are not specified.
  • Scale and novelty: On ImageNet-1K, the average gain of WFD (+2.11) is slightly lower than RSD (+2.18) and FBT (+2.21). In addition, VkD (Miles et al., 2024) also investigates teacher feature whitening in distillation, mainly for generative tasks, and relates it to a cross-feature objective, while the student decorrelation builds on Barlow Twins and VICReg. The novelty and methodological distinction from these works are not sufficiently explained.
Questions for authors

Clarification Questions

  1. Baseline configuration: Were RSD and other baselines re-run under the same configuration as WFD (EMA, AMP, epochs, batch size)? Is the first row of Table 3 your own run?
  2. Fixed whitening: Since the teacher is frozen, W could be computed once from the full training set. How does this compare with the batch-wise W?
  3. Information in the whitened target: In the B < D regime, how much inter-sample structure remains in the whitened teacher features (e.g., kNN accuracy or CKA with the original features)? What happens with a control target, such as whitened features of an untrained teacher?
  4. Variance: Can the authors report the mean and standard deviation over multiple seeds for Table 1 and Table 3?
  5. Hyperparameter selection: Was a validation split used for tuning λ, β, κ, and γ on CIFAR-100?
  6. ImageNet CNN students: Why does WFD not improve over RSD for CNN students on ImageNet-1K, even though the batch size exceeds the feature dimension there?

Suggestions

  1. Please discuss the teacher whitening in VkD and clarify the difference from WFD.
  2. Please revise the explanation of how anisotropy affects the RSD loss under column-wise standardization.
  3. If possible, analyze the relation between anisotropy and the gain over RSD using more teachers or checkpoints.
Reviewer_2kVeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies cross-architecture knowledge distillation on the image classification task. It observes that the features from heterogeneous architectures exhibit substantially different eigenspectra, and argues that this anisotropy makes the identity target used by RSD poorly matched to the teacher’s actual feature geometry. It proposes ZCA whitening of teacher embeddings and a self-decorrelation loss for projected student embeddings to mitigate the identified problem. The resulting method WFD outperforms RSD on CIFIAR-100 and remains competitive on ImageNet-1k, at an increased training cost in the reported SVD implementation.

Critical strengths
  1. This paper identifies a relevant limitation of the existing RSD method. The proposed remedy is well motivated by a preliminary eigenspectrum analysis. The method is clearly justified and easy to implement.

  2. Experimental results on CIFAR-100 are positive and the component ablations clearly support the effectiveness of each proposed component.

Critical weaknesses
  1. By introducing ZCA whitening and a self-decorrelation loss, the proposed method appears as a targeted incremental fix to the redundancy suppression formulation in RSD. Feature whitening has also been explored in the knowledge distillation literature, as acknowledged in Related Work. These factors limit the technical significance of the contributions.

  2. The empirical results on the large-scale ImageNet-1k dataset are not better than RSD on average, and neither is the computational cost lower. Therefore, the current results do not establish WFD as a consistently better or more efficient replacement for RSD.

  3. While the Related Work section provides a thorough and up-to-date discussion of the cross-architecture KD literature, the generic KD methods introduced are outdated. Post-2022 methods should be included, e.g., [1-4].

[1] Parchami-Araghi et al. Good Teachers Explain: Explanation-Enhanced Knowledge Distillation. ECCV, 2024.

[2] Shi et al. SelKD: Selective Knowledge Distillation via Optimal Transport Perspective. ICLR, 2025.

[3] Zhang et al. VRM: Knowledge Distillation via Virtual Relation Matching. ICCV, 2025.

[4] Huang et al. DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer. NeurIPS, 2025.

Questions for authors
  1. Does the observed anisotropy genuinely stem from architectural heterogeneity? The central motivation attributes differences in teacher anisotropy to architectural heterogeneity. However, Figure 2 uses only one checkpoint per architecture family, and ViT-S additionally differs in feature dimensionality. Can the authors provide results across multiple independently trained teachers within the same architecture family, or dimension-matched controls, to separate architecture effects from those of the checkpoint, training-recipe, and feature-dimension?

  2. Does the reduction in teacher anisotropy quantitatively predict the improvement over RSD? If anisotropy supports the primary mechanism of WFD, should it be expected to correlate with the relative gain?

  3. Does the proposed method also improve homogeneous KD pairs? Such results would help clarify whether whitening specifically addresses cross-architecture heterogeneity or instead acts as a more general representation regularizer.

Reviewer_iGnCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WFD for cross-architecture knowledge distillation. It applies ZCA whitening to teacher features to reduce anisotropy and introduces student decorrelation to reduce feature redundancy. Experiments on CIFAR-100 and ImageNet-1K evaluate the method across different teacher–student architectures.

Critical strengths
  1. The paper offers an interesting perspective on cross-architecture distillation by examining how teacher feature anisotropy affects feature alignment. The motivation is clear, and the proposed method is simple and easy to follow.
  2. The results on CIFAR-100 are consistent across different teacher–student pairs. The analysis of whitening quality, batch size, and computational overhead also provides useful insights into the method.
Critical weaknesses
  1. The methodological advance over RSD seems somewhat incremental. Both methods rely on cross-correlation alignment and redundancy reduction, while WFD mainly introduces teacher whitening and student decorrelation. Although these modifications are reasonable, it remains unclear whether they offer clear advantages over simpler covariance normalization or regularization strategies.
  2. The results on ImageNet-1K are less convincing. WFD performs slightly worse than RSD on average and does not consistently improve across different teacher–student pairs. This contrasts with the consistent gains on CIFAR-100 and raises questions about how well the method generalizes to larger-scale settings.
  3. Although the paper demonstrates that whitening reduces teacher feature anisotropy, the connection between this reduction and student performance is not fully established. The batch-size analysis provides some evidence, but it also changes the learning rate. Moreover, the component ablation is limited to a single teacher–student pair, making it difficult to assess the general effectiveness of both components.
Questions for authors
  1. How does WFD compare with simpler feature normalization or whitening strategies under the same training settings? Such comparisons would help justify the proposed design.
  2. Why are the improvements much less consistent on ImageNet-1K? Could the authors provide component ablations on a few representative ImageNet-1K pairs, especially those where WFD underperforms RSD?
  3. Could the authors provide a more controlled analysis of how the degree of whitening affects student performance, while keeping batch size and other training settings fixed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.