- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies cross-architecture knowledge distillation and argues that teacher features from different architecture families have different degrees of anisotropy, which distorts identity-target alignment such as RSD. The authors propose WFD, which applies batch-wise ZCA whitening to frozen teacher features and adds a student-side decorrelation loss on the projector output, on top of the RSD-style cross-correlation objective. WFD outperforms RSD on all 12 teacher-student pairs on CIFAR-100, while on ImageNet-1K it is comparable to RSD.
Critical strengths
- Simple and low-cost method with consistent CIFAR-100 gains.
- Teacher whitening adds no learnable parameters, and an optimized eigendecomposition reduces its computational overhead to approximately 4.4% per step.
- WFD outperforms RSD on all 12 heterogeneous teacher–student pairs on CIFAR-100.
- Transparent analysis and reporting.
- The paper analyzes teacher anisotropy, the rank-induced floor of batch whitening, and the effect of batch size.
- The mixed ImageNet-1K results and the gap to FBT are reported honestly, and the code and checkpoints are released.
Critical weaknesses
- The motivation and mechanism of teacher whitening are not clearly established.
- Anisotropy and performance: There is no direct evidence that the different anisotropy of teachers causes inconsistent results of RSD. The largest gains appear with ViT-S, which has the lowest anisotropy (8×), and the authors note that anisotropy does not predict gains across teachers.
- Explanation of the problem: After column-wise standardization, every teacher dimension has unit variance, so the statement that high-variance dimensions dominate the diagonal loss seems imprecise. The more direct issue may be that P = I is not achievable when teacher dimensions are correlated, but its practical effect is unclear with κ = 0.01.
- Batch whitening with B < D: All CIFAR-100 experiments use B (128 or 256) smaller than D_T (384 or 768). In this regime, near-complete batch whitening can make inter-sample distances nearly uniform before the final column-wise normalization. It remains unclear how much of the teacher’s original relational structure is preserved in the final target. Also, W is re-estimated for every batch, which may make the target for the same sample less stable.
- ImageNet results: The CNN runs on ImageNet-1K use batches large enough to avoid rank deficiency (B ≥ 1024), but WFD shows no gain over RSD (−0.04 pp on average). This raises questions about the generality of the whitening-quality explanation proposed in Appendix C.
- The experimental evidence could be strengthened.
- Baseline parity: Baseline numbers are taken from RSD and OFA, while WFD uses its own training configuration, including model EMA and AMP. The RSD row in Table 3 (83.92) is the same as in Table 1, so it is unclear whether the ablation starts from the same configuration.
- Variance: Results are reported without mean and standard deviation across independent runs. Several gains over RSD are below 0.5 pp (e.g., +0.35, +0.43, +0.48), making their statistical reliability unclear.
- Hyperparameter tuning: Hyperparameters are tuned on Swin-T → ResNet-18, which is also used for the ablations, but the validation protocol and selection criteria are not specified.
- Scale and novelty: On ImageNet-1K, the average gain of WFD (+2.11) is slightly lower than RSD (+2.18) and FBT (+2.21). In addition, VkD (Miles et al., 2024) also investigates teacher feature whitening in distillation, mainly for generative tasks, and relates it to a cross-feature objective, while the student decorrelation builds on Barlow Twins and VICReg. The novelty and methodological distinction from these works are not sufficiently explained.
Questions for authors
Clarification Questions
- Baseline configuration: Were RSD and other baselines re-run under the same configuration as WFD (EMA, AMP, epochs, batch size)? Is the first row of Table 3 your own run?
- Fixed whitening: Since the teacher is frozen, W could be computed once from the full training set. How does this compare with the batch-wise W?
- Information in the whitened target: In the B < D regime, how much inter-sample structure remains in the whitened teacher features (e.g., kNN accuracy or CKA with the original features)? What happens with a control target, such as whitened features of an untrained teacher?
- Variance: Can the authors report the mean and standard deviation over multiple seeds for Table 1 and Table 3?
- Hyperparameter selection: Was a validation split used for tuning λ, β, κ, and γ on CIFAR-100?
- ImageNet CNN students: Why does WFD not improve over RSD for CNN students on ImageNet-1K, even though the batch size exceeds the feature dimension there?
Suggestions
- Please discuss the teacher whitening in VkD and clarify the difference from WFD.
- Please revise the explanation of how anisotropy affects the RSD loss under column-wise standardization.
- If possible, analyze the relation between anisotropy and the gain over RSD using more teachers or checkpoints.