- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Authors of this work introduce a new post-hoc supervised, training-free, representation-space normalization that aims to preserve intra-class discriminative information. They showed that the anisotropy of representations from SSL foundation models depends on data size and training objective. They showed that their method provides the largest gains in classification tasks in comparison to previous normalization methods and that the gains are weaker for newer, stronger models.
Critical strengths
Analysis beyond the proposed normalization method. The analysis is not limited to evaluating the performance of within-class whitening. The authors also investigate how pretraining objectives and dataset size affect representation anisotropy.
Promising empirical results with a simple within-class whitening approach. The proposed approach achieves substantial improvements in linear classification accuracy, particularly for weaker foundation models. The evaluation covers a diverse set of image classification benchmarks, including medical imaging, demonstrating the potential practical usefulness of the method.
Critical weaknesses
Insuffient control for optimization effects. Table 2 shows surprisingly large improvements from Centered alone (+10.8 for BEiT and +7.1 for DINO v1). Since centering is an affine transformation that can be exactly absorbed into the bias of a linear classifier, it does not change the set of representable functions. These substantial gains suggest that the reported improvements may be strongly influenced by the optimization protocol. It would be great to see albation study on the hyperparameters for training.
Insufficient analysis of the sources of performance gains, lack of statistical analysis, and limited evaluation on recent foundation models and against total-covariance whitening. The improvements over unsupervised total-covariance whitening are relatively small and there is no reported standard deviation of the results (0.7–1.8 p.p. on CIFAR-100) in Table 2. The paper lacks ablation studies investigating the extent to which the gains depend on the use of ground-truth labels (e.g., what happens if the labels are shuffled before estimating the transformation?). Furthermore, the evaluation does not include recent foundation models such as DINOv3, raising the question of whether the gains will vanish entirely. It would also be valuable to evaluate models trained with different objectives, such as I-JEPA. Given the small performance differences between within-class and total-covariance whitening in Table 2, it would be useful to include the total-covariance whitening results in Table 3 to determine whether within-class whitening consistently outperforms total-covariance whitening or whether the relative performance varies across settings.
Questions for authors
- Could the authors explain why the centering alone achieves such large gains in Table 2? Could the authors also provide an ablation study on the training hyperparameters for Table 2?
- Could the authors provide the total-covariance whitening results for the benchmarks in Table 3? This would enable a more detailed comparison beyond Table 2.
- Could the authors extend the evaluation to DINOv3? Given the decreasing gains observed for stronger models, it would be particularly interesting to see what the gains are for DINOv3.
- Could the authors provide more detailed ablation study to investigate the source of the performance gains? In particular, how much of the improvement comes from the use of ground-truth labels during normalization (for example, what would happen if we shuffled labels)?
- Given that the gains decrease for stronger models, how do the authors see the practical relevance of the proposed method for future generations of foundation models?
6 comments
Insufficient analysis of the sources of performance gains, lack of statistical analysis, and limited evaluation on recent foundation models and against total-covariance whitening.
We do not believe there is a need for any statistical analysis. For post-hoc normalisation we estimate first and second order moments. The sampling distributions for the mean and covariance matrices are well known. The idea of preconditioning is also well known. The argument is made convincingly and rigorous in Le Cun (1998) and other papers mentioned in the related work section. For the linear probe, the input data normalisation case applies. Please note that Adam optimiser estimates the preconditioner as part of its iterations. This is most appropriate for non-linear optimisation such as training a foundation model. For linear probe the preconditioner can be estimated effectively using the whole dataset. This is what speeds up the convergence and enables optimisation with fixed parameters and is the crux of our contribution. We will include this explanation in our revised manuscript.
To illustrate the benefit of normalisation under the assumption of fixed computational budget we show the impact of various types of second order statistics preconditioning in Table 1 on a subset of classes. (For all four scenarios the data is centered, which eliminates of the conditioning variables.) The results show that even Adam is unable to cope very well for some datasets such as Aircraft. As suggested, we include the results that show the effect of normalisation using total covariance matrix.
Table1: Decomposing the Preconditioner: Which Normalisation Speeds Up the Probe?
Results on the four most anisotropic benchmarks (DINOv2 ViT-B/14, CLS features). denotes the condition number of the training feature covariance matrix (the ratio of its largest to smallest eigenvalue), with lower values indicating better conditioning. The SGD and Adam columns report budget-limited test top-1 accuracy (%) after 50 epochs at a shared learning rate.
All variants are rescaled to equal mean norm, so only covariance shape differs. Per-dimension standardisation (diagonal) barely helps; full whitening is required. Label-free total-covariance whitening captures most of the gain, with within-class whitening providing additional improvements under Adam on several datasets.
| Dataset | Normalisation | Condition Number () | SGD (50 epochs) | Adam (50 epochs) |
|---|---|---|---|---|
| CUB-200-2011 | None (raw) | 81.6 | 88.5 | |
| Std (diagonal) | 81.8 | 88.7 | ||
| Total cov | 1.2 | 88.3 | 89.1 | |
| Within-cls (sup.) | 74 | 86.4 | 89.8 | |
| DTD | None (raw) | 74.5 | 79.6 | |
| Std (diagonal) | 74.7 | 79.4 | ||
| Total cov (full) | 1.2 | 81.0 | 79.7 | |
| Within-cls (sup.) | 14 | 79.8 | 81.9 | |
| FGVC Aircraft | None (raw) | 33.2 | 50.0 | |
| Std (diagonal) | 35.9 | 52.9 | ||
| Total cov | 4.0 | 71.3 | 76.7 | |
| Within-cls (sup.) | 73.4 | 80.5 | ||
| SUN397 | None (raw) | 67.7 | 77.1 | |
| Std (diagonal) | 67.9 | 77.1 | ||
| Total cov | 1.6 | 72.1 | 76.2 | |
| Within-cls (sup.) | 21 | 68.9 | 77.0 |
For completeness, we also add results using DINOv3 in Table 2 and 3. As expected, features from a stronger model are less anisotropic, and therefore post-hoc normalisation in many cases becomes redundant. However, for a limited computational budget, within-class whitening is still beneficial. Most importantly, the proposed approach, after a fixed number of iterations is able to identify the model of the highest potential for a particular dataset. For the selected model one can then perform a full grid optimisation to convergence to estimate its performance, rather than for all the possible candidates.
Table2: Accuracy Neutrality of Post-Hoc Normalisation Across Foundation Models
DINOv3 ViT-B/16 converged logistic probe, using the same full-data sources and protocol as DINOv2 with CLS features. The last two columns report our best DINOv2 ViT-B results on the same data and Meta's published DINOv2 ViT-B/14 results.
| Dataset | DINOv3 Raw | DINOv3 Std | DINOv3 WCW | DINOv2 (Ours) | DINOv2 (Meta) |
|---|---|---|---|---|---|
| FGVC Aircraft | 85.2 | 84.4 | 84.3 | 82.2 | 79.4 |
| CUB-200-2011 | 89.3 | 89.0 | 88.3 | 89.9 | 89.6 |
| SUN397 | 78.4 | 77.6 | 78.2 | 77.8 | 77.3 |
| DTD | 84.2 | 82.6 | 82.3 | 85.0 | 83.3 |
Table3: Fixed-Budget Linear-Probe Evaluation on DINOv2 and DINOv3
Under the fixed-budget probe, within-class whitening's gain becomes apparent on DINOv3 using the 60-epoch AdamW protocol. Comparing raw vs within-class test top-1 accuracy (%) and their difference , for patch-mean and CLS features. Datasets are ordered by raw anisotropy. The gain reflects under-convergence under the fixed training budget, present in both backbones and generally smaller for the better-conditioned one.
| Dataset | Backbone | Patch-Mean Raw | Patch-Mean Within | Patch-Mean | CLS Raw | CLS Within | CLS |
|---|---|---|---|---|---|---|---|
| FGVC Aircraft | DINOv2 | 27.4 | 68.0 | +40.6 | 44.6 | 78.7 | +34.1 |
| DINOv3 | 43.6 | 74.6 | +31.0 | 72.1 | 82.1 | +10.0 | |
| CUB-200-2011 | DINOv2 | 42.1 | 76.5 | +34.4 | 87.6 | 89.3 | +1.7 |
| DINOv3 | 53.9 | 77.8 | +23.9 | 87.6 | 87.8 | +0.2 | |
| SUN397 | DINOv2 | 75.2 | 76.5 | +1.3 | 75.9 | 76.3 | +0.4 |
| DINOv3 | 75.2 | 77.4 | +2.2 | 77.4 | 77.8 | +0.4 | |
| DTD | DINOv2 | 77.5 | 82.6 | +5.1 | 78.3 | 81.2 | +2.9 |
| DINOv3 | 77.2 | 81.3 | +4.1 | 78.7 | 81.5 | +2.8 |
Response to Reviewer
We thank the reviewer for carefully reading our paper and for their constructive comments and suggestions. We appreciate the reviewer's recognition of our analysis of representation anisotropy across different pretraining objectives and dataset sizes, as well as the practical relevance of our proposed methodology.
We also kindly ask the reviewer to refer to our General Response to All Reviewers, where we clarify the main contributions, objectives, and scope of our work.
Insuffient control for optimization effects.
We would like to emphasise that we use the same hyperparameters and training protocol for all normalisation variants, without any model-specific parameter tuning. The observed improvements are therefore not the result of hyperparameter tweaking. We do not claim that these improvements are independent of optimisation effects. On the contrary, this is precisely our point, i.e. post-hoc normalisation improves the conditioning of the representations, making linear-probe optimisation faster and more robust under same training settings.
Centering is routinely used for optimisation purposes when training all layers of neural networks. We are simply pointing out that it is equally beneficial for designing the linear probe on features produced by a pretrained network. We agree that centering does not change the representational capacity of a linear classifier with bias, but it can substantially improve the conditioning of the optimisation problem.
All the control that is needed is to assess the degree of anisotropy, which can easily be done. Whitening is useful as well, and can be much more readily applied to fixed features than during foundation model training. The resulting preconditioning makes the optimisation process easier.
If the data for a particular application is already centred and isotropic, post-hoc normalisation is unlikely to provide any significant benefit.
We are not suggesting that standard grid-based hyperparameter tuning cannot solve the optimisation problem well. We are saying that post-hoc normalisation makes optimisation easier, faster, and more robust, without requiring extensive hyperparameter tuning.