acceptodds
Under review as a conference paper at ICLR 2027

POST-HOC NORMALISATION: A PREREQUISITE FOR LINEAR EVALUATION OF SELF-SUPERVISED FOUNDATION MODELS

Abstract

Linear evaluation by fitting a linear classifier on frozen features is the standard yardstick for the quality of self-supervised representations. It is, however, acutely sensitive to a geometric artefact that has nothing to do with representational content: anisotropy, the concentration of features in a narrow, off-centre, correlated cone, which is severe in vision transformers and inflates apparent inter-class similarity. We show that the benefit of correcting this geometry post hoc, before the linear probe, is not a fixed property of the correction, but scales inversely with how well-conditioned the frozen features already are. We propose a supervised, training-free, representation-space normalization that isotropizes intra-class (noise) covariance while leaving inter-class (signal) structure intact, and position this normalisation precisely to distinguish it from in-network nor- malization (BatchNorm/LayerNorm), projector-space anti-collapse normalization (DINO centering), and unsupervised post-hoc whitening. On CIFAR-100, the correction lifts a reconstruction-pretrained backbone (BEiT) by +27.3 points, a self-distillation backbone trained on ImageNet-1k (≈1.3M images, DINO v1) by +12.2 points, an otherwise-identical backbone that adds a patch-level masked- image term at the same scale (iBOT) by +8.4 points, and a self-distillation back- bone trained on 142M curated images (DINOv2) by +1.2 points. This monotone ladder separates two contributing factors of anisotropy: at fixed data scale a stronger objective (DINO v1→iBOT), and at fixed objective term ≈ 110× more data (iBOT→DINOv2). Fairness of linear evaluation is clouded by anisotropy induced complexity of optimisation, which the proposed post-hoc normalisation mitigates. Anisotropy diagnostics on 30000 CIFAR-100 photographs and on ten other benchmarks corroborate the mechanism. We also demonstrate the merits of the normalisation on the task of organ classification in medical data analysis.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
linear evaluation, foundation models, representation anisotropy, post-hoc normalisation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Z82zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors of this work introduce a new post-hoc supervised, training-free, representation-space normalization that aims to preserve intra-class discriminative information. They showed that the anisotropy of representations from SSL foundation models depends on data size and training objective. They showed that their method provides the largest gains in classification tasks in comparison to previous normalization methods and that the gains are weaker for newer, stronger models.

Critical strengths

Analysis beyond the proposed normalization method. The analysis is not limited to evaluating the performance of within-class whitening. The authors also investigate how pretraining objectives and dataset size affect representation anisotropy.

Promising empirical results with a simple within-class whitening approach. The proposed approach achieves substantial improvements in linear classification accuracy, particularly for weaker foundation models. The evaluation covers a diverse set of image classification benchmarks, including medical imaging, demonstrating the potential practical usefulness of the method.

Critical weaknesses

Insuffient control for optimization effects. Table 2 shows surprisingly large improvements from Centered alone (+10.8 for BEiT and +7.1 for DINO v1). Since centering is an affine transformation that can be exactly absorbed into the bias of a linear classifier, it does not change the set of representable functions. These substantial gains suggest that the reported improvements may be strongly influenced by the optimization protocol. It would be great to see albation study on the hyperparameters for training.

Insufficient analysis of the sources of performance gains, lack of statistical analysis, and limited evaluation on recent foundation models and against total-covariance whitening. The improvements over unsupervised total-covariance whitening are relatively small and there is no reported standard deviation of the results (0.7–1.8 p.p. on CIFAR-100) in Table 2. The paper lacks ablation studies investigating the extent to which the gains depend on the use of ground-truth labels (e.g., what happens if the labels are shuffled before estimating the transformation?). Furthermore, the evaluation does not include recent foundation models such as DINOv3, raising the question of whether the gains will vanish entirely. It would also be valuable to evaluate models trained with different objectives, such as I-JEPA. Given the small performance differences between within-class and total-covariance whitening in Table 2, it would be useful to include the total-covariance whitening results in Table 3 to determine whether within-class whitening consistently outperforms total-covariance whitening or whether the relative performance varies across settings.

Questions for authors
  1. Could the authors explain why the centering alone achieves such large gains in Table 2? Could the authors also provide an ablation study on the training hyperparameters for Table 2?
  2. Could the authors provide the total-covariance whitening results for the benchmarks in Table 3? This would enable a more detailed comparison beyond Table 2.
  3. Could the authors extend the evaluation to DINOv3? Given the decreasing gains observed for stronger models, it would be particularly interesting to see what the gains are for DINOv3.
  4. Could the authors provide more detailed ablation study to investigate the source of the performance gains? In particular, how much of the improvement comes from the use of ground-truth labels during normalization (for example, what would happen if we shuffled labels)?
  5. Given that the gains decrease for stronger models, how do the authors see the practical relevance of the proposed method for future generations of foundation models?
6 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for carefully reading our paper and for their constructive comments and suggestions. We appreciate the reviewer's recognition of our analysis of representation anisotropy across different pretraining objectives and dataset sizes, as well as the practical relevance of our proposed methodology.

We also kindly ask the reviewer to refer to our General Response to All Reviewers, where we clarify the main contributions, objectives, and scope of our work.

Insuffient control for optimization effects.

We would like to emphasise that we use the same hyperparameters and training protocol for all normalisation variants, without any model-specific parameter tuning. The observed improvements are therefore not the result of hyperparameter tweaking. We do not claim that these improvements are independent of optimisation effects. On the contrary, this is precisely our point, i.e. post-hoc normalisation improves the conditioning of the representations, making linear-probe optimisation faster and more robust under same training settings.

Centering is routinely used for optimisation purposes when training all layers of neural networks. We are simply pointing out that it is equally beneficial for designing the linear probe on features produced by a pretrained network. We agree that centering does not change the representational capacity of a linear classifier with bias, but it can substantially improve the conditioning of the optimisation problem.

All the control that is needed is to assess the degree of anisotropy, which can easily be done. Whitening is useful as well, and can be much more readily applied to fixed features than during foundation model training. The resulting preconditioning makes the optimisation process easier.

If the data for a particular application is already centred and isotropic, post-hoc normalisation is unlikely to provide any significant benefit.

We are not suggesting that standard grid-based hyperparameter tuning cannot solve the optimisation problem well. We are saying that post-hoc normalisation makes optimisation easier, faster, and more robust, without requiring extensive hyperparameter tuning.

Insufficient analysis of the sources of performance gains, lack of statistical analysis, and limited evaluation on recent foundation models and against total-covariance whitening.

We do not believe there is a need for any statistical analysis. For post-hoc normalisation we estimate first and second order moments. The sampling distributions for the mean and covariance matrices are well known. The idea of preconditioning is also well known. The argument is made convincingly and rigorous in Le Cun (1998) and other papers mentioned in the related work section. For the linear probe, the input data normalisation case applies. Please note that Adam optimiser estimates the preconditioner as part of its iterations. This is most appropriate for non-linear optimisation such as training a foundation model. For linear probe the preconditioner can be estimated effectively using the whole dataset. This is what speeds up the convergence and enables optimisation with fixed parameters and is the crux of our contribution. We will include this explanation in our revised manuscript.

To illustrate the benefit of normalisation under the assumption of fixed computational budget we show the impact of various types of second order statistics preconditioning in Table 1 on a subset of classes. (For all four scenarios the data is centered, which eliminates of the conditioning variables.) The results show that even Adam is unable to cope very well for some datasets such as Aircraft. As suggested, we include the results that show the effect of normalisation using total covariance matrix.

Table1: Decomposing the Preconditioner: Which Normalisation Speeds Up the Probe?

Results on the four most anisotropic benchmarks (DINOv2 ViT-B/14, CLS features). κ\kappa denotes the condition number of the training feature covariance matrix (the ratio of its largest to smallest eigenvalue), with lower values indicating better conditioning. The SGD and Adam columns report budget-limited test top-1 accuracy (%) after 50 epochs at a shared learning rate.

All variants are rescaled to equal mean norm, so only covariance shape differs. Per-dimension standardisation (diagonal) barely helps; full whitening is required. Label-free total-covariance whitening captures most of the gain, with within-class whitening providing additional improvements under Adam on several datasets.

DatasetNormalisationCondition Number (κ\kappa)SGD (50 epochs)Adam (50 epochs)
CUB-200-2011None (raw)5.2×1045.2 \times 10^{4}81.688.5
Std (diagonal)2.1×1032.1 \times 10^{3}81.888.7
Total cov1.288.389.1
Within-cls (sup.)7486.489.8
DTDNone (raw)2.2×1042.2 \times 10^{4}74.579.6
Std (diagonal)8.1×1028.1 \times 10^{2}74.779.4
Total cov (full)1.281.079.7
Within-cls (sup.)1479.881.9
FGVC AircraftNone (raw)3.9×1053.9 \times 10^{5}33.250.0
Std (diagonal)1.6×1041.6 \times 10^{4}35.952.9
Total cov4.071.376.7
Within-cls (sup.)1.4×1021.4 \times 10^{2}73.480.5
SUN397None (raw)7.0×1037.0 \times 10^{3}67.777.1
Std (diagonal)2.2×1022.2 \times 10^{2}67.977.1
Total cov1.672.176.2
Within-cls (sup.)2168.977.0

For completeness, we also add results using DINOv3 in Table 2 and 3. As expected, features from a stronger model are less anisotropic, and therefore post-hoc normalisation in many cases becomes redundant. However, for a limited computational budget, within-class whitening is still beneficial. Most importantly, the proposed approach, after a fixed number of iterations is able to identify the model of the highest potential for a particular dataset. For the selected model one can then perform a full grid optimisation to convergence to estimate its performance, rather than for all the possible candidates.

Table2: Accuracy Neutrality of Post-Hoc Normalisation Across Foundation Models

DINOv3 ViT-B/16 converged logistic probe, using the same full-data sources and protocol as DINOv2 with CLS features. The last two columns report our best DINOv2 ViT-B results on the same data and Meta's published DINOv2 ViT-B/14 results.

DatasetDINOv3 RawDINOv3 StdDINOv3 WCWDINOv2 (Ours)DINOv2 (Meta)
FGVC Aircraft85.284.484.382.279.4
CUB-200-201189.389.088.389.989.6
SUN39778.477.678.277.877.3
DTD84.282.682.385.083.3

Table3: Fixed-Budget Linear-Probe Evaluation on DINOv2 and DINOv3

Under the fixed-budget probe, within-class whitening's gain becomes apparent on DINOv3 using the 60-epoch AdamW protocol. Comparing raw vs within-class test top-1 accuracy (%) and their difference Δ\Delta, for patch-mean and CLS features. Datasets are ordered by raw anisotropy. The gain reflects under-convergence under the fixed training budget, present in both backbones and generally smaller for the better-conditioned one.

DatasetBackbonePatch-Mean RawPatch-Mean WithinPatch-Mean Δ\DeltaCLS RawCLS WithinCLS Δ\Delta
FGVC AircraftDINOv227.468.0+40.644.678.7+34.1
DINOv343.674.6+31.072.182.1+10.0
CUB-200-2011DINOv242.176.5+34.487.689.3+1.7
DINOv353.977.8+23.987.687.8+0.2
SUN397DINOv275.276.5+1.375.976.3+0.4
DINOv375.277.4+2.277.477.8+0.4
DTDDINOv277.582.6+5.178.381.2+2.9
DINOv377.281.3+4.178.781.5+2.8
Reviewer_74HYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how anisotropy affects linear evaluation of frozen representations. It uses downstream labels to estimate a centering and whitening transform without retraining the backbone. Experiments on image classification and medical imaging show gains that are generally larger for weaker backbones or more challenging tasks.

Critical strengths
  1. The paper highlights a useful concern that feature geometry can affect how we judge representation quality.
  2. The method is simple to apply and requires no backbone retraining.
Critical weaknesses
  1. The main method depends on class labels to estimate the covariance within each class. This is reasonable for supervised linear probing, but in recent applications that use pretrained representations for image and video generation, VLM understanding, or world modeling, labels are unavailable and the proposed method may fail.

  2. The paper does not discuss DSE [1], a closely related method that assesses representation structure using clustering, class separability, and effective dimensionality. A discussion and comparison would help clarify the contribution.

  3. These two concerns suggest a natural experiment: estimate the whitening transform using clustering assignments as pseudo labels. Comparing this variant with whitening based on true labels and the existing unsupervised whitening baseline would show whether approximate groups are enough to retain the gains. Neither the main text nor the appendix explores this possibility.

  4. Evaluation at larger downstream data scales is missing. Appendix A caps the Table 3 training sets at 20,000 images and SUN397 at 8,000. Using backbones pretrained on large corpora does not fill this gap. Results on a full benchmark such as ImageNet-1k would make the scalability claim more convincing.

  5. It is unclear how much of the gain comes from the probe's optimization settings. Appendix A fixes training at 60 epochs with shared hyperparameters. An invertible affine transform does not change what a linear classifier with a bias can represent. Separate tuning and training to convergence would help distinguish optimization and regularization effects from evidence that whitening is a prerequisite for linear evaluation.

[1] Dai et al. Exploring Structural Degradation in Dense Representations for Self-supervised Learning. NeurIPS 2025.

Questions for authors

Please refer to the weaknesses above. I am willing to increase my score if my concerns are adequately addressed.

6 comments
AuthorsOpenReview ↗

Response to the Reviewer

We thank the reviewer for carefully reading our paper and for their constructive comments and suggestions. We also appreciate the reviewer's recognition of the importance and the simplicity of our proposed methodology. We kindly ask the reviewer to refer to our General Response to All Reviewers, where we clarify the main contributions, objectives, and scope of our work.

1. Dependence on Class Labels and Applicability to Unsupervised Settings

Centering does not require class labels, and whitening can be performed using the total covariance matrix rather than the within-class covariance matrix. Hence, post-hoc normalisation is not restricted to supervised settings and is generally applicable without class labels.

As discussed in Section 3, within-class whitening is particularly suitable for supervised linear evaluation because it preserves the between class structure, whereas total covariance whitening also normalises the directions separating class means.

2. Relationship to DSE and Clarification of Our Contribution

The authors of the referenced paper propose an alternative measure for assessing representation quality, rather than improving the reliability of linear probing. They do not address the issue of post hoc representation normalisation or its impact on linear probe optimisation. Therefore, while both works concern representation evaluation, their objectives and contributions are fundamentally different, and DSE is not a direct baseline for our proposed methodology.

3. Pseudo-Label Whitening Using Clustering Assignments

Thank you for this suggestion. This would be an interesting experiment, especially to check whether that would give better performance as compared with the easy approach using the total covariance matrix. However, clustering in high dimensional spaces is non-trivial, and we have some doubts that it will be an easy approach to follow. We will mention this in future research directions. However, performance using no labels without clustering (pseudo labelling) is shown for most anisotropic datasets in the table below, referred to as total covariance. In conjunction with Adam optimiser, within class covariance prewhitening is either beneficial or neutral.

Table: Decomposing the Preconditioner: Which Normalisation Speeds Up the Probe?

Results on the four most anisotropic benchmarks (DINOv2 ViT-B/14, CLS features). κ\kappa is the condition number of the training feature covariance. The SGD and Adam columns report budget-limited test top-1 accuracy (%) after 50 epochs at a shared learning rate. All variants are rescaled to equal mean norm, so only covariance shape differs. Per-dimension standardisation (diagonal) barely helps; full whitening is required. Label-free total-covariance whitening captures most of the gain, with within-class whitening adding only a small edge under Adam.

DatasetNormalisationκ\kappaSGD (50 epochs)Adam (50 epochs)
CUB-200-2011None (raw)5.2×1045.2 \times 10^{4}81.688.5
Std (diagonal)2.1×1032.1 \times 10^{3}81.888.7
Total cov1.288.389.1
Within-cls (sup.)7486.489.8
DTDNone (raw)2.2×1042.2 \times 10^{4}74.579.6
Std (diagonal)8.1×1028.1 \times 10^{2}74.779.4
Total cov (full)1.281.079.7
Within-cls (sup.)1479.881.9
FGVC AircraftNone (raw)3.9×1053.9 \times 10^{5}33.250.0
Std (diagonal)1.6×1041.6 \times 10^{4}35.952.9
Total cov4.071.376.7
Within-cls (sup.)1.4×1021.4 \times 10^{2}73.480.5
SUN397None (raw)7.0×1037.0 \times 10^{3}67.777.1
Std (diagonal)2.2×1022.2 \times 10^{2}67.977.1
Total cov1.672.176.2
Within-cls (sup.)2168.977.0

4. Evaluation at Larger Downstream Data Scales and ImageNet-1K

We are not making any specific claims about scalability. If the representation for a particular application is not excessively anisotropic, then the proposed normalisation is unlikely to be useful. As ImageNet-1K is the main source for training many models, post-hoc normalisation is expected to provide limited benefits when evaluating these models on the same pre-training data, especially for invariance-based SSL approaches.

Please see our response to Reviewer TKBZ regarding ImageNet-1K linear evaluation.

For larger datasets, if the representation is anisotropic, an additional advantage is that the covariance matrix estimates can become more reliable as more samples become available. The beauty of the proposed approach is that anisotropy can be assessed very simply, allowing practitioners to determine whether normalisation is likely to be beneficial.

5. Impact of Optimisation, Hyperparameter Tuning, and Training to Convergence

We agree exactly with your insightful comments. The post-hoc normalisation does not affect the representation. It only conditions the representation vectors so that optimisation becomes considerably straightforward. The difference in linear probe optimisation between raw and normalised data is shown in Fig 1 of our paper for our medical data application.

The impact of post-hoc normalisation is evident in the table below. This does not mean that the solution obtained with or without post-hoc normalisation would differ, if the optimisation is done to convergence and the hyper-parameters are optimised using grid based method.

Table: Within-Class Whitening Accelerates Convergence of a Gradient-Trained Probe

CIFAR-100 test top-1 accuracy (%) by epoch. Plain SGD (learning rate = 0.01, momentum = 0.9), with scale-normalised features (so the difference is conditioning, not scale). Adam rows (DINOv2) show that an adaptive optimiser partly self-preconditions.

OptimizerBackbone / RepresentationEpoch 1Epoch 2Epoch 3Epoch 5Epoch 10Epoch 20Epoch 40
SGDBEiT raw1.11.32.64.59.111.112.6
BEiT within-cls1.94.79.720.335.043.146.0
DINOv2 raw33.357.567.272.975.676.879.2
DINOv2 within-cls29.760.172.579.582.984.185.5
AdamDINOv2 raw81.582.683.384.886.888.289.2
DINOv2 within-cls86.786.987.488.188.889.690.2

As the rebuttal period ends tomorrow, we would appreciate any follow up comments or questions to allow us to engage in further discussion.

Reviewer_TKBZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript proposes applying within-class covariance whitening to the frozen embeddings of self-supervised image models before linear probing, and shows that it improves performance across multiple datasets and backbones, with larger gains for weaker models.

Critical strengths
  • The manuscript evaluates the method on multiple datasets and models: four ViT-B backbones (BEiT, DINO v1, iBOT, DINOv2), eleven natural image benchmarks, and a CT classification task.
Critical weaknesses
  • The experiments are limited to small- and medium-scale datasets. Since ImageNet-1k linear probing is the standard protocol for evaluating self-supervised representations, the authors should report results on ImageNet-1k.

  • The reported linear-probe accuracies are substantially below published results. For example, DINOv2-base reaches 35.6% on Stanford Cars here, compared with 88.2% in the DINOv2 paper (Table 8).

Arch Food C10 C100 SUN Cars Aircr VOC DTD Pets Cal101 Flowers CUB Avg

ViT-S/14 89.1 97.7 87.5 74.4 81.6 74.0 87.8 80.6 95.1 97.0 99.6 88.1 87.7

ViT-B/14 92.8 98.7 91.3 77.3 88.2 79.4 88.2 83.3 96.2 96.1 99.6 89.6 90.1

ViT-L/14 94.3 99.3 93.4 78.7 90.1 81.5 88.3 84.0 96.6 97.5 99.7 90.5 91.2

ViT-g/14 94.7 99.5 94.4 78.7 91.4 87.2 89.0 84.5 96.7 97.6 99.7 91.6 92.1

Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." arXiv preprint arXiv:2304.07193 (2023).

Minor:

  • "73.0% test accuracy on raw features and 87.9% on within-class-whitened ones, a gain of +14.8 points from a transform with no learned parameters": 87.9% - 73.0% = 14.9%
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for carefully reading our paper and for their constructive comments and suggestions. We would also kindly ask the reviewer to refer to our General Response to All Reviewers, where we clarify the main contributions, objectives, and scope of our work.

1. ImageNet-1K Linear Evaluation

Thank you for the suggestion. We agree with the reviewer that ImageNet-1k linear probing is a standard evaluation protocol. However, the main objective of the paper is not to maximise linear probing accuracy on a particular benchmark, but to establish when and why post-hoc normalisation is useful.

As shown in Section 5 (page 6, first paragraph), supported by Tables 2 and 3, the improvement depends on how anisotropic the representations are. For example, Table 3 (page 7) shows that even for the same foundation model, the improvement can be very small on some datasets and very large on others. This is why we deliberately evaluate our method on different downstream datasets.

Furthermore, Table 1 (page 5) already examines the representations of all four foundation models on 30,000 ImageNet-100 images. Thus, we have not excluded the ImageNet domain from our analysis.

We therefore believe that our existing experiments provide more direct evidence for the main argument of the paper, while ImageNet-1k linear probing would mainly add another benchmark result rather than provide new insight into when normalisation is useful.

We hope that the reviewer will appreciate the rationale behind our experimental choices and agree that the absence of this additional experiment does not undermine the main findings or conclusions of our work.

Nevertheless, we appreciate the suggestion and will make an effort to include ImageNet-1k linear-probing results in the camera-ready version.

2. Discrepancy with Published Linear Probe Results

This observation is actually closely related to the core motivation of our paper. When pretrained foundation models are evaluated using a linear probe without extensive hyperparameter tuning, the resulting accuracy can be substantially lower than published results.

Our work demonstrates that much of this difference can arise from poorly conditioned representations, which can be corrected through post-hoc normalisation. An additional benefit of this correction is that it reduces the need for model specific hyperparameter tuning, allowing different foundation models to be evaluated more fairly using the same linear probe settings.

For the example you have raised, we used same weights (DINOv2) and fixed hyper-paramaters and the accuracy jumped from 35.6% to 79.8% simply by applying post-hoc normalisation, with all other settings unchanged.

Moreover, as stated in Page 5 Section 4, "Unless stated otherwise the classifier is a linear probe trained for 60 epochs", our reported results were obtained using only 60 training epochs, without data augmentation.

In response to another reviewer 74HY, we have extended training to convergence. As expected, accuracy improved by a further few percentage points, in some cases even surpassing the published DINOv2 results, as shown in table below.

DatasetDINOv2 (ours)DINOv2 (Published results)
FGVC Aircraft82.279.4
CUB-200-201189.989.6
SUN39777.877.3
DTD85.083.3

We believe these findings reinforce the main argument of our paper, i.e. poor representation conditioning can substantially affect linear-probe accuracy, and post-hoc normalisation helps reveal the quality of the underlying representations.

These results directly address the reviewer's concern by demonstrating that, with sufficient training and, most importantly, without model specific hyperparameter tuning, our linear-probe accuracies match or surpass the published DINOv2 results on all four datasets. Again, we want to emphasise that outperforming published results is not the goal of this paper, although our results already demonstrate this on several datasets.

With only one day left in the rebuttal period, we would greatly appreciate any further feedback or questions so we can have the opportunity to discuss and address your concerns.

With the rebuttal discussion closing in about an hour, we would greatly appreciate the opportunity to have at least one interaction with you before the deadline, as we have not yet received any feedback on our responses. We hope to hear your thoughts and address any remaining concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

Response to All Reviewers

Thank you very much for reading our paper and for your comments.

1. Scope and Main Contribution

As a pre-amble, we would like to emphasize that our paper is a discovery and methodology paper which we believe will play an important role in the development of solutions for diverse applications.

The discovery is the anisotropy of fixed network representations for a given dataset, which we showed to be inversely proportional to the representational power of a foundation model.

2. Implications for Linear Probes

This anisotropy can have an overarching impact on the design and reliability of the linear probe for measuring the effectiveness of a fixed model representation for a given target data.

Linear probes are just like a measurement tool. They can be more or less precise, they can be easy or difficult to use, they may require more or less time to accomplish a set of measurements, and they can be more or less robust.

3. Proposed Methodology: Post-hoc Normalisation

We propose a simple, and in statistics, as well as in machine learning, well know corrections for the anisotropy in terms of centering, pre-whitening by total covariance matrix and by within-class covariance matrix, which make the linear probe measurement less intricate, considerably faster, and more robust than when used without it.

Although centering and prewhitening are the standard tools for normalisation at different layers of a model, we are the first to propose the use of such normalisation at the output of a foundation model.

It makes the linear probe fast a more robust which are important properties from the designed point of view.

4. Clarification of Our Objective

We are not competing with anybody in terms of performance. We are showing that the conditioning achieved by the post-hoc normalisation is extremely helpful.

5. Robustness and Reliability: The Case of BEiT

We are not saying that with very carefully designed linear probe for BEiT you could not get a reliable reading of the true strength of the model, but the anisotropy of its representation is so strong that it can lead a fragile tool to wrong conclusions, unless the designer devotes extraordinary amount of time in fine tuning to prevent it.

After post-hoc normalisation the tool becomes robust, easy to use and very fast.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.