acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Zero-Shot Capabilities in Vision Foundation Models

Abstract

Frozen vision foundation models are used zero-shot on tasks they were never trained for, such as matching features across images for semantic and geometric correspondence, tracking points through video, and nearest-neighbour classification. Evaluations extract features, compare them by cosine similarity, and choose the candidate with the highest score. Cosine similarity does not account for how variance is distributed across feature directions. In many models, a few directions carry most of the variance. We measure this concentration by the effective dimension of the covariance of the extracted feature vectors. Across 36 models and 46 extraction points, ranges from to , despite feature dimensions ranging from to . Such low effective dimensions matter because the cosine similarity between unrelated features has variance of about , so when is small, wrong candidates can score almost as highly as the correct match. We therefore whiten the features with their mean and covariance, estimated on unlabeled images, before computing cosine similarity. Whitening requires no labels or training and has long been standard in image retrieval, yet it is not part of the usual zero-shot evaluation of foundation-model features. Across eleven dense benchmarks, whitening improves performance on of model–benchmark pairs, by points on average and up to PCK points. The gain is larger for models with lower pre-whitening . Diffusion transformers gain the most. Their raw features match poorly under cosine similarity, but after whitening they achieve matching performance comparable to strong self-supervised encoders. Whitening should be part of the zero-shot evaluation of frozen vision features.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
foundation models, zero-shot

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ajuCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that cosine similarity, the standard metric for zero-shot evaluation of frozen vision features, is sensitive to how feature variance is distributed across directions. The proposed fix whitens features using statistics estimated from unlabeled images before computing similarity. This simple correction consistently improves performance on various benchmarks.

Critical strengths
  • The paper gives a clear explanation for why cosine similarity can perform poorly when feature variance is concentrated in a few directions.
  • The evaluation is large-scale and carefully controlled, covering many models and benchmarks.
Critical weaknesses
  • The methodological contribution is somewhat limited. Whitening and per-channel standardization are already established in retrieval and NLP, and much of the gain appears to come from simple per-channel rescaling.
  • Each diffusion-transformer evaluation uses a single noise sample, and the paper does not report how stable the large gains are across noise draws.
Questions for authors
  • Could the authors discuss how the effective-dimension account relates to simpler explanations based on per-channel scale?
  • Do the large diffusion-transformer gains persist across multiple noise samples?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ajuC (1/3)

Thank you for your valuable feedback, which helps us improve our paper. We detail our response below; please let us know if it addresses your concerns.


W1: The methodological contribution is somewhat limited. Whitening and per-channel standardization are already established in retrieval and NLP, and much of the gain appears to come from simple per-channel rescaling.

Short answer. We agree on both points, and the paper says so (Sections 2 and 6): whitening and per-channel standardization are established, we do not claim them, and for raw DiTs most of the gain is per-channel rescaling (Q1 below shows where this stops being true). Our contribution is about how frozen features are evaluated, in three parts.

  1. Cosine evaluation is unfair, and we show why. Zero-shot dense benchmarks and prior comparisons of backbones (e.g. Zhang et al., 2025; Dünkel et al., 2026) and of DiT readouts (Gan et al., 2025) all score frozen features with cosine, so a low score can come from the features or from the unfair metric. The cosine of two unrelated features has a variance of about 1/d_eff (Eq. 2), where d_eff is the effective dimension of the features: the participation ratio (Tr Σ)² / Tr(Σ²), roughly the number of directions that carry the variance. Features with low d_eff get noisy matches whatever they encode. d_eff ranges from 1.7 to 374 across our 46 readouts and predicts which ones gain.
  2. A simple, known fix corrects it and changes the rankings. Whitening with moments from 250 unlabeled images improves 459 of 506 evaluations, by up to 48 points (FLUX.1 raw on CC). Kendall τ between the cosine and whitened rankings is 0.63–0.77, and the top readout changes on 4 of 11 benchmarks.
  3. It unlocks representations that cosine hides, across models and across blocks. Under cosine, raw DiT features average 34.8 over our 11 benchmarks, below the 26 vision encoders (37.6), and prior work changes the readout to fix them. Whitened, the same features average 48.8, above the encoders (41.2) and level with the engineered post-norm readouts (48.9); raw FLUX.1, for example, moves from 36th to 2nd on ScanNet. Two raw DiTs (GLM-Image, HiDream-O1) stay weak, so whitening shows which models were held back rather than making every model good. The same happens within a model. For this response we ran all 32 blocks of FLUX.2-klein raw and 24 of the 36 blocks of ERNIE-Image raw on the SPair-71k test split, and every block of DINOv2-B and DINOv3-B (blocks around the DiT peaks shown; blocks 11 and 7 are the paper's numbers):
FLUX.2 block8910111213141516171819
cosine53.055.555.854.552.648.444.638.934.130.830.338.2
whitened56.559.160.562.964.565.165.164.763.863.162.662.3
ERNIE block6789101112131415161718
cosine39.638.236.334.233.932.732.034.737.341.745.344.644.7
whitened57.258.657.756.155.254.152.853.754.656.357.756.254.3

Interestingly, whitening also changes which block looks best. On the SPair-71k test split, FLUX.2-klein under cosine peaks at block 10 (55.8) and falls to 30.3 by block 18, so the deeper blocks look unusable; after whitening every block improves, the deeper blocks become the best (65.1 at blocks 13–14), and blocks 11–19 are all within 3 points. ERNIE-Image under cosine is highest at block 16 (45.3), far from the block chosen on validation (7, 38.2); after whitening block 7 is the best (58.6) and blocks 6–18 are within 6 points. So whitening raises every block, flattens the curve, and makes the block choice much less critical. This does not contradict the paper: there we chose each DiT's block by cosine on the SPair-71k validation split (Table 4) and kept it for whitening, so the paper's whitened numbers are conservative (FLUX.2-klein: 62.9 at block 11, 65.1 at its best block). Even at each metric's own best block, whitening gains +9.3 (FLUX.2-klein) and +13.3 (ERNIE-Image). The encoders behave differently; their last block is best under both metrics:

Block01234567891011
DINOv2-B, cosine5.28.610.313.319.928.329.039.046.752.756.156.5
DINOv2-B, whitened6.08.711.015.323.131.036.143.648.153.456.557.7
DINOv3-B, cosine4.88.410.614.722.229.438.245.849.854.055.555.7
DINOv3-B, whitened5.37.89.915.223.430.639.547.351.355.457.858.8

We therefore recommend reporting whitened scores next to cosine.

AuthorsOpenReview ↗

Response to Reviewer ajuC (2/3)

Q1: Could the authors discuss how the effective-dimension account relates to simpler explanations based on per-channel scale?

Short answer. Per-channel scale is one way to get a low d_eff, and for raw DiTs it is the main one. But d_eff also explains three things that channel scale cannot: lowering d_eff with equal channel scales still hurts, removing correlations still helps encoders, and the matching noise follows d_eff at every step. We see per-channel scale as a special case of the d_eff account and will present it that way.

1. Lowering d_eff with equal channel scales. To isolate d_eff, we change it while keeping the readout, the images, the matching rule and every channel's variance fixed. We start from each readout's whitened features, which have the same variance in every direction. We then multiply every feature by one fixed matrix R D Rᵀ: R is a random rotation, and D = diag(1, 2^(−s), …, d^(−s)) scales the k-th random direction by k^(−s), so its variance becomes k^(−2s). For s > 0 the variance concentrates in the first few directions and d_eff falls, from a median of 562 at s = 0.125 to 2.5 at s = 1 (six values of s). Because the random directions are not aligned with the channels, every channel keeps about the same variance, and cosine ignores the overall scale. If only per-channel scale mattered, PCK would not change. Instead it drops in all 46 readouts on SPair-71k, SOC, CC, ScanNet and NAVI (SOC and CC: the semantic object and concept correspondence tasks of SOCO (Dünkel et al., 2026), the largest and newest semantic correspondence benchmark, ECCV 2026), and the drop follows d_eff. Median PCK loss on SPair-71k:

d_eff after the change≥ 500250–500150–25050–15010–50< 10
Median PCK loss0.20.80.73.813.332.0

The paper's own results show the same threshold. All 506 readout × benchmark cells, grouped by the readout's d_eff:

d_effReadoutsCells improvedMedian gain
below 5016174/176+3.8
50 to 15015158/165+4.4
150 to 250994/99+1.1
above 250633/66+0.0

2. Encoders need more than rescaling. Mean gain over cosine of diagonal (per-channel) scaling and full whitening, as in Table 2:

SOC diagSOC fullCC diagCC fullSPair diagSPair full
Encoders (26)+2.05+3.67+2.73+4.94+2.80+4.48
Raw DiTs (10)+16.23+16.65+18.45+18.82+18.49+18.33
Post-norm DiTs (10)+1.01+1.43+1.33+2.00+1.30+1.67

In raw DiTs a few channels carry most of the variance, so rescaling fixes almost everything. Encoders have fairly even channel variances, yet removing correlations adds another 1.6 to 2.2 points. Across the 46 readouts, the spread of channel variances predicts the gain of diagonal scaling (Spearman +0.64 on SPair) but not the extra gain of full whitening (−0.15); the d_eff of the correlation matrix predicts the extra gain (−0.69).

3. The matching noise follows d_eff at every step. Eq. 2 comes from a null model that treats the query and a wrong patch as independent draws from the feature distribution (mean μ, covariance Σ, estimated on ImageNet). From Σ alone it predicts that, after each correction, the scores of wrong matches have a std of 1/√d_eff of the corrected features' covariance. We measured this std directly: for every annotated keypoint of SOC and SPair-71k, the std of the scores of all wrong patches (outside the PCK radius), median over keypoints. Over 92 cells (46 readouts × 2 benchmarks; medians over cells):

AfterCovariance after the stepPredicted std ↓ (1/√d_eff)Measured std ↓Measured / predicted per cell, median (IQR)
centeringΣ0.1160.1091.00 (0.82–1.21)
diagonalD^(−1/2) Σ D^(−1/2), D = per-channel variances0.0730.0751.03 (0.89–1.21)
fullΣ_α^(−1/2) Σ Σ_α^(−1/2)0.0410.0501.19 (1.08–1.35)

D and Σ_α include the shrinkage α of Eq. 3. After diagonal scaling every channel has the same scale, yet the measured noise still matches 1/√d_eff of the correlation matrix D^(−1/2) Σ D^(−1/2). So what sets the noise is d_eff, and channel scale is one part of it. In practice we keep full whitening as the default and recommend diagonal scaling as a cheaper option for raw DiTs.

AuthorsOpenReview ↗

Response to Reviewer ajuC (3/3)

W2 / Q2: Each diffusion-transformer evaluation uses a single noise sample, and the paper does not report how stable the large gains are across noise draws. Do the large diffusion-transformer gains persist across multiple noise samples?

Short answer. Yes. Across 5 noise seeds the gain of whitening varies by a standard deviation of about 0.1 PCK and never changes sign.

We re-ran all 21 noise-dependent readouts (the 20 DiT readouts and DIFT at t = 250) with 5 noise seeds. Each seed is a full evaluation with one noise sample per image, as in the paper, and μ and Σ were re-estimated for each seed, so both sides of the comparison see new noise. Gain of whitening over cosine, mean ± std over the 5 runs:

FLUX.1SD3.5-LPixArt-αERNIE-ImageGLM-ImageKrea2HiDream-O1Qwen-ImageFLUX.2-kleinZ-ImageDIFT (t = 250)
SOC+41.0 ± 0.11+17.7 ± 0.09+16.0 ± 0.17+17.2 ± 0.14+19.0 ± 0.08+19.0 ± 0.13+16.5 ± 0.17+12.8 ± 0.14+4.2 ± 0.11+3.3 ± 0.13+3.2 ± 0.14
CC+48.5 ± 0.13+20.3 ± 0.11+18.8 ± 0.13+21.0 ± 0.07+22.3 ± 0.14+19.5 ± 0.13+17.5 ± 0.19+12.1 ± 0.12+5.2 ± 0.09+3.2 ± 0.14+4.8 ± 0.13

Over all 21 readouts the median std is 0.11 (SOC) and 0.09 (CC), the largest is 0.19, and the sign of the gain never changes (0 of 42 cases).

The paper uses one noise sample per image so that DiTs, like encoders, get a single forward pass per image; the table shows that this choice does not affect the conclusions. We will add it to the appendix.

Thank you for the detailed response and the additional experiments. The response and new experiments have addressed my concerns, so I will maintain my positive score.

Reviewer_EuofOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how feature anisotropy affects the zero-shot performance of vision foundation models. The authors introduce effective dimension to characterize the variance distribution of extracted features and analyze its relationship with cosine similarity. They apply unsupervised feature whitening to improve feature matching without additional training. Experiments across 36 models and 11 dense correspondence benchmarks show substantial improvements, particularly for diffusion transformers.

Critical strengths

Extensive experimental validation. The evaluation covers 36 models and multiple correspondence benchmarks, providing convincing evidence that feature normalization substantially affects zero-shot performance and model rankings.

Critical weaknesses
  1. Limited methodological novelty. Feature whitening and anisotropy correction are well-established techniques. The main contribution lies in their systematic evaluation on modern foundation models rather than a new method.
  2. Limited investigation of alternative similarity metrics. The paper primarily compares cosine similarity with whitening-based variants. A broader comparison with other unsupervised similarity metrics would strengthen the conclusion that whitening should become a standard evaluation procedure.
Questions for authors
  1. How does the proposed whitening approach compare with other unsupervised similarity metrics, such as alternative covariance-normalized similarity measures?
  2. Table 2 shows that diagonal covariance normalization already achieves most of the improvement. Is full covariance whitening necessary, considering its additional computational and memory costs?
  3. How sensitive are the results to the distribution and size of the unlabeled images used to estimate the whitening statistics, particularly under significant domain shifts?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Euof (1/3)

Thank you for your valuable feedback, which helps us improve our paper. We detail our response below; please let us know if it addresses your concerns.


W1: Limited methodological novelty. Feature whitening and anisotropy correction are well-established techniques. The main contribution lies in their systematic evaluation on modern foundation models rather than a new method.

Short answer. We agree that whitening itself is not new: it is established in retrieval and NLP, and we do not claim it (Sections 2 and 6). Our contribution is about how frozen features are evaluated, in three parts.

  1. Cosine evaluation is unfair, and we show why. Zero-shot dense benchmarks and prior comparisons of backbones (e.g. Zhang et al., 2025; Dünkel et al., 2026) and of DiT readouts (Gan et al., 2025) all score frozen features with cosine, so a low score can come from the features or from the unfair metric. The cosine of two unrelated features has a variance of about 1/d_eff (Eq. 2), where d_eff is the effective dimension of the features: the participation ratio (Tr Σ)² / Tr(Σ²), roughly the number of directions that carry the variance. Features with low d_eff get noisy matches whatever they encode. d_eff ranges from 1.7 to 374 across our 46 readouts and predicts which ones gain.
  2. A simple, known fix corrects it and changes the rankings. Whitening with moments from 250 unlabeled images improves 459 of 506 evaluations, by up to 48 points (FLUX.1 raw on CC). Kendall τ between the cosine and whitened rankings is 0.63–0.77, and the top readout changes on 4 of 11 benchmarks.
  3. It unlocks representations that cosine hides, across models and across blocks. Under cosine, raw DiT features average 34.8 over our 11 benchmarks, below the 26 vision encoders (37.6), and prior work changes the readout to fix them. Whitened, the same features average 48.8, above the encoders (41.2) and level with the engineered post-norm readouts (48.9); raw FLUX.1, for example, moves from 36th to 2nd on ScanNet. Two raw DiTs (GLM-Image, HiDream-O1) stay weak, so whitening shows which models were held back rather than making every model good. The same happens within a model. For this response we ran all 32 blocks of FLUX.2-klein raw and 24 of the 36 blocks of ERNIE-Image raw on the SPair-71k test split, and every block of DINOv2-B and DINOv3-B (blocks around the DiT peaks shown; blocks 11 and 7 are the paper's numbers):
FLUX.2 block8910111213141516171819
cosine53.055.555.854.552.648.444.638.934.130.830.338.2
whitened56.559.160.562.964.565.165.164.763.863.162.662.3
ERNIE block6789101112131415161718
cosine39.638.236.334.233.932.732.034.737.341.745.344.644.7
whitened57.258.657.756.155.254.152.853.754.656.357.756.254.3

Interestingly, whitening also changes which block looks best. On the SPair-71k test split, FLUX.2-klein under cosine peaks at block 10 (55.8) and falls to 30.3 by block 18, so the deeper blocks look unusable; after whitening every block improves, the deeper blocks become the best (65.1 at blocks 13–14), and blocks 11–19 are all within 3 points. ERNIE-Image under cosine is highest at block 16 (45.3), far from the block chosen on validation (7, 38.2); after whitening block 7 is the best (58.6) and blocks 6–18 are within 6 points. So whitening raises every block, flattens the curve, and makes the block choice much less critical. This does not contradict the paper: there we chose each DiT's block by cosine on the SPair-71k validation split (Table 4) and kept it for whitening, so the paper's whitened numbers are conservative (FLUX.2-klein: 62.9 at block 11, 65.1 at its best block). Even at each metric's own best block, whitening gains +9.3 (FLUX.2-klein) and +13.3 (ERNIE-Image). The encoders behave differently; their last block is best under both metrics:

Block01234567891011
DINOv2-B, cosine5.28.610.313.319.928.329.039.046.752.756.156.5
DINOv2-B, whitened6.08.711.015.323.131.036.143.648.153.456.557.7
DINOv3-B, cosine4.88.410.614.722.229.438.245.849.854.055.555.7
DINOv3-B, whitened5.37.89.915.223.430.639.547.351.355.457.858.8

We therefore recommend reporting whitened scores next to cosine.

AuthorsOpenReview ↗

Response to Reviewer Euof (2/3)

W2 / Q1: Limited investigation of alternative similarity metrics. A broader comparison with other unsupervised similarity metrics would strengthen the conclusion that whitening should become a standard evaluation procedure. How does the proposed whitening approach compare with other unsupervised similarity metrics, such as alternative covariance-normalized similarity measures?

Short answer. Whitened cosine (Eq. 3) is the only measure that beats cosine for every model family on all five benchmarks we tested. CSLS, a hubness correction, adds to it on semantic matching but hurts geometric matching.

Besides the measures in Table 2 (centering, all-but-the-top, LayerNorm, diagonal scaling), we compared, on all 46 readouts with the same features and matching rule:

  • Euclidean distance on the raw features;
  • Mahalanobis distance, i.e. Euclidean distance in the whitened space of Eq. 3 (Eq. 3 without the normalization);
  • truncated PCA-whitening (Jégou & Chum, 2012), keeping the top 64 or 256 directions;
  • CSLS (Conneau et al., 2018): 2 cos(x, y) − r(y), where r(y) is the mean cosine of target patch y to its 10 nearest source patches, on raw and on whitened features.

Semantic matching, mean PCK on SPair-71k and on the two tasks of SOCO (Dünkel et al., 2026), the largest and newest semantic correspondence benchmark (ECCV 2026): SOC (semantic object correspondence) and CC (concept correspondence):

cosEq. 3 (paper)Euclid.Mahal.PCA-w 64PCA-w 256CSLSCSLS + Eq. 3
SPair, encoders33.337.829.625.730.135.435.538.3
SPair, raw DiTs33.551.830.023.523.045.538.453.3
SPair, post-norm DiTs50.352.039.432.242.548.253.253.7
SOC, encoders35.939.632.829.632.437.338.040.6
SOC, raw DiTs33.249.830.524.823.043.837.351.5
SOC, post-norm DiTs48.650.139.832.441.546.651.452.0
CC, encoders48.253.144.239.543.750.350.754.0
CC, raw DiTs44.363.240.830.731.256.449.164.8
CC, post-norm DiTs61.363.349.940.053.258.864.565.3

Over these 138 readout × benchmark cells, Euclidean and Mahalanobis distance never beat Eq. 3, because without normalization patches with a large norm dominate the search. Truncated PCA-whitening beats it in one cell. CSLS beats it in 42 cells alone and in 120 on top of whitening: the two fix different problems (anisotropy vs. hubness) and add up.

Geometric matching, mean recall (ScanNet Recall@10px, NAVI Recall@2cm at 30–60°, both with a ratio test):

cosEq. 3 (paper)Euclid.Mahal.PCA-w 64PCA-w 256CSLSCSLS + Eq. 3
ScanNet, encoders29.135.625.526.423.632.824.927.3
ScanNet, raw DiTs26.341.823.630.015.634.920.532.8
ScanNet, post-norm DiTs38.941.433.331.130.838.531.231.7
NAVI, encoders54.658.652.853.349.155.552.454.4
NAVI, raw DiTs47.364.946.555.436.858.144.961.0
NAVI, post-norm DiTs62.964.759.058.153.660.960.361.0

Eq. 3 beats cosine on 44 of 46 readouts on ScanNet and 45 of 46 on NAVI, and no other measure beats Eq. 3 in more than 5 of 92 cells. CSLS lowers recall here, likely because the ratio test already rejects targets that match many queries.

In the paper: we will add these comparisons, keep Eq. 3 as the default, and mention CSLS as an add-on for semantic matching. If you had other measures in mind, please let us know which ones and we will add them.

AuthorsOpenReview ↗

Response to Reviewer Euof (3/3)

Q2: Table 2 shows that diagonal covariance normalization already achieves most of the improvement. Is full covariance whitening necessary, considering its additional computational and memory costs?

Short answer. For encoders, full whitening adds another 1.6 to 2.2 points over diagonal scaling, while for raw DiTs diagonal scaling already gives almost the whole gain. In both cases the cost is small, at most 1.5% of the forward pass.

Mean gain over cosine, as in Table 2:

SOC diagSOC fullCC diagCC fullSPair diagSPair full
Encoders (26)+2.05+3.67+2.73+4.94+2.80+4.48
Raw DiTs (10)+16.23+16.65+18.45+18.82+18.49+18.33
Post-norm DiTs (10)+1.01+1.43+1.33+2.00+1.30+1.67

Cost per 800-px image on one GPU:

DINOv2-B (d = 768)FLUX.1 (d = 3072)Krea2 (d = 6144)
Backbone forward pass92 ms545 ms548 ms
Diagonal scaling0.08 ms0.21 ms0.39 ms
Full whitening0.21 ms2.7 ms8.4 ms
Σ^(−1/2), computed once on CPU37 ms1.3 s11 s
Stored transform (fp32)2.3 MB36 MB144 MB

The transform is computed once per readout, offline, and reused for every test image; at test time full whitening is one matrix multiplication. Diagonal scaling stores only d numbers (3 to 25 KB). We will keep full whitening as the default, which is also fairer to vision encoders, whose channels are more correlated, and recommend diagonal scaling as a cheaper option for raw DiTs.


Q3: How sensitive are the results to the distribution and size of the unlabeled images used to estimate the whitening statistics, particularly under significant domain shifts?

Short answer. Not very sensitive. Moments from very different images, or from only 10 images, keep most of the gain. One model, VGGT, is the exception.

Table 10 of the paper varies the number of ImageNet images. We now also estimated μ and Σ from ScanNet indoor frames and FFHQ face photos. Mean SPair-71k PCK over the 46 readouts after whitening (cosine, which uses no moments, gives 37.1; for 50 and 10 images we average 3 disjoint draws):

Images used for μ and ΣImageNetScanNet framesFFHQ faces
25043.9 (paper's setting)43.943.6
5043.843.843.5
1043.443.543.2
  • Ten face photos still give 43.2, keeping 6.1 of the 6.8-point gain. Different draws differ little (std 0.06–0.23).
  • VGGT is the sensitive case: with 250 face photos it loses up to 9.1 PCK compared with ImageNet moments. Every other readout loses at most 2.6 (ScanNet frames) and 1.7 (faces) over SOC, CC and SPair.
  • The paper's benchmarks are already far from ImageNet (animals in AP-10K, indoor scenes in ScanNet, single objects in NAVI, video in TAP-Vid), and all of them use ImageNet moments. Estimating μ and Σ from the test image itself, with no external images, keeps a median 84% of the gain (Table 3).

We will add the table to the appendix.

We hope these results address your concerns on novelty and on alternative metrics. If they do, we would be grateful if you would consider raising your score.

I appreciate the authors' detailed rebuttal and additional experiments, which address most of my concerns. After carefully reading the rebuttal and other reviewers’ comments, I will raise the final rating to 3.

Reviewer_Q7CmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how the similarity metric affects zero-shot evaluation of frozen vision features. It estimates feature moments on unlabeled images and applies shrinkage whitening before the same nearest-neighbor matching rule. Across 46 readouts and 11 dense evaluations, the authors report gains in 459 of 506 cases. The largest gains come from raw diffusion-transformer features. They recommend reporting corrected results alongside cosine results.

Critical strengths
  1. Broad, controlled experiments. The study compares the same frozen features under two metrics, including the normalization and channel-removal baseline from DiTF. Mean gains across the dense evaluations are +14.0 points for raw DiTs, +3.6 for vision encoders and +1.4 for post-normalized DiTs. FLUX.1 raw features rise from 16.8 to 61.8 on SPair-71k (Table 1). The gains vary substantially, but are not confined to untreated raw features.
  2. Useful ablations and disclosed failures. Table 2 separates centering, dominant-component removal, diagonal scaling, and full whitening. Tables 6–7 test fixed parameters and transfer from one validation set. The paper also reports small-data and per-image estimates, task-dependent gains, and failure cases. These controls give the evaluation study value beyond the already established whitening method.
Critical weaknesses
  1. The mechanism and scope of the gains need clearer limits. The readout comparison changes normalization and removes channels, so it does not isolate effective dimension. On SOCO, diagonal standardization already recovers a median 90% of the full gain, including 99% for raw DiTs (Table 2). Table 8 changes readout rankings without changing the leading model family. Given the related findings in prior work, the paper needs to explain what new understanding follows beyond correcting poorly scaled features. Aggregate true-match margins and matched distractor distributions would be more informative than one selected query.
Questions for authors
  1. Can you compare centering, diagonal scaling and full whitening on a fixed readout, reporting true-match margins and distractor distributions under the same sampling protocol? Figure 11 shows improved separation but no reduced distractor spread; it does not directly test the centered null model.
  2. Can you summarize gains and ranks by backbone and dataset family, alongside readout-level results, and measure the overhead of diagonal versus full whitening? This would show where the full transform adds enough value to justify its cost.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Q7Cm (1/3)

Thank you for your valuable feedback, which helps us improve our paper. We detail our response below; please let us know if it addresses your concerns.


W1: The mechanism and scope of the gains need clearer limits. The readout comparison changes normalization and removes channels, so it does not isolate effective dimension.

Short answer. We separated the two factors, and we also changed the effective dimension d_eff alone (the participation ratio (Tr Σ)² / Tr(Σ²) of the feature covariance, roughly the number of directions that carry the variance). Lowering d_eff alone hurts matching in all 46 readouts, and whitening helps mainly when d_eff is below about 250.

Separating the two factors. The paper's post-norm readout follows DiTF (Gan et al., 2025): it reads after the block's norm layer and also discards a few massive-activation channels. We ran the two missing combinations for all 10 DiTs. PCK on SOC, the semantic object correspondence task of SOCO (Dünkel et al., 2026), the largest and newest semantic correspondence benchmark (ECCV 2026); cosine → whitened with Eq. 3 (subtract μ, multiply by Σ_α^(−1/2), the inverse square root of the covariance with shrinkage α), d_eff in parentheses:

Modelraw (paper)raw + discardpost-normpost-norm + discard (paper)
Qwen-Image32.5 → 45.1 (94)41.3 → 45.1 (158)43.1 → 43.4 (260)43.2 → 43.0 (309)
GLM-Image19.9 → 38.9 (65)19.2 → 38.9 (81)34.6 → 39.1 (192)34.7 → 39.1 (208)
FLUX.2-klein58.6 → 62.8 (65)61.9 → 62.8 (350)63.0 → 63.2 (407)62.4 → 63.2 (247)
PixArt-α28.9 → 44.9 (55)32.0 → 44.8 (56)43.4 → 45.4 (145)43.4 → 45.4 (162)
Krea228.9 → 47.8 (53)29.2 → 47.5 (72)46.2 → 45.9 (216)46.3 → 45.8 (258)
SD3.5-L34.3 → 52.0 (49)36.8 → 51.7 (62)52.1 → 53.1 (292)52.9 → 53.3 (341)
ERNIE-Image38.5 → 55.7 (21)39.5 → 55.5 (20)49.0 → 55.3 (169)48.5 → 55.4 (185)
HiDream-O122.5 → 38.8 (16)23.6 → 38.7 (16)42.1 → 43.0 (168)42.8 → 43.1 (174)
FLUX.113.4 → 54.5 (3)20.3 → 54.4 (93)53.0 → 54.6 (249)54.0 → 54.4 (289)
Z-Image55.0 → 58.5 (2)58.3 → 56.9 (278)58.7 → 58.5 (313)58.6 → 58.4 (374)
  • Reading after the norm layer is the main factor: it raises d_eff in all ten models and cosine PCK by 3.7 to 39.6. Discarding channels helps most in the four models where it raises d_eff a lot (Qwen-Image, FLUX.2-klein, FLUX.1, Z-Image: +3.3 to +8.8); in the other six it changes cosine PCK by −0.7 to +3.1.
  • Within a model, the readout with the higher d_eff has the higher cosine PCK in 53 of 59 pairs. After whitening the four readouts are 1.4 points apart on average, against 15.6 under cosine.

Changing d_eff alone. To isolate d_eff, we change it while keeping the readout, the images, the matching rule and every channel's variance fixed. We start from each readout's whitened features, which have the same variance in every direction. We then multiply every feature by one fixed matrix R D Rᵀ: R is a random rotation, and D = diag(1, 2^(−s), …, d^(−s)) scales the k-th random direction by k^(−s), so its variance becomes k^(−2s). For s > 0 the variance concentrates in the first few directions and d_eff falls, from a median of 562 at s = 0.125 to 2.5 at s = 1 (six values of s). Because the random directions are not aligned with the channels, every channel keeps about the same variance, and cosine ignores the overall scale. If only per-channel scale mattered, PCK would not change. Instead it drops in all 46 readouts on SPair-71k, SOC, CC (SOCO's concept correspondence task), ScanNet and NAVI, and the drop follows d_eff. Median PCK loss on SPair-71k:

d_eff after the change≥ 500250–500150–25050–15010–50< 10
Median PCK loss0.20.80.73.813.332.0

Scope. The same threshold shows up in the paper's results. All 506 readout × benchmark cells, grouped by the readout's d_eff:

d_effReadoutsCells improvedMedian gain
below 5016174/176+3.8
50 to 15015158/165+4.4
150 to 250994/99+1.1
above 250633/66+0.0

Limit we will add to the paper: above d_eff ≈ 250 whitening neither helps nor hurts (worst case −1.8), and this group holds 33 of the 47 cells that do not improve. d_eff needs only unlabeled images, so this can be checked before any evaluation.

AuthorsOpenReview ↗

Response to Reviewer Q7Cm (2/3)

W1: On SOCO, diagonal standardization already recovers a median 90% of the full gain, including 99% for raw DiTs (Table 2).

Short answer. Yes for raw DiTs, but not for encoders. We keep full whitening as the default, so that encoders and DiTs are evaluated with the same metric, and will recommend diagonal scaling as a cheaper option for raw DiTs.

Mean gain over cosine, as in Table 2:

SOC diagSOC fullCC diagCC fullSPair diagSPair full
Encoders (26)+2.05+3.67+2.73+4.94+2.80+4.48
Raw DiTs (10)+16.23+16.65+18.45+18.82+18.49+18.33
Post-norm DiTs (10)+1.01+1.43+1.33+2.00+1.30+1.67

In raw DiTs a few channels carry most of the variance, so rescaling them fixes almost everything. Encoders have fairly even channel variances, and there removing correlations adds 1.6 to 2.2 points (full whitening is better for 19 of 26 encoders on each benchmark).


W1: Table 8 changes readout rankings without changing the leading model family. Given the related findings in prior work, the paper needs to explain what new understanding follows beyond correcting poorly scaled features.

Short answer. We agree that the leading family does not change, and we will state the claim as a change in rankings. Beyond correcting poorly scaled features, the study gives three new insights.

  1. The cause is low d_eff; poor channel scale is one way to get there. Prior NLP work (Timkey & Van Schijndel, 2021) blames a few rogue channels. Eq. 2 is more general: the cosine of a wrong match spreads by about 1/√d_eff. Two results above show this: lowering d_eff with equal channel scales hurts all 46 readouts, and in encoders, whose channel scales are even, removing correlations still helps.
  2. Rankings change. Kendall τ between the cosine and whitened rankings is 0.63–0.77 per benchmark, so 12–18% of readout pairs swap order, and the top readout changes on 4 of 11 benchmarks.
  3. Whitening unlocks representations that cosine hides, across models and across blocks. Raw DiT features look weak under cosine; prior work changes the readout to fix them (DiTF). With the same features and whitening, they average 48.8 over our 11 benchmarks, above the 26 encoders (41.2); raw FLUX.1, for example, moves from 36th to 2nd on ScanNet. The same holds within a model. For this response we ran all 32 blocks of FLUX.2-klein raw and 24 of the 36 blocks of ERNIE-Image raw on the SPair-71k test split, plus every block of DINOv2-B and DINOv3-B (blocks around the peaks shown; blocks 11 and 7 are the paper's numbers):
FLUX.2 block8910111213141516171819
cosine53.055.555.854.552.648.444.638.934.130.830.338.2
whitened56.559.160.562.964.565.165.164.763.863.162.662.3
ERNIE block6789101112131415161718
cosine39.638.236.334.233.932.732.034.737.341.745.344.644.7
whitened57.258.657.756.155.254.152.853.754.656.357.756.254.3

Interestingly, whitening also changes which block looks best. On the SPair-71k test split, FLUX.2-klein under cosine peaks at block 10 (55.8) and falls to 30.3 by block 18, so the deeper blocks look unusable; after whitening every block improves, the deeper blocks become the best (65.1 at blocks 13–14), and blocks 11–19 are all within 3 points. ERNIE-Image under cosine is highest at block 16 (45.3), far from the block chosen on validation (7, 38.2); after whitening block 7 is the best (58.6) and blocks 6–18 are within 6 points. So whitening raises every block, flattens the curve, and makes the block choice much less critical. This is consistent with the paper: there we chose each DiT's block by cosine on the SPair-71k validation split (Table 4) and kept it, so the paper's whitened numbers are conservative (FLUX.2-klein: 62.9 at block 11, 65.1 at its best block). Even at each metric's own best block, whitening gains +9.3 (FLUX.2-klein) and +13.3 (ERNIE-Image). The encoders behave differently; their last block is best under both metrics:

Block01234567891011
DINOv2-B, cosine5.28.610.313.319.928.329.039.046.752.756.156.5
DINOv2-B, whitened6.08.711.015.323.131.036.143.648.153.456.557.7
DINOv3-B, cosine4.88.410.614.722.229.438.245.849.854.055.555.7
DINOv3-B, whitened5.37.89.915.223.430.639.547.351.355.457.858.8

This is why we recommend reporting whitened scores next to cosine.

AuthorsOpenReview ↗

Response to Reviewer Q7Cm (3/3)

W1 / Q1: Aggregate true-match margins and matched distractor distributions would be more informative than one selected query. Can you compare centering, diagonal scaling and full whitening on a fixed readout, reporting true-match margins and distractor distributions under the same sampling protocol? Figure 11 shows improved separation but no reduced distractor spread; it does not directly test the centered null model.

Short answer. Over all keypoints the distractor spread does shrink at each step (centering → diagonal → full) and matches the null model's prediction of 1/√d_eff. Fig. 11's single query missed this; we will replace it with these statistics.

Setup: every annotated keypoint, all 46 readouts, same pairs for every metric; the medians and the null-model test pool SOC and SPair-71k. Distractors: patches outside the PCK radius. Margin = (true-match score − distractor mean) / distractor std. The GT-patch rank stays above 1 even for correct matches, because neighbouring patches in the PCK radius are near-duplicates.

Two fixed readouts on SOC (medians over 149,646 keypoints):

ReadoutMetricDistractor meanDistractor std ↓Margin ↑Rank of GT patch ↓Best rank in PCK radius ↓
FLUX.1 rawcosine0.860.0200.942410
centered0.000.0821.61497
diagonal0.020.0524.471
full0.010.0235.471
DINOv2-Bcosine0.210.1044.161
centered0.080.1064.761
diagonal0.080.1034.861
full0.070.0965.061

For FLUX.1 raw, cosine scores every wrong patch near 0.86, so the true match ranks 424th; its std is small only because all scores sit near this offset. Centering removes the offset, diagonal scaling narrows the distractors, and full whitening halves their std (0.052 → 0.023), giving the largest margin. DINOv2-B (higher d_eff) changes little.

Medians across readouts:

cosinecentereddiagonalfull
Distractor std ↓, raw DiTs0.0630.1300.0670.039
Distractor std ↓, encoders0.0790.1170.0850.054
Margin ↑, raw DiTs1.11.93.64.6
Margin ↑, encoders2.32.42.83.4

Directly testing the centered null model (Eq. 2). It treats the query and a wrong patch as independent draws from the feature distribution (mean μ, covariance Σ, estimated on ImageNet). From μ and Σ alone it predicts the scores of wrong matches: under cosine a shared offset ‖μ‖² / (‖μ‖² + Tr Σ), and after each correction a mean of 0 and a std of 1/√d_eff of the corrected features' covariance. We compare them with the measured values in 92 cells (46 readouts × SOC and SPair-71k). The predicted offset matches (Spearman 0.94). For the std (medians over cells):

AfterCovariance after the stepPredicted std ↓ (1/√d_eff)Measured std ↓Measured / predicted, median (IQR)
centeringΣ0.1160.1091.00 (0.82–1.21)
diagonalD^(−1/2) Σ D^(−1/2), D = per-channel variances0.0730.0751.03 (0.89–1.21)
fullΣ_α^(−1/2) Σ Σ_α^(−1/2)0.0410.0501.19 (1.08–1.35)

D and Σ_α include the shrinkage α of Eq. 3. After full whitening the measured std is 19% above the prediction but still the lowest (below centering in 80 of 80 cells with α < 1).


Q2: Can you summarize gains and ranks by backbone and dataset family, alongside readout-level results, and measure the overhead of diagonal versus full whitening? This would show where the full transform adds enough value to justify its cost.

Short answer. Raw DiTs gain most (+4.6 to +16.5) and move up 6.5–13.2 places among the 46 readouts; encoders gain +0.1 to +5.3; post-norm DiTs gain least. The overhead is at most 1.5% of the forward pass.

Mean gain (improved cells) and mean rank change (positive = up, range over 11 benchmarks); per-readout: Table 9.

FamilySemanticGeometricTrackingGeom.-awareRank change
Encoders+4.0 (182/182)+5.3 (50/52)+1.2 (26/26)+0.1 (15/26)−1.3 to −3.7
Raw DiTs+15.7 (70/70)+16.5 (20/20)+6.9 (10/10)+4.6 (10/10)+6.5 to +13.2
Post-norm DiTs+1.5 (47/70)+2.0 (18/20)+0.3 (8/10)−0.1 (3/10)−3.0 to −5.8

Cost: μ and Σ^(−1/2) are computed once per readout, offline, from 250 unlabeled images (37 ms for DINOv2-B and 11 s for Krea2 on CPU), and reused for every test image. At test time it is one matrix multiplication per image: 0.21 ms for DINOv2-B (0.23% of the forward pass) and 8.4 ms for Krea2 (d = 6144, 1.5%); diagonal scaling takes 0.08 to 0.39 ms. Full whitening is worth this cost for encoders (+1.6 to 2.2 over diagonal); for raw DiTs, diagonal scaling suffices.


Flag For Ethics Review: Yes

Only public benchmarks and released models are used; no new data or human subjects.

I appreciate the authors detailed reply. After reading it carefully, I've decided to retain my original positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their time and helpful feedback. Their questions led to new experiments that made the paper stronger, and we are very grateful that all three acknowledged our response and gave a positive final rating. The paper shows that cosine similarity, the standard score in zero-shot benchmarks, is unfair to features whose variance sits in a few directions (low effective dimension). We explain why, and show that whitening with statistics from unlabeled images fixes it. On 11 large-scale benchmarks for semantic and geometric correspondence and video tracking, plus ImageNet k-NN classification, it improves nearly all evaluations over 46 readouts, by up to +48 points, and changes how models rank. The reviewers liked the clear explanation (ajuC), the large, controlled experiments (Q7Cm, Euof, ajuC), the ablations and reported failures (Q7Cm), and the effect on rankings (Euof). The main concern was novelty, since whitening is well known, and what we learn beyond fixing badly scaled channels (Q7Cm, Euof, ajuC). We clarified that our contribution is about evaluation, and new experiments show that a low effective dimension hurts matching even when all channels have the same scale (Q7Cm, ajuC). Other new results cover the mechanism and scope of the gains and a direct test of the null model (Q7Cm), other similarity measures (Euof), diagonal vs. full whitening and its cost (Q7Cm, Euof), and robustness to the image statistics and to noise seeds in DiTs (Euof, ajuC). An especially interesting new finding: on FLUX.2-klein and ERNIE-Image, whitening improves every block we tested and peaks at a different block than cosine, so it reveals representations that cosine hides. On FLUX.2-klein the best whitened block reaches 65.1 on SPair-71k, against 62.9 for the block the paper chose with cosine. We think these results support reporting whitened cosine next to cosine, and we will add all of them to the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.