- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that cosine similarity, the standard metric for zero-shot evaluation of frozen vision features, is sensitive to how feature variance is distributed across directions. The proposed fix whitens features using statistics estimated from unlabeled images before computing similarity. This simple correction consistently improves performance on various benchmarks.
Critical strengths
- The paper gives a clear explanation for why cosine similarity can perform poorly when feature variance is concentrated in a few directions.
- The evaluation is large-scale and carefully controlled, covering many models and benchmarks.
Critical weaknesses
- The methodological contribution is somewhat limited. Whitening and per-channel standardization are already established in retrieval and NLP, and much of the gain appears to come from simple per-channel rescaling.
- Each diffusion-transformer evaluation uses a single noise sample, and the paper does not report how stable the large gains are across noise draws.
Questions for authors
- Could the authors discuss how the effective-dimension account relates to simpler explanations based on per-channel scale?
- Do the large diffusion-transformer gains persist across multiple noise samples?
4 comments
Response to Reviewer ajuC (2/3)
Q1: Could the authors discuss how the effective-dimension account relates to simpler explanations based on per-channel scale?
Short answer. Per-channel scale is one way to get a low d_eff, and for raw DiTs it is the main one. But d_eff also explains three things that channel scale cannot: lowering d_eff with equal channel scales still hurts, removing correlations still helps encoders, and the matching noise follows d_eff at every step. We see per-channel scale as a special case of the d_eff account and will present it that way.
1. Lowering d_eff with equal channel scales. To isolate d_eff, we change it while keeping the readout, the images, the matching rule and every channel's variance fixed. We start from each readout's whitened features, which have the same variance in every direction. We then multiply every feature by one fixed matrix R D Rᵀ: R is a random rotation, and D = diag(1, 2^(−s), …, d^(−s)) scales the k-th random direction by k^(−s), so its variance becomes k^(−2s). For s > 0 the variance concentrates in the first few directions and d_eff falls, from a median of 562 at s = 0.125 to 2.5 at s = 1 (six values of s). Because the random directions are not aligned with the channels, every channel keeps about the same variance, and cosine ignores the overall scale. If only per-channel scale mattered, PCK would not change. Instead it drops in all 46 readouts on SPair-71k, SOC, CC, ScanNet and NAVI (SOC and CC: the semantic object and concept correspondence tasks of SOCO (Dünkel et al., 2026), the largest and newest semantic correspondence benchmark, ECCV 2026), and the drop follows d_eff. Median PCK loss on SPair-71k:
| d_eff after the change | ≥ 500 | 250–500 | 150–250 | 50–150 | 10–50 | < 10 |
|---|---|---|---|---|---|---|
| Median PCK loss | 0.2 | 0.8 | 0.7 | 3.8 | 13.3 | 32.0 |
The paper's own results show the same threshold. All 506 readout × benchmark cells, grouped by the readout's d_eff:
| d_eff | Readouts | Cells improved | Median gain |
|---|---|---|---|
| below 50 | 16 | 174/176 | +3.8 |
| 50 to 150 | 15 | 158/165 | +4.4 |
| 150 to 250 | 9 | 94/99 | +1.1 |
| above 250 | 6 | 33/66 | +0.0 |
2. Encoders need more than rescaling. Mean gain over cosine of diagonal (per-channel) scaling and full whitening, as in Table 2:
| SOC diag | SOC full | CC diag | CC full | SPair diag | SPair full | |
|---|---|---|---|---|---|---|
| Encoders (26) | +2.05 | +3.67 | +2.73 | +4.94 | +2.80 | +4.48 |
| Raw DiTs (10) | +16.23 | +16.65 | +18.45 | +18.82 | +18.49 | +18.33 |
| Post-norm DiTs (10) | +1.01 | +1.43 | +1.33 | +2.00 | +1.30 | +1.67 |
In raw DiTs a few channels carry most of the variance, so rescaling fixes almost everything. Encoders have fairly even channel variances, yet removing correlations adds another 1.6 to 2.2 points. Across the 46 readouts, the spread of channel variances predicts the gain of diagonal scaling (Spearman +0.64 on SPair) but not the extra gain of full whitening (−0.15); the d_eff of the correlation matrix predicts the extra gain (−0.69).
3. The matching noise follows d_eff at every step. Eq. 2 comes from a null model that treats the query and a wrong patch as independent draws from the feature distribution (mean μ, covariance Σ, estimated on ImageNet). From Σ alone it predicts that, after each correction, the scores of wrong matches have a std of 1/√d_eff of the corrected features' covariance. We measured this std directly: for every annotated keypoint of SOC and SPair-71k, the std of the scores of all wrong patches (outside the PCK radius), median over keypoints. Over 92 cells (46 readouts × 2 benchmarks; medians over cells):
| After | Covariance after the step | Predicted std ↓ (1/√d_eff) | Measured std ↓ | Measured / predicted per cell, median (IQR) |
|---|---|---|---|---|
| centering | Σ | 0.116 | 0.109 | 1.00 (0.82–1.21) |
| diagonal | D^(−1/2) Σ D^(−1/2), D = per-channel variances | 0.073 | 0.075 | 1.03 (0.89–1.21) |
| full | Σ_α^(−1/2) Σ Σ_α^(−1/2) | 0.041 | 0.050 | 1.19 (1.08–1.35) |
D and Σ_α include the shrinkage α of Eq. 3. After diagonal scaling every channel has the same scale, yet the measured noise still matches 1/√d_eff of the correlation matrix D^(−1/2) Σ D^(−1/2). So what sets the noise is d_eff, and channel scale is one part of it. In practice we keep full whitening as the default and recommend diagonal scaling as a cheaper option for raw DiTs.
Response to Reviewer ajuC (3/3)
W2 / Q2: Each diffusion-transformer evaluation uses a single noise sample, and the paper does not report how stable the large gains are across noise draws. Do the large diffusion-transformer gains persist across multiple noise samples?
Short answer. Yes. Across 5 noise seeds the gain of whitening varies by a standard deviation of about 0.1 PCK and never changes sign.
We re-ran all 21 noise-dependent readouts (the 20 DiT readouts and DIFT at t = 250) with 5 noise seeds. Each seed is a full evaluation with one noise sample per image, as in the paper, and μ and Σ were re-estimated for each seed, so both sides of the comparison see new noise. Gain of whitening over cosine, mean ± std over the 5 runs:
| FLUX.1 | SD3.5-L | PixArt-α | ERNIE-Image | GLM-Image | Krea2 | HiDream-O1 | Qwen-Image | FLUX.2-klein | Z-Image | DIFT (t = 250) | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SOC | +41.0 ± 0.11 | +17.7 ± 0.09 | +16.0 ± 0.17 | +17.2 ± 0.14 | +19.0 ± 0.08 | +19.0 ± 0.13 | +16.5 ± 0.17 | +12.8 ± 0.14 | +4.2 ± 0.11 | +3.3 ± 0.13 | +3.2 ± 0.14 |
| CC | +48.5 ± 0.13 | +20.3 ± 0.11 | +18.8 ± 0.13 | +21.0 ± 0.07 | +22.3 ± 0.14 | +19.5 ± 0.13 | +17.5 ± 0.19 | +12.1 ± 0.12 | +5.2 ± 0.09 | +3.2 ± 0.14 | +4.8 ± 0.13 |
Over all 21 readouts the median std is 0.11 (SOC) and 0.09 (CC), the largest is 0.19, and the sign of the gain never changes (0 of 42 cases).
The paper uses one noise sample per image so that DiTs, like encoders, get a single forward pass per image; the table shows that this choice does not affect the conclusions. We will add it to the appendix.
Thank you for the detailed response and the additional experiments. The response and new experiments have addressed my concerns, so I will maintain my positive score.
Response to Reviewer ajuC (1/3)
Thank you for your valuable feedback, which helps us improve our paper. We detail our response below; please let us know if it addresses your concerns.
Short answer. We agree on both points, and the paper says so (Sections 2 and 6): whitening and per-channel standardization are established, we do not claim them, and for raw DiTs most of the gain is per-channel rescaling (Q1 below shows where this stops being true). Our contribution is about how frozen features are evaluated, in three parts.
Interestingly, whitening also changes which block looks best. On the SPair-71k test split, FLUX.2-klein under cosine peaks at block 10 (55.8) and falls to 30.3 by block 18, so the deeper blocks look unusable; after whitening every block improves, the deeper blocks become the best (65.1 at blocks 13–14), and blocks 11–19 are all within 3 points. ERNIE-Image under cosine is highest at block 16 (45.3), far from the block chosen on validation (7, 38.2); after whitening block 7 is the best (58.6) and blocks 6–18 are within 6 points. So whitening raises every block, flattens the curve, and makes the block choice much less critical. This does not contradict the paper: there we chose each DiT's block by cosine on the SPair-71k validation split (Table 4) and kept it for whitening, so the paper's whitened numbers are conservative (FLUX.2-klein: 62.9 at block 11, 65.1 at its best block). Even at each metric's own best block, whitening gains +9.3 (FLUX.2-klein) and +13.3 (ERNIE-Image). The encoders behave differently; their last block is best under both metrics:
We therefore recommend reporting whitened scores next to cosine.