- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies why intramodal retrieval with contrastively aligned models like CLIP is bad. The paper analyzes a phenomenon often called "hubness" in the literature, the phenomenon that all embeddings sharing a modality share a common average component that adds nose to the similarity score. The authors propose a method that estimates "conformity" (alignment between an embeddings alignment with a reference mean) and uses the leftover direction to be rescaled and used for retrieval.
Critical strengths
The identified hubness property continues to plague intramodal retrieval on modern backbones, and the authors frame how the "conformity" property impacts scoring in a coherent way. The proposed approach is relatively simple and easy to implement across general pipelines for embedding-based search.
Critical weaknesses
The paper's metric of conformity is a well-known property of contrastively aligned models usually referred to as hubness. Much of the paper's discussion about this can be reduced to additional related work. See: Radovanovic, Milos, Alexandros Nanopoulos, and Mirjana Ivanovic. "Hubs in space: Popular nearest neighbors in high-dimensional data." Journal of machine learning research 11.sept (2010): 2487-2531. Suzuki, Ikumi, et al. "Centering similarity measures to reduce hubs." Proceedings of the 2013 conference on empirical methods in natural language processing. 2013.
The first stage mechanism especially seems like: Wang, Yan, et al. "Simpleshot: Revisiting nearest-neighbor classification for few-shot learning." arXiv preprint arXiv:1911.04623 (2019).
The novel angular shrinkage term should additionally be further justified. The shrinkage schedule seems arbitrary, and it is not clarified why the clamping is introduced empirically. It can also be compared to other training-free baselines: Jégou, Hervé, and Ondřej Chum. "Negative evidences and co-occurences in image retrieval: The benefit of PCA and whitening." European conference on computer vision. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012 Mu, Jiaqi, Suma Bhat, and Pramod Viswanath. "All-but-the-top: Simple and effective postprocessing for word representations." arXiv preprint arXiv:1702.01417 (2017). Radenović, Filip, Giorgos Tolias, and Ondřej Chum. "Fine-tuning CNN image retrieval with no human annotation." IEEE transactions on pattern analysis and machine intelligence 41.7 (2018): 1655-1668.
Questions for authors
If the conformity component is a constant bias term, why does removing it degrade performance on most datasets?
Does the second step still work with a simpler schedule?
Is there a reason some metric of gallery element popularity isn't reported to measure hubness directly?
10 comments
Response to Reviewer hhoC [2/5]
W2: Relationship to SimpleShot and Mean-centering
Weakness (W2). The first stage mechanism especially seems like: Wang, Yan, et al. "Simpleshot: Revisiting nearest-neighbor classification for few-shot learning." arXiv preprint arXiv:1911.04623 (2019).
Answer (W2). We agree that mean-centering followed by L2 normalization is an established operation, as used in SimpleShot. We do not claim this preprocessing step as novel and will explicitly acknowledge SimpleShot in the revised paper. Our contribution is the geometric analysis connecting conformity to retrieval bias and the resulting progressive correction framework. Specifically, we (i) exactly characterize conformity's contribution to retrieval similarity (Eq. 4), (ii) identify residual conformity that persists after centering and renormalization, and (iii) propose CGD to progressively correct both global and residual conformity.
The residual correction further improves retrieval from 35.77 to 39.50 mAP on T2T and 40.50 to 41.23 on I2I (Table 10). We further justify the angular correction and its schedule in W3.
Response to Reviewer hhoC [3/5]
W3: Angular Shrinkage, Its Schedule, and Numerical Clamping
Weakness (W3). The novel angular shrinkage term should additionally be further justified. The shrinkage schedule seems arbitrary, and it is not clarified why the clamping is introduced empirically.
Answer (W3). We thank the Reviewer for this important question. Our angular correction is motivated by three considerations: (i) Semantic preservation: the residual direction contains useful semantic information, motivating partial rather than complete removal; (ii) Geometric motivation: angular contraction attenuates residual conformity while preserving the orthogonal direction and unit norm; and (iii) Empirical validation: the specific retention schedule is heuristic, and comparisons with simpler alternatives support its effectiveness.
(i) Why partial correction? Hard removal assumes that all information along the residual conformity direction is undesirable. However, this direction may also encode useful class-discriminative information.
To investigate this, we conduct a label-based diagnostic on three T2T and three I2I datasets, decomposing each sample's residual projection into a class-mean component, capturing systematic differences between classes, and a within-class residual, capturing variations within each class. We then evaluate the effect of retaining each component separately, compared with completely removing the residual direction.
Table 12. Semantic decomposition of the residual direction (mAP, %).
| Correction | T2T ↑ | I2I ↑ |
|---|---|---|
| Hard Removal | 40.99 | 40.06 |
| Keep Class-Mean Component Only | 42.61 | 42.47 |
| Keep Within-Class Residual Only | 38.29 | 39.86 |
Retaining the class-mean component consistently outperforms hard removal (42.61 vs. 40.99 on T2T; 42.47 vs. 40.06 on I2I), whereas retaining only the within-class residual performs worse. These findings reveal that the residual conformity direction also carries useful class-related information. This motivates our partial angular contraction.
(ii) Why angular contraction and this schedule? Our analysis identifies residual conformity as a direction on the unit sphere. Angular contraction naturally reduces projections along this direction while preserving the orthogonal direction and unit norm.
We do not claim that our specific schedule t_i=(1-p_i)/2 is theoretically optimal. In retrieval, establishing a universally optimal correction is challenging because both representations and relevance judgments are approximations. Many existing methods therefore exploit geometric properties to improve retrieval through empirically validated corrections, including ABTT [ICLR 2018], RootSIFT [CVPR 2012], CSLS [ICLR 2018], and QB-Norm [CVPR 2022]. Similarly, CGD's correction direction follows from geometric analysis, while its attenuation strength is an empirically validated design choice.
To verify that the gains do not depend on this exact schedule, we compare fixed retention ratios of 1/2 and 1/4.
Table 13. T2T: Fixed vs. adaptive angular retention (mAP, %).
| Dataset | Raw | Stage 1 | Fixed 1/2 | Fixed 1/4 | Adaptive (Ours) |
|---|---|---|---|---|---|
| COCO Text | 26.20 | 27.94 | 28.33 | 28.59 | 29.20 |
| Flickr30K | 51.68 | 50.82 | 55.47 | 55.35 | 56.08 |
| NoCaps | 35.07 | 37.66 | 39.02 | 38.96 | 39.63 |
Table 14. I2I: Fixed vs. adaptive angular retention (mAP, %).
| Dataset | Raw | Stage 1 | Fixed 1/2 | Fixed 1/4 | Adaptive (Ours) |
|---|---|---|---|---|---|
| EuroSAT | 47.93 | 48.89 | 50.09 | 50.21 | 50.42 |
| Oxford Pets | 30.53 | 31.84 | 32.98 | 32.86 | 33.19 |
| Food101 | 32.32 | 33.21 | 34.08 | 34.20 | 34.41 |
| Stanford Cars | 24.60 | 24.95 | 25.39 | 25.27 | 25.60 |
All experiments use CLIP ViT-B/32.
Both fixed schedules improve retrieval, while the adaptive schedule performs best across all seven datasets. This supports the effectiveness of angular correction without requiring one theoretically optimal retention rule.
(iii) Numerical clamping. Since theoretically, we use a margin of solely to prevent floating-point errors in and . This margin does not affect the contraction schedule.
Response to Reviewer hhoC [1/5]
We sincerely thank the Reviewer for the insightful feedback, particularly for highlighting the connection between conformity and classical hubness, which we had not sufficiently explored. Motivated by this observation, we conducted additional analyses and found that conformity and hubness are only partially correlated, while changes in overall hubness cannot fully explain CGD's retrieval gains. These findings help clarify the relationship to prior work and strengthen our analysis. We address W1–W4 first, followed by Q1–Q3.
W1: Conformity, hubness, and gallery popularity
Answer (W1).We thank the Reviewer for highlighting the connection between conformity and hubness, which motivated us to investigate their relationship more closely.Conformity and hubness are related but measure different properties. Following Levi and Gilboa [ICML 2025], conformity is defined as the average cosine similarity of an instance to the other instances in a representative dataset, while hubness measures how frequently a gallery sample appears among the nearest neighbors of different queries. We acknowledge the relevant prior work on hubness (Radovanović et al. [JMLR 2010]) and centered similarity (Suzuki et al. [EMNLP 2013]).
We also want to clarify that Stage 1 uses established mean-centering followed by L2 normalization (as in SimpleShot), which we do not claim as novel. Our contribution includes the exact characterization of conformity's query-dependent contribution to retrieval similarity (Eq. 4) and the identification and correction of residual conformity after Stage 1 (see W2).
Following the Reviewer's suggestion, we further investigate the relationship between conformity, hubness, and retrieval performance. For each representation ( Raw, Stage 1, and full CGD ), we report:
If CGD's gains were primarily explained by reducing aggregate hubness, we would expect retrieval improvements to be accompanied by reductions in skewness and Gini.
Table 10. Conformity and gallery popularity on three T2T and three I2I datasets.
Our results reveal that conformity and gallery popularity are only weakly correlated (ρ = 0.248 on T2T and 0.233 on I2I for Raw embeddings). More importantly, on I2I, Stage 2 improves retrieval from 40.50 to 41.23, while skewness and Gini remain unchanged at the reported precision. This indicates that aggregate hubness reduction alone cannot fully explain CGD's retrieval gains .
We further compare CGD with existing hubness correction methods, QB-Norm [CVPR 2022] and NNN [EMNLP 2024]. Adding CGD improves both methods on T2T (+3.51/+2.29) and I2I (+1.23/+1.73), demonstrating their complementarity ( W4, Table 15 ).