acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Correcting Conformity in Intra-Modal Retrieval

Abstract

Frozen vision–language representations are widely used for intra-modal retrieval, yet their embedding geometry contains systematic shared structures that can distort cosine similarity and degrade ranking quality. We first show that conformity, the tendency of an embedding to be similar to many samples, is exactly determined by the embedding's projection onto a single shared direction defined by the normalized reference mean. We further show that mean correction alone is insufficient: after subtracting the reference mean, the -normalization implicitly imposed by cosine similarity in retrieval gives rise to a distinct residual conformity direction. Motivated by these observations, we propose Conformity-Guided Debiasing(CGD), a training-free geometric correction that attenuates conformity and residual conformity. CGD first applies mean correction and -normalization, and then performs a sample-dependent angular contraction toward the subspace orthogonal to the residual conformity direction. The resulting transform is plug-and-play, requires no backbone modification or query gallery fitting, and can be applied on top of existing intra-modal retrieval methods. Across multiple CLIP-like backbones and image-to-image and text-to-text benchmarks, our method consistently improves retrieval performance, showing that explicitly correcting conformity-related directional structure provides a simple and effective way to refine frozen intra-modal representations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-language models, Intra-modal retrieval, Embedding geometry, CLIP

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_hhoCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why intramodal retrieval with contrastively aligned models like CLIP is bad. The paper analyzes a phenomenon often called "hubness" in the literature, the phenomenon that all embeddings sharing a modality share a common average component that adds nose to the similarity score. The authors propose a method that estimates "conformity" (alignment between an embeddings alignment with a reference mean) and uses the leftover direction to be rescaled and used for retrieval.

Critical strengths

The identified hubness property continues to plague intramodal retrieval on modern backbones, and the authors frame how the "conformity" property impacts scoring in a coherent way. The proposed approach is relatively simple and easy to implement across general pipelines for embedding-based search.

Critical weaknesses

The paper's metric of conformity is a well-known property of contrastively aligned models usually referred to as hubness. Much of the paper's discussion about this can be reduced to additional related work. See: Radovanovic, Milos, Alexandros Nanopoulos, and Mirjana Ivanovic. "Hubs in space: Popular nearest neighbors in high-dimensional data." Journal of machine learning research 11.sept (2010): 2487-2531. Suzuki, Ikumi, et al. "Centering similarity measures to reduce hubs." Proceedings of the 2013 conference on empirical methods in natural language processing. 2013.

The first stage mechanism especially seems like: Wang, Yan, et al. "Simpleshot: Revisiting nearest-neighbor classification for few-shot learning." arXiv preprint arXiv:1911.04623 (2019).

The novel angular shrinkage term should additionally be further justified. The shrinkage schedule seems arbitrary, and it is not clarified why the clamping is introduced empirically. It can also be compared to other training-free baselines: Jégou, Hervé, and Ondřej Chum. "Negative evidences and co-occurences in image retrieval: The benefit of PCA and whitening." European conference on computer vision. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012 Mu, Jiaqi, Suma Bhat, and Pramod Viswanath. "All-but-the-top: Simple and effective postprocessing for word representations." arXiv preprint arXiv:1702.01417 (2017). Radenović, Filip, Giorgos Tolias, and Ondřej Chum. "Fine-tuning CNN image retrieval with no human annotation." IEEE transactions on pattern analysis and machine intelligence 41.7 (2018): 1655-1668.

Questions for authors

If the conformity component is a constant bias term, why does removing it degrade performance on most datasets?

Does the second step still work with a simpler schedule?

Is there a reason some metric of gallery element popularity isn't reported to measure hubness directly?

10 comments
AuthorsOpenReview ↗

Response to Reviewer hhoC [1/5]

We sincerely thank the Reviewer for the insightful feedback, particularly for highlighting the connection between conformity and classical hubness, which we had not sufficiently explored. Motivated by this observation, we conducted additional analyses and found that conformity and hubness are only partially correlated, while changes in overall hubness cannot fully explain CGD's retrieval gains. These findings help clarify the relationship to prior work and strengthen our analysis. We address W1–W4 first, followed by Q1–Q3.


W1: Conformity, hubness, and gallery popularity

W1: The paper's metric of conformity is a well-known property of contrastively aligned models usually referred to as hubness. Much of the paper's discussion about this can be reduced to additional related work. See: Radovanovic, Milos, Alexandros Nanopoulos, and Mirjana Ivanovic. "Hubs in space: Popular nearest neighbors in high-dimensional data." Journal of machine learning research 11.sept (2010): 2487-2531. Suzuki, Ikumi, et al. "Centering similarity measures to reduce hubs." Proceedings of the 2013 conference on empirical methods in natural language processing. 2013.

Answer (W1). We thank the Reviewer for highlighting the connection between conformity and hubness, which motivated us to investigate their relationship more closely.

Conformity and hubness are related but measure different properties. Following Levi and Gilboa [ICML 2025], conformity is defined as the average cosine similarity of an instance to the other instances in a representative dataset, while hubness measures how frequently a gallery sample appears among the nearest neighbors of different queries. We acknowledge the relevant prior work on hubness (Radovanović et al. [JMLR 2010]) and centered similarity (Suzuki et al. [EMNLP 2013]).

We also want to clarify that Stage 1 uses established mean-centering followed by L2 normalization (as in SimpleShot), which we do not claim as novel. Our contribution includes the exact characterization of conformity's query-dependent contribution to retrieval similarity (Eq. 4) and the identification and correction of residual conformity after Stage 1 (see W2).

Following the Reviewer's suggestion, we further investigate the relationship between conformity, hubness, and retrieval performance. For each representation ( Raw, Stage 1, and full CGD ), we report:

  • Spearman ρ(C, N₁₀): The correlation between conformity and gallery popularity, where N₁₀ counts how often a gallery sample appears in query Top-10 results.
  • Skewness and Gini: Measure how concentrated Top-10 occurrences are among gallery samples. Lower values indicate less hubness.

If CGD's gains were primarily explained by reducing aggregate hubness, we would expect retrieval improvements to be accompanied by reductions in skewness and Gini.
Table 10. Conformity and gallery popularity on three T2T and three I2I datasets.

TaskMethodSpearman ρ(C, N₁₀)Skewness ↓Gini ↓Retrieval (mAP (%) )↑
T2TRaw0.24814.130.73833.49
CGD-Stage10.09711.620.71835.77
CGD0.0969.380.71439.50
I2IRaw0.2332.240.51939.84
CGD-Stage10.0092.060.49640.50
CGD0.0052.060.49641.23

Our results reveal that conformity and gallery popularity are only weakly correlated (ρ = 0.248 on T2T and 0.233 on I2I for Raw embeddings). More importantly, on I2I, Stage 2 improves retrieval from 40.50 to 41.23, while skewness and Gini remain unchanged at the reported precision. This indicates that aggregate hubness reduction alone cannot fully explain CGD's retrieval gains .

We further compare CGD with existing hubness correction methods, QB-Norm [CVPR 2022] and NNN [EMNLP 2024]. Adding CGD improves both methods on T2T (+3.51/+2.29) and I2I (+1.23/+1.73), demonstrating their complementarity ( W4, Table 15 ).


AuthorsOpenReview ↗

Response to Reviewer hhoC [2/5]

W2: Relationship to SimpleShot and Mean-centering

Weakness (W2). The first stage mechanism especially seems like: Wang, Yan, et al. "Simpleshot: Revisiting nearest-neighbor classification for few-shot learning." arXiv preprint arXiv:1911.04623 (2019).

Answer (W2). We agree that mean-centering followed by L2 normalization is an established operation, as used in SimpleShot. We do not claim this preprocessing step as novel and will explicitly acknowledge SimpleShot in the revised paper. Our contribution is the geometric analysis connecting conformity to retrieval bias and the resulting progressive correction framework. Specifically, we (i) exactly characterize conformity's contribution to retrieval similarity (Eq. 4), (ii) identify residual conformity that persists after centering and renormalization, and (iii) propose CGD to progressively correct both global and residual conformity.

The residual correction further improves retrieval from 35.77 to 39.50 mAP on T2T and 40.50 to 41.23 on I2I (Table 10). We further justify the angular correction and its schedule in W3.


AuthorsOpenReview ↗

Response to Reviewer hhoC [3/5]

W3: Angular Shrinkage, Its Schedule, and Numerical Clamping

Weakness (W3). The novel angular shrinkage term should additionally be further justified. The shrinkage schedule seems arbitrary, and it is not clarified why the clamping is introduced empirically.

Answer (W3). We thank the Reviewer for this important question. Our angular correction is motivated by three considerations: (i) Semantic preservation: the residual direction contains useful semantic information, motivating partial rather than complete removal; (ii) Geometric motivation: angular contraction attenuates residual conformity while preserving the orthogonal direction and unit norm; and (iii) Empirical validation: the specific retention schedule is heuristic, and comparisons with simpler alternatives support its effectiveness.

(i) Why partial correction? Hard removal assumes that all information along the residual conformity direction is undesirable. However, this direction may also encode useful class-discriminative information.

To investigate this, we conduct a label-based diagnostic on three T2T and three I2I datasets, decomposing each sample's residual projection into a class-mean component, capturing systematic differences between classes, and a within-class residual, capturing variations within each class. We then evaluate the effect of retaining each component separately, compared with completely removing the residual direction.

Table 12. Semantic decomposition of the residual direction (mAP, %).

CorrectionT2T ↑I2I ↑
Hard Removal40.9940.06
Keep Class-Mean Component Only42.6142.47
Keep Within-Class Residual Only38.2939.86

Retaining the class-mean component consistently outperforms hard removal (42.61 vs. 40.99 on T2T; 42.47 vs. 40.06 on I2I), whereas retaining only the within-class residual performs worse. These findings reveal that the residual conformity direction also carries useful class-related information. This motivates our partial angular contraction.

(ii) Why angular contraction and this schedule? Our analysis identifies residual conformity as a direction on the unit sphere. Angular contraction naturally reduces projections along this direction while preserving the orthogonal direction and unit norm.

We do not claim that our specific schedule t_i=(1-p_i)/2 is theoretically optimal. In retrieval, establishing a universally optimal correction is challenging because both representations and relevance judgments are approximations. Many existing methods therefore exploit geometric properties to improve retrieval through empirically validated corrections, including ABTT [ICLR 2018], RootSIFT [CVPR 2012], CSLS [ICLR 2018], and QB-Norm [CVPR 2022]. Similarly, CGD's correction direction follows from geometric analysis, while its attenuation strength is an empirically validated design choice.

To verify that the gains do not depend on this exact schedule, we compare fixed retention ratios of 1/2 and 1/4.

Table 13. T2T: Fixed vs. adaptive angular retention (mAP, %).

DatasetRawStage 1Fixed 1/2Fixed 1/4Adaptive (Ours)
COCO Text26.2027.9428.3328.5929.20
Flickr30K51.6850.8255.4755.3556.08
NoCaps35.0737.6639.0238.9639.63

Table 14. I2I: Fixed vs. adaptive angular retention (mAP, %).

DatasetRawStage 1Fixed 1/2Fixed 1/4Adaptive (Ours)
EuroSAT47.9348.8950.0950.2150.42
Oxford Pets30.5331.8432.9832.8633.19
Food10132.3233.2134.0834.2034.41
Stanford Cars24.6024.9525.3925.2725.60

All experiments use CLIP ViT-B/32.

Both fixed schedules improve retrieval, while the adaptive schedule performs best across all seven datasets. This supports the effectiveness of angular correction without requiring one theoretically optimal retention rule.

(iii) Numerical clamping. Since βi∈[−1,1]\beta_i \in [-1,1] theoretically, we use a margin of 10−710^{-7} solely to prevent floating-point errors in arcsin⁡(βi)\arcsin(\beta_i) and 1−βi2\sqrt{1-\beta_i^2}. This margin does not affect the contraction schedule.


Reviewer_vVg2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies conformity bias in CLIP-like representations for image-to-image and text-to-text retrieval. The authors argue that some samples become similar to many other samples because they are close to a shared conformity direction, which can distort retrieval rankings. The paper proposes Conformity-Guided Debiasing (CGD). CGD first estimates a mean direction from an external reference set and performs mean-centering. It then applies adaptive angular contraction to reduce the residual conformity direction that appears after re-normalization. The method is training-free and does not use query or gallery statistics. Experiments cover 6 CLIP-like backbones, 9 text retrieval benchmarks, 13 image retrieval benchmarks, and 11 few-shot classification datasets. The results generally show improvements from CGD, and CGD can also be combined with existing methods such as IsoCLIP and PCA-based correction.

Critical strengths

1、The problem is important, and the geometric analysis is interesting. The paper connects conformity to a specific direction on the unit hypersphere and derives how this direction affects retrieval similarity. 2、CGD is simple, lightweight, and easy to deploy. It does not modify or retrain the backbone and does not require query or gallery statistics. The correction can be fitted once using an external reference set and then reused. 3、The experimental coverage is broad. The method is evaluated across multiple backbones, text retrieval, image retrieval, and few-shot classification tasks. CGD improves performance in most settings, suggesting reasonable generality. 4、The paper includes useful ablations on: reference-set source; reference-set size; mean-centering and hard projection; one-sided versus bidirectional correction; angular-retention strength; iterative correction. These experiments help clarify the design choices behind CGD.

Critical weaknesses

1、CGD relies on a single dominant conformity direction and one residual direction. Real CLIP representations may contain multiple semantic, class-specific, or modality-related bias directions. A single-direction correction may remove useful semantic information. The paper acknowledges this limitation, but does not compare against multi-direction or subspace-based alternatives. 2、Performance improvements alone do not fully establish that conformity is the causal source of the retrieval errors.The transformation may improve performance for other reasons, such as changing the similarity distribution, reducing anisotropy, or modifying the embedding scale. More direct evidence is needed to connect the gains specifically to the removal of conformity bias. 3、The reference-set distribution still has a noticeable effect. Although the paper claims that CGD does not require target-distribution data, in-distribution reference sets often provide larger gains than out-of-distribution sets.Moreover, the residual conformity direction is highly reference-dependent. The reported cross-set cosine similarity is about 0.601 for image representations and only 0.445 for text representations. This suggests that the second stage of CGD may depend strongly on the reference distribution. 4、The baseline comparison could be broader and more systematic. CGD is mainly compared with Raw, IsoCLIP, and PCA, but there is limited comparison with other post-processing methods for hubness, anisotropy, or intra-modal retrieval. The default reference set is often COCO, while the evaluation tasks cover very different domains. The paper should clarify whether the choice of reference data gives CGD an additional advantage.

Questions for authors

How can you verify that the gains come specifically from conformity correction rather than from general anisotropy reduction or redistribution of the embeddings? The residual conformity direction is sensitive to the reference set. Why does CGD still produce stable gains when the cross-reference cosine similarity is only 0.445 for text representations? When the reference set is substantially different from the target task, can CGD remove task-specific semantic structure? Could you report more cross-domain experiments? Could you report how CGD separately changes the similarity distributions of positive and negative pairs, rather than only reporting the overall mean similarity?

8 comments
AuthorsOpenReview ↗

Response to Reviewer vVg2 [1/5]

We thank the Reviewer for the constructive feedback and for highlighting important concerns regarding single-direction correction, semantic preservation, the source of CGD's improvements, reference-set sensitivity, and baseline comparisons. We provide broader baseline comparisons and additional analyses and experiments to address these concerns. We address W1–W4 and Q1–Q5.


W1: Single-direction Correction and Semantic Preservation

Weakness (W1). CGD relies on a single dominant conformity direction and one residual direction. Real CLIP representations may contain multiple semantic, class-specific, or modality-related bias directions. A single-direction correction may remove useful semantic information. The paper acknowledges this limitation, but does not compare against multi-direction or subspace-based alternatives.

Answer (W1). Conformity, as previously defined by Levi and Gilboa, is mathematically characterized by a single mean direction (Lemma 1). This does not imply that all embedding biases lie along a single direction. Moreover, iterative CGD can progressively correct multiple directions, as demonstrated in Appendix Fig. 6. To address the Reviewer's concerns, we provide additional evidence from two perspectives: (i) comparisons with multi-directional and subspace-based methods, and (ii) semantic preservation analysis.

(i) Broader comparisons: We added the requested comparisons with multi-directional and subspace-based alternatives, including anisotropy correction (PCA Whitening [ECCV 2012], Learned Whitening [TPAMI 2019]) and subspace correction (ABTT [ICLR 2018], Cluster-based ABTT [ACL 2021]). We further extend the evaluation to hubness correction (QB-Norm [CVPR 2022], NNN [EMNLP 2024]). As shown in Table 1, CGD outperforms all six standalone baselines on T2T retrieval, and adding CGD consistently improves all six methods in both modalities.

Table 1. Comparison with existing post-hoc correction methods (average mAP, %).

FamilyMethodT2T Avg. ↑I2I Avg. ↑
BaselineRaw35.1940.92
OursRaw + CGD40.76 (↑ +5.57)42.31 (↑ +1.39)
Anisotropy correctionPCA Whitening33.7337.00
↳PCA Whitening + CGD35.18 (↑ +1.45)37.62 (↑ +0.62)
Learned Whitening37.7842.90
↳Learned Whitening + CGD39.58 (↑ +1.80)43.10 (↑ +0.20)
Subspace correctionABTT37.8340.97
↳ABTT + CGD40.18 (↑ +2.35)41.86 (↑ +0.89)
Cluster-based ABTT34.4038.99
↳Cluster-based ABTT + CGD35.33 (↑ +0.93)39.43 (↑ +0.44)
Hubness correctionQB-Norm36.1840.15
↳QB-Norm + CGD39.69 (↑ +3.51)41.38 (↑ +1.23)
NNN39.1839.69
↳NNN + CGD41.47 (↑ +2.29)41.42 (↑ +1.73)

All methods use the same external COCO training reference bank (images for I2I, captions for T2T). Averages cover the nine T2T and thirteen I2I benchmarks in the main paper.

CGD improves all six methods in both modalities.

(ii) Semantic preservation: We evaluate foreground-object (FG) and background-context (BG) retrieval on Waterbirds (DAT [ICML 2026]), NICO++ [CVPR 2023], and COCO-Stuff [CVPR 2018]. FG and BG retrieval use the same corrected image embeddings but different relevance labels, allowing us to assess whether CGD preserves both types of semantic information. Specifically, FG retrieval evaluates whether images with the same foreground-object category are retrieved, while BG retrieval evaluates whether images sharing the same background-context category are retrieved.

Table 2. Foreground-object and background-context retrieval (group-macro mAP, %, using the full COCO training set as the reference bank).

DatasetTaskRawPCA WhiteningIsoCLIPCGD (Ours)
WaterbirdsFG / Object58.69658.406 (−0.290)59.378 (+0.682)59.057 (+0.361)
BG / Context66.18865.570 (−0.618)65.938 (−0.250)67.129 (+0.941)
NICO++FG / Object51.11449.422 (−1.692)53.360 (+2.246)53.206 (+2.092)
BG / Context38.72340.096 (+1.373)38.787 (+0.064)39.090 (+0.367)
COCO-StuffFG / Object46.14249.412 (+3.270)48.931 (+2.789)49.436 (+3.294)
BG / Context15.75313.078 (−2.675)15.241 (−0.512)16.666 (+0.913)
Overall Avg. (FG + BG)6 tasks, equal weight46.10345.997 (−0.105)46.939 (+0.837)47.431 (+1.328)

CGD is the only method that consistently improves all six FG/BG retrieval tasks, achieving the highest combined score (47.431 mAP). This demonstrates that CGD effectively reduces conformity while preserving useful semantic information about both foreground objects and background context.


AuthorsOpenReview ↗

Response to Reviewer vVg2 [2/5]

W2: Conformity Correction Beyond General Geometric Changes

Weakness (W2). Performance improvements alone do not fully establish that conformity is the causal source of the retrieval errors.The transformation may improve performance for other reasons, such as changing the similarity distribution, reducing anisotropy, or modifying the embedding scale. More direct evidence is needed to connect the gains specifically to the removal of conformity bias.

Answer (W2). Thanks, this motivated us to investigate the source of CGD's improvements more carefully. While CGD naturally affects embedding geometry and similarity distributions, our new experiments show that changes in general geometric statistics alone do not fully account for its retrieval gains. The results further support the importance of correcting the estimated conformity direction. We examine the three alternative explanations below.

(i) Similarity distributions. CGD lowers the mean similarity of positive and negative pairs, with a larger reduction for negatives. Yet pairwise AUC improves from 92.2% to 96.7% on T2T and from 87.3% to 89.1% on I2I, showing better relative ranking beyond global shifts or monotonic rescaling(see Q5).

(ii) Embedding scale. Since retrieval uses L2-normalized cosine similarity, changes in embedding norms alone cannot affect retrieval rankings.

(iii) Anisotropy. We measure isotropy with IsoScore (Rudman et al. [ACL 2022]); higher scores indicate lower anisotropy.

CGD has two stages. Stage 1 (Global Conformity Correction) subtracts the reference mean and applies L2 normalization. This removes the global conformity direction (Lemma 1) and changes mean-related geometry (e.g., average cosine similarity), so the two effects cannot be disentangled. Stage 2 (Residual Conformity Correction) estimates the residual conformity direction from Stage-1-transformed reference embeddings and applies adaptive angular contraction.

Stage 2 lets us examine the role of anisotropy. With Stage 1 fixed, we compare three Stage-2 controls: (1) Random Direction, using a random unit vector; (2) Variance-Matched Random, matching reference-set projection variance; and (3) Anisotropy-Matched Random Subspace, matching CGD's final IsoScore through contraction in a random orthogonal subspace.

We evaluate these controls on three T2T datasets (COCO Text, Flickr30K, NoCaps) and eight I2I datasets (Pets, Flowers, Aircraft, DTD, EuroSAT, Food101, Caltech, CIFAR-10), using COCO Train as the reference bank.

Table 3. Stage-2 direction and anisotropy controls on T2T and I2I retrieval (average mAP, %).

MethodT2T mAP ↑T2T IsoScore ↑I2I mAP ↑I2I IsoScore ↑
Raw37.650.128141.80.0845
CGD Stage 1 only38.810.128242.540.0845
+ Stage 2: Random Direction38.80 (-0.01)0.128342.54 (+0.00)0.0845
+ Stage 2: Variance-Matched Random Direction38.87 (+0.06)0.132242.64 (+0.10)0.0880
+ Stage 2: Anisotropy-Matched Random Subspace39.13 (+0.32)0.142742.42 (−0.12)0.0857
+ Stage 2: Residual Conformity Correction (Full CGD)41.64 (+2.83)0.142243.50 (+0.96)0.0857

All Stage-2 corrections share the same Stage-1 embeddings. Gains in parentheses are relative to CGD Stage 1. Higher IsoScore indicates greater isotropy.

The results show that matching isotropy alone cannot reproduce CGD's gains. Stage 1 improves retrieval while IsoScore remains nearly unchanged (0.1281 → 0.1282 on T2T; 0.0845 → 0.0845 on I2I). The anisotropy-matched control reaches nearly the same IsoScore as CGD on T2T (0.1427 vs. 0.1422) and the same IsoScore on I2I (0.0857), yet yields only +0.32 mAP on T2T and −0.12 on I2I, compared with CGD's +2.83 and +0.96. Together with Table 1, these results show that anisotropy changes alone cannot explain CGD's gains.

To test whether conformity contributes to retrieval errors (Eq. 4), we increase selected gallery samples' conformity projections while preserving their orthogonal-component directions. The experiment shows that their Top-10 occurrence increases by 0.32, 0.63, and 0.96 for low-, medium-, and high-conformity queries, respectively, consistent with the predicted query-dependent ranking bias.

We also test whether CGD corrects these errors.

Table 4. Correct-ranking probability for low- and high-conformity negatives on T2T (%).

MethodLow-Conformity Negatives ↑High-Conformity Negatives ↑
Raw98.6%82.3%
Random Direction98.6%82.2%
CGD (Ours)98.2%95.3%

CGD improves the correct-ranking probability for high-conformity negatives from 82.3% to 95.3%, whereas random-direction correction leaves it nearly unchanged (82.2%).

Together, the anisotropy controls, direct conformity interventions, and ranking results link conformity correction to the observed retrieval gains.

AuthorsOpenReview ↗

Response to Reviewer vVg2 [3/5]

W3: Reference sensitivity and cross-domain robustness

W3: The reference-set distribution still has a noticeable effect. Although the paper claims that CGD does not require target-distribution data, in-distribution reference sets often provide larger gains than out-of-distribution sets.Moreover, the residual conformity direction is highly reference-dependent. The reported cross-set cosine similarity is about 0.601 for image representations and only 0.445 for text representations. This suggests that the second stage of CGD may depend strongly on the reference distribution.

Answer (W3). We have already investigated reference sensitivity in Appendix B.2 of the original submission. Motivated by the Reviewer's concern, we further extend this analysis. We agree that reference choice affects CGD's performance. Our experiments show that COCO provides consistent improvements across diverse retrieval domains without requiring task-specific reference selection. Domain-specific references can offer larger in-domain gains, but their advantages do not necessarily transfer to other domains. This motivates our choice of COCO as a fixed, general-purpose reference bank.

To further examine reference sensitivity, we keep CGD Stage 1 fixed using COCO and vary the reference used for Stage 2's residual direction and percentile calibration. This isolates the effect of Stage-2 reference selection while keeping the initial correction identical.

Table 5. I2I retrieval with fixed COCO Stage 1 and varied Stage-2 references (mAP, %).

DatasetStage 1COCOImageNetEuroSAT valPets valFood101 valSUN
EuroSAT48.89050.42049.88352.08649.91749.93050.052
Oxford Pets31.84933.19033.53132.97435.76832.93233.353
Food10133.21034.41034.49634.33034.27935.84834.339

Table 6. T2T retrieval with fixed COCO Stage 1 and varied Stage-2 references (mAP, %).

DatasetStage 1COCO captionsMedicalFinanceQuoraWANDSSciFact
SciFact26.99631.54733.40932.49734.10128.08238.031
NoCaps37.66039.62838.68738.53638.73437.76938.761

NoCaps does not provide a separate validation reference set for our experiments; among the available reference banks, COCO captions have the closest distribution to NoCaps.

All tested references improve retrieval over the shared Stage-1 baseline. Independent in-domain references achieve the highest performance on EuroSAT, Oxford Pets, Food101, and SciFact. However, their advantages are task-dependent. COCO consistently improves all evaluated targets using a single reference source, whereas domain-specific references do not consistently outperform COCO across different domains.

To further investigate whether out-of-domain references remove useful semantic information, we evaluate foreground-object (FG) and background-context (BG) retrieval on Waterbirds, NICO++, and COCO-Stuff using three reference banks: COCO, ImageNet, and EuroSAT satellite images. Each reference contains 1,000 samples, and the same corrected embeddings are used for both FG and BG retrieval.

Table 7. Cross-domain semantic preservation under different reference banks (FG / BG mAP, %).

Reference BankWaterbirdsNICO++COCO-Stuff
Raw58.696 / 66.18851.114 / 38.72346.142 / 15.753
CGD: COCO-1K59.054 / 67.12253.176 / 39.08049.362 / 16.639
CGD: ImageNet-1K59.375 / 67.60253.585 / 39.35048.932 / 16.538
CGD: EuroSAT-1K58.967 / 66.62052.611 / 38.94348.579 / 16.163

Each cell reports FG / BG mAP (%). Bold indicates the best reference for each target dataset.

CGD improves both FG and BG retrieval under all three reference banks, including the substantially different EuroSAT satellite-image reference. This further supports the preservation of task-relevant semantic information under reference-domain shifts.

Overall, these results confirm that reference selection matters, while COCO offers a consistent and practical general-purpose choice across diverse retrieval tasks. We discuss why substantially different residual directions can still yield positive retrieval gains in Q2.


Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer wqh2 [1/3]

We thank the Reviewer for recognizing the geometric analysis and broad empirical performance of CGD (54/54 T2T and 77/78 I2I backbone–dataset combinations improved). The questions raised in this review round also prompted us to substantially expand our experiments, bringing to light several aspects that were not sufficiently explored in the original submission, including the semantic information in the residual direction, the robustness of the contraction schedule, and the statistical reliability of the improvements. We address these points below.


W1: Motivation and Selection of the Contraction Rule

Weakness (W1). The residual correction rule is not derived from a semantic risk objective. The percentile-based contraction t = (1 - p)/2 and the guarantee that at least half of the residual angle is removed are heuristic. The paper shows that hard projection fails, but it does not provide a loss-based or ranking-probability argument for the specific partial contraction. This leaves the boundary between removing nuisance commonality and removing useful semantic structure insufficiently characterized.

Answer (W1). We thank the reviewer for raising this concern. Our semantic decomposition (Table 16) shows that the residual direction contains useful class-related information: retaining its class-mean component performs better than completely removing it. This motivates partial angular contraction rather than hard removal. The specific retention schedule is an empirically validated heuristic, not a theoretically optimal solution. Simpler fixed schedules also improve retrieval, while our adaptive schedule performs best across the seven tested datasets (Tables 17–18).

(i) Why partial correction? Hard removal assumes that all information along the residual conformity direction is undesirable. However, this direction may also encode useful class-discriminative information. On three T2T and three I2I datasets, we use labels to split residual projections into class-mean and within-class components, comparing both.

Table 16. Semantic decomposition of the residual direction (mAP, %).

CorrectionT2T ↑I2I ↑
Hard Removal40.9940.06
Keep Class-Mean Component Only42.6142.47
Keep Within-Class Residual Only38.2939.86

Retaining the class-mean component outperforms hard removal on both T2T (42.61 vs. 40.99) and I2I (42.47 vs. 40.06). This reveals that the residual direction also carries useful class-related information, explaining why complete removal can lead to overcorrection.

(ii) Why angular contraction and this schedule? In retrieval, establishing a universally optimal correction is challenging because representations and relevance judgments are approximations. Many existing methods therefore adopt geometry-motivated, empirically validated transformations, including ABTT [ICLR 2018], RootSIFT [CVPR 2012], CSLS [ICLR 2018], and QB-Norm [CVPR 2022].

CGD follows the same principle: the correction direction is analytically identified, while the attenuation strength is empirically validated. Our geometric analysis identifies the residual conformity direction on the unit sphere, motivating angular contraction that preserves unit norm and the orientation of orthogonal components. The specific schedule $t_i=(1-p_i)/2$, where $p_i=F_{\mathrm{ref}}(|\beta_i|)$, applies stronger contraction to larger residual projections without task-specific tuning. We do not claim theoretical optimality or a semantic-risk guarantee. The bound $t_i\leq 1/2$ is a design choice, and its effectiveness is supported by the fixed-schedule comparisons below.

To assess whether the gains depend on this particular rule, we compare two simpler schedules retaining a fixed fraction (1/2 or 1/4) of the original residual angle.

Table 17. T2T: Fixed vs. adaptive angular retention (mAP, %).

DatasetRawStage 1Fixed 1/2Fixed 1/4Adaptive (Ours)
COCO Text26.2027.9428.3328.5929.20
Flickr30K51.6850.8255.4755.3556.08
NoCaps35.0737.6639.0238.9639.63

Table 18. I2I: Fixed vs. adaptive angular retention (mAP, %).

DatasetRawStage 1Fixed 1/2Fixed 1/4Adaptive (Ours)
EuroSAT47.9348.8950.0950.2150.42
Oxford Pets30.5331.8432.9832.8633.19
Food10132.3233.2134.0834.2034.41
Stanford Cars24.6024.9525.3925.2725.60

All experiments use CLIP ViT-B/32.

Both fixed schedules improve upon Stage 1, while the adaptive schedule performs best on all seven evaluated datasets. These results show that the benefits of residual conformity correction do not depend on one precise retention formula, while supporting our adaptive design.


AuthorsOpenReview ↗

Response to Reviewer wqh2 [2/3]

W2: Statistical reliability of small improvements

W2: Many reported gains are small and lack uncertainty estimates. Several improvements on already corrected representations are between 0.1 and 0.5 points. The main tables do not report confidence intervals, split variance, or paired tests. A consistently positive sign across many settings is reassuring, but the absence of uncertainty makes it difficult to judge which per-dataset differences are stable.

Answer (W2). All 95% confidence intervals in the main comparisons below have positive lower bounds, including CGD applied on top of IsoCLIP and PCA←.
We perform query-level paired bootstrap with 10,000 resamples across all nine T2T and thirteen I2I datasets reported in the original paper. We retain the original benchmark-average gains as point estimates and use bootstrap resampling to compute their 95% confidence intervals.

As shown in Tables 19–20, all reported 95% confidence intervals have positive lower bounds, supporting statistically reliable average improvements across the evaluated benchmarks.

Table 19 — T2T: bootstrap mean gains and 95% CIs.

BackboneRaw + CGDIsoCLIP + CGD
CLIP B/32+5.57 [5.29, 5.85]+0.78 [0.66, 0.89]
SigLIP B/16+6.06 [5.65, 6.47]+0.98 [0.75, 1.41]
SigLIP2 B/16+4.82 [4.43, 5.31]+0.97 [0.83, 1.20]

Table 20 — I2I: bootstrap mean gains and 95% CIs.

BackboneRaw + CGDIsoCLIP + CGDPCA← + CGD
CLIP B/32+1.39 [1.32, 1.45]+0.56 [0.44, 0.62]+0.52 [0.44, 0.57]
SigLIP B/16+1.60 [1.37, 1.83]+0.96 [0.93, 0.99]+0.75 [0.62, 0.78]
SigLIP2 B/16+1.23 [1.20, 1.29]+0.47 [0.37, 0.55]+1.16 [1.03, 1.29]

Mean gain over each corresponding baseline [95% CI], percentage points. Original T2T and I2I tables are shown separately.

These intervals support positive average gains under query resampling.


AuthorsOpenReview ↗

Response to Reviewer wqh2 [3/3]

Questions for Authors

Q1: Can the authors report confidence intervals or paired bootstrap tests for the main gains, especially CGD on top of IsoCLIP and PCA←?

Answer (Q1). Yes. We conducted query-level paired bootstrap with 10,000 resamples across all nine T2T and thirteen I2I datasets. All reported 95% CIs for benchmark-average gains exclude zero, including improvements on top of IsoCLIP and PCA←, supporting the statistical reliability of the average gains (see W2, Tables 19–20).


Q2: Why is t = (1 - p)/2 the preferred contraction, rather than a value fitted on a validation split? Does the optimal contraction correlate systematically with backbone, modality, or dataset distribution?

Answer (Q2). We have discussed the motivation and empirical validation of this contraction rule in detail in W1 (Tables 16–18), including semantic preservation and comparisons with fixed retention schedules. The adaptive rule performs best across the seven tested datasets without task-specific tuning. We do not want to claim that this particular schedule is theoretically optimal.


Q3: Can a random-direction or norm-matched control isolate whether the gain comes specifically from the estimated residual conformity axis rather than from general angular regularization?

Answer (Q3). Yes. We thank the Reviewer for this valuable suggestion, which motivated us to conduct additional controlled experiments. The results show that CGD's improvements depend on the estimated residual conformity direction.

We evaluate two controls on 3 T2T and 8 I2I datasets, keeping Stage 1 and the reference bank fixed:

  • Random Direction: Replaces the estimated residual conformity axis with a random unit direction while retaining the same contraction rule.
  • Norm-Matched Random: Applies the same per-sample angular displacement as CGD, but along a random direction.

Table 21. Random-direction and Norm-Matched Random controls (gain over Stage 1, mAP points).

Stage-2 CorrectionT2T (3) ↑I2I (8) ↑
Random Direction (10 draws)−0.01 [−0.04, 0.01]+0.00 [−0.01, 0.00]
Norm-Matched Random (10 draws)−0.25 [−0.39, −0.11]−0.10 [−0.10, −0.09]
Residual Conformity Direction (CGD)+2.83 [2.76, 3.10]+0.96 [0.95, 0.96]

Results are reported as gains over Stage 1 with 95% confidence intervals estimated using 10,000 query-level paired bootstrap resamples.

These results provide direct evidence that the estimated residual conformity direction is important for CGD's effectiveness.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.