acceptodds
Under review as a conference paper at ICLR 2027

GeoRx: A Reference-Based Benchmark for Attribution Diagnosis of Cross-Modal Embeddings

Abstract

Cross-modal encoders map multimodal data into a shared embedding space for semantic alignment and retrieval. However, geometric deviations in this space can cause cosine-based rankings to diverge from semantic relevance, limiting retrieval accuracy. Existing benchmarks primarily measure end-to-end retrieval accuracy, offering limited support for attributing retrieval failures to geometric causes. To address this limitation, we introduce GeoRx, a reference-based benchmark for geometric attribution diagnosis in cross-modal retrieval. For each pair, GeoRx constructs a geometric reference model matched to the encoder's embedding dimension and the workload's corpus size. Five diagnostic metrics derived from candidate–corpus, candidate–candidate, and within-list relationships identify geometric deviations against this reference without requiring additional information. These relationships respectively characterize candidate occurrence frequency across queries, similarity among retrieved candidates, and variation in query–candidate scores. We validate diagnostic accuracy through controlled deviation injection. Guided by the diagnoses, we map correction methods from existing work to the identified deviations for individual and joint correction. Experiments demonstrate target detection rates of 98.93–100.0% and an overall hit rate of 32.35% across diagnosis–correction mappings. Code is available at https://anonymous.4open.science/r/GeoRx-3CC3/.

Primary area:
datasets and benchmarks
Keywords:
Cross-modal retrieval, Embedding geometry, Geometric diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C9tMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

GeoRx is a reference-based benchmark for attributing geometric patterns in cross-modal embedding spaces. For each encoder-workload pair, it constructs an isotropic unit-sphere reference matched to the embedding dimension and corpus size, then compares five unsupervised statistics against that reference: global hubness (H1), local-neighborhood concentration (H2), local graph fragmentation (H3), top-1/top-2 score-margin ambiguity (H4), and a query-specific interaction-energy ratio (H5). Robust centers and scales are estimated from reference windows, and a joint conformal threshold is used to report a set of detected deviations. The experiments cover nine encoders and ten M-BEIR workloads, yielding 90 encoder-workload pairs with three random seeds. The authors validate the diagnostic metrics using controlled single- and dual-deviation injections, report 100.00% target detection for single injections and 98.93% for dual injections, and evaluate mappings from H1-H5 to CSLS, alpha-QE, diffusion, listwise cross-encoders, and pointwise cross-encoders. They also report a 32.35% overall diagnosis-to-correction hit rate and ablations on reference matching and joint threshold calibration.

Critical strengths

The paper separates diagnosis of embedding-space statistics from end-to-end retrieval scores and does not use correction outcomes to define the diagnostic set, which avoids an obvious circular evaluation loop. Broad and reasonably structured evaluation. The study spans nine encoders, ten workloads, three seeds, disjoint fitting/calibration/evaluation subsets, and an explicit threshold ablation. Matching the reference to both dimension and corpus size is a sensible and practically motivated design choice.

Critical weaknesses
  1. A deviation from an isotropic null can reflect legitimate semantic structure, task imbalance, or multimodal data properties rather than a retrieval pathology. The paper does not establish that H2, H4, or H5 deviations predict retrieval failures, nor that the reported false-positive control transfers from held-out spherical windows to real encoder spaces. The 1.04% false-positive result may be therefore much narrower than the practical claim suggested by the benchmark framing.

  2. The main validation is metric-aligned and the correction utility is not yet demonstrated. The controlled injections are designed around the same five statistics and use a single shared strength of 0.75; exact operators for H1-H4 are not specified in Appendix A.3. In addition, the 32.35% mapping hit rate has no always-one-method, random, or best-single-method baseline, uses an arbitrary Delta Recall@10 >= 0.5% criterion, reports no uncertainty, and the joint experiment covers only four pairs and ten pair-seed observations. H3 cannot be evaluated because it is never diagnosed in the real benchmark.

Questions for authors
  1. Please provide exact transformation formulas or pseudocode for all five injection operators, especially H1-H4, together with the number of successfully formed targets and sensitivity to injection strength and random seed. How is the H3 formation criterion (nonzero second-largest component in Appendix A.3) reconciled with the diagnostic requirement x_H3 >= 0.15 in Section 3.2?
  2. Can you evaluate the diagnostic set on independently generated non-spherical embedding spaces or naturally occurring perturbations with known ground truth, and report its correlation with retrieval degradation? How do you distinguish a sphere-relative deviation from beneficial semantic structure in a real workload?
  3. Please report per-pair and per-seed results with confidence intervals, and compare diagnosis-conditioned correction against no correction, always-CSLS, the best fixed single method, and a random mapping. Please justify the 0.5% effectiveness threshold and state how fusion weights are selected without using the reported evaluation observations.
9 comments
AuthorsOpenReview ↗

Response to Q1.

1. Q1: Exact Injection Operators (H1–H5), Formed Targets across Strengths/Seeds, and Reconciling the Two H3 Criteria

A1. We sincerely thank the reviewer for these detailed questions. Rather than editing similarity scores directly, our operators perturb L2-normalized embeddings (unit⁡(v)=v∣∣v∣∣2\operatorname{unit}(v) = v||v||_2) with strength t∈[0,1]t \in [0, 1] and recompute full-corpus retrieval from scratch. We provide detailed pseudocode (Appendix A.3, marked in Orange) for all five injection operators (H1–H5):

Let cpool=unit⁡((1/∣C∣)∑jcj)c_pool = \operatorname{unit}((1/|\mathcal{C}|) \sum_j c_j) be the mean corpus direction:

H1. Select hub chub=argmax(cj)[cjTcpool]c_{\mathrm{hub}} = argmax(c_j) [c_j^T c_{\mathrm{pool}}]; update queries via qi′=unit⁡((1−t)qi+tchub)q_i' = \operatorname{unit}((1-t)q_i + t c_{\mathrm{hub}}).

H2. Select support set SS closest to cpoolc_{\mathrm{pool}} with center cS=unit⁡(∑j∈Scj)c_S = \operatorname{unit}(\sum_{j \in S} c_j); update cj′=unit⁡((1−t)cj+tcS)(∀j∈S)c_j' = \operatorname{unit}((1-t)c_j + t c_S) (\forall j \in S) and qi′=unit⁡((1−t)qi+tcS)q_i' = \operatorname{unit}((1-t)q_i + t c_S).

H3. Split candidate subset into G+G_+, G−G_- along axis uu, shift via unit⁡(cj±tu)\operatorname{unit}(c_j ± t u), contract toward group centers g+g_+, g−g_-, and align queries with midpoint m=unit⁡(g++g−)m = \operatorname{unit}(g_+ + g_-) orthogonal to uu.

H4. Select top-2 candidates cac_a, cbc_b closest to cpoolc_{\mathrm{pool}} with midpoint o=unit⁡(ca+cb)o = \operatorname{unit}(c_a + c_b); update cj′=unit⁡((1−t)cj+to)c_j' = \operatorname{unit}((1-t)c_j + t o) for j∈{a,b}j \in \{a,b\}, and project out the query component parallel to δ=unit⁡(ca′−cb′)\delta = \operatorname{unit}(c_a' - c_b') from v=unit⁡((1−t)qi+to)v = \operatorname{unit}((1-t)q_i + t o) via qi′=unit⁡(v−(vTδ)δ)q_i' = \operatorname{unit}(v - (v^T \delta)\delta).

H5. Let qˉ=unit⁡(∑q∈Qtrainq)\bar q = \operatorname{unit}(\sum_{q \in Q_{\mathrm{train}}} q); update queries via qi′=unit⁡((1−t)qi+tqˉ)q_i' = \operatorname{unit}((1-t)q_i + t \bar q).

Formed Targets and Sensitivity to Injection & Random Seed. We evaluated 1,350 single-target injections across different injection strengths t∈{0.25,0.50,0.75,1.00}t \in \{0.25, 0.50, 0.75, 1.00\} (90 pairs × 5 metrics × 3 seeds). Results are highly consistent across random seeds (e.g., 100% formation across all 3 seeds at t≥0.75t \geq 0.75, and 78/90 uninjected pairs yield identical diagnostic sets across all 3 seeds). Even at t=0.25t = 0.25, GeoRx detects 99.83% (594/595) of formed targets:

Injection strengthNumber of formed targets
0.25595
0.51240
0.751350
1.01350

Here, the detection rate is the proportion of formed targets that GeoRx detects, while the false-positive rate measures detections among targets that did not yet meet the strict 90%-query formation cutoff. Moreover, formed injections directly cause end-to-end Recall@10 degradation: under H4 injection (averaged over 3 seeds below), recall changes little when H4 fails to form (t=0.25t = 0.25), but drops sharply once formed (t≥0.50t \geq 0.50):

Injection strength (tt)<GME, WebQA> Formed<GME, WebQA> Recall@10 before → after<CLIP-SF, MSCOCO> Formed<CLIP-SF, MSCOCO> Recall@10 before → after
0.25No (0/3)97.71% → 97.08%No (0/3)87.19% → 86.25%
0.50Yes (3/3)97.71% → 89.58%Yes (3/3)87.19% → 58.96%
0.75Yes (3/3)97.71% → 0.63%Yes (3/3)87.19% → 1.67%
1.00Yes (3/3)97.71% → 0.00%Yes (3/3)87.19% → 0.21%

Reconciling the Two H3 Criteria (Appendix A.3 vs. Section 3.2) and Evaluating H3. The formation criterion (Appendix A.3) tests whether a structural split has occurred after injection: for at least 90% of queries, the kNN graph on Ln(q)L_n(q) must split into ≥ 2 connected components (i.e., nonzero second-largest component). The diagnostic criterion (Section 3.2) additionally requires the split to be non-trivial in size (xH3≥0.15x_{\mathrm{H3}} \geq 0.15) along with zH3>τz_{\mathrm{H3}} \gt \tau to filter out tiny peripheral fragments in real spaces. Across the 270 seed-level evaluations on the 90 real pairs, 136 had a nonzero secondary component (xH3>0x_{\mathrm{H3}} \gt 0), but the maximum was only 4.34% and no zH3z_{\mathrm{H3}} ever exceeded τ\tau (0/270)—meaning even without the xH3≥0.15x_{\mathrm{H3}} \geq 0.15 filter, H3 is 0/270 at k=10k=10. Evaluating its mapped correction (Diffusion) across all 90 pairs yields a near-zero mean ΔRecall@10\Delta \mathrm{Recall}@10 of -0.09% (Figure 3(a)), confirming H3's utility in ruling out non-existent fragmentation.

AuthorsOpenReview ↗

Response to W1 and Q2.

2. W1 & Q2: Distinguishing Sphere-Relative Deviations from Beneficial Semantic Structure, False-Positive Transfer, and Correlation with Retrieval Degradation

A2. Thank you for raising this point. In the embedding spaces we study, queries and candidates lie on the unit hypersphere after L2 normalization (Section 3.1, marked in Blue). Thus, we use the isotropic spherical distribution as a scale-calibrated null baseline (matched to d and ∣C∣|\mathcal{C}|) rather than an ideal semantic space, and evaluate non-isotropic perturbations via both controlled injections (with known ground truth) and unperturbed real workloads:

① Transfer of False-Positive Control to Real Encoder Spaces. Under the spherical reference, the ground-truth diagnostic set is strictly empty, allowing Table 2 to isolate false alarms (1.04% for joint conformal thresholding vs. 32.29% for independent thresholding). As noted in Appendix A.2 (marked in Orange), this 1.04% false-positive rate applies to spherical reference windows and does not claim a false-positive rate for real encoder spaces where the ground-truth deviation set is unknown. Instead, in real spaces, zm>τz_m \gt \tau measures how far the space deviates from the spherical reference, and we validate these diagnoses by their strong association with retrieval degradation.

② Distinguishing Harmful Deviations from Beneficial Semantic Structure and Task Imbalance (MSCOCO). Within any workload, the dataset, class balance, and annotated relevance structure are held strictly fixed across encoders. On MSCOCO, each candidate image is genuinely relevant to five caption queries. If a sphere-relative deviation merely reflected beneficial semantic structure or multi-positive imbalance, all encoders on MSCOCO would show high zH1z_{\mathrm{H1}}. Instead, with this relevance structure held fixed, CLIP-SF has much lower zH1z_{\mathrm{H1}} and higher Recall@10 than CLIP on all three seeds (GeoRx detects H1 for CLIP on 3/3 seeds, but for CLIP-SF only on seed 3):

EncoderzH1z_{\mathrm{H1}}, seed1Recall@10, seed1zH1z_{\mathrm{H1}}, seed2Recall@10, seed2zH1z_{\mathrm{H1}}, seed3Recall@10, seed3
CLIP22.36368.2%16.78968.0%15.38068.4%
CLIP-SF1.59388.2%0.97589.6%5.11985.0%

③ Correlation with Retrieval Degradation Across H1–H5 (OVEN & Real Workloads). Across the 270 pair–seed observations (90 encoder–workload pairs × 3 seeds), the 95th percentile of Recall@10 is 86.4% and the median is 41.1%; because multiple deviations (e.g., H1, H2, H5) coexist within a single pair, GeoRx detects 816 deviation instances in total across these low-recall settings (complete data at https://anonymous.4open.science/r/GeoRx_rebuttal-0516/diagnosis_z_recall.csv). On OVEN, the Spearman rank correlation between zH1z_{\mathrm{H1}} and Recall@10 across the nine encoders is -0.92 when averaged over three seeds (-0.93 on seed 1, and -0.90 for raw xH1x_{\mathrm{H1}}). Moreover, deviations on H4 and H5 also directly track actionable retrieval errors: H5 →\rightarrow P-CE is effective on 33.33% (30/90) of diagnosed pairs (boosting Recall@10 by up to +7.6% on OVEN), and H4 →\rightarrow L-CE is effective on 20.00% (2/10) while avoiding severe L-CE degradation (-23.16% mean recall loss) on the remaining 80/90 unflagged pairs.

EncoderzH1z_{\mathrm{H1}}, seed1Recall@10, seed1
SigLIP5710.832.4%
BLIP-21369.368.2%
CLIP-SF93.0650.6%

In response to this comment, we have made the corresponding revisions in Section 3.1, which are marked in Blue, and in Appendix A.2, which are marked in Orange.

AuthorsOpenReview ↗

Response to W2 and Q3

3. W2 & Q3: Baseline Comparisons, Confidence Intervals, the 0.5% Threshold, and Fusion Weight Selection

A3. Thank you for these constructive suggestions. Below, we compare diagnosis-conditioned correction against no correction, always-CSLS, the best fixed single method, and a random mapping, provide 95% confidence intervals and per-pair/per-seed results, justify the 0.5% effectiveness threshold, and clarify how fusion weights are evaluated:

① Comparison Against Selection Baselines. A correction is effective if it improves Recall@10 over the cosine baseline by ≥ 0.5% on at least two of the three seeds (Appendix A.4). Across all 450 pair–method evaluations (90 pairs × 5 methods), CSLS, α-QE, Diffusion, L-CE, and P-CE were effective on 55, 3, 6, 5, and 30 pairs, respectively, yielding 99 effective pair–method combinations in total. GeoRx recommended 272 individual corrections across the 90 pairs and 88 were effective (32.35% hit rate), capturing 88 of the 99 total effective combinations (88.89% effective correction coverage; missing only 11: 2 CSLS, 6 Diffusion, 3 L-CE). To compare multi-label GeoRx recommendations and single-method baselines on a common denominator, the table below reports both Hit rate and Effective correction coverage:

Strategy / BaselineHit rateEffective correction coverage
GeoRx (Diagnosis-conditioned)32.35% (88/272)88.89% (88/99)
Uniform random mapping22.00% (expected)20.00% (expected)
Fixed-CSLS (best fixed single method)61.11% (55/90)55.56% (55/99)
Fixed α-QE3.33% (3/90)3.03% (3/99)
Fixed Diffusion6.67% (6/90)6.06% (6/99)
Fixed L-CE5.56% (5/90)5.05% (5/99)
Fixed P-CE33.33% (30/90)30.30% (30/99)

② 95% Confidence Intervals and Per-Pair/Per-Seed Results. We emphasize that the correction mapping serves as an empirical audit of whether existing methods recover lost recall when their targeted deviation is present (Insight 2). For each <encoder, workload> pair, we determine the diagnosis by majority vote across three seeds. The table below reports the 95% confidence intervals for the hit rates, and complete per-pair and per-seed results (ΔRecall@10\Delta \mathrm{Recall}@10 across all 90 pairs and 3 seeds) are provided at https://anonymous.4open.science/r/GeoRx_rebuttal-0516/diagnosed_mapped_repairs.csv:

Diagnosis–correction mappingEffective / recommendedHit rate95% CI
H1 → CSLS53/8264.63%[53.66%, 75.31%]
H2 → α-QE3/903.33%[0.00%, 8.89%]
H4 → L-CE2/1020.00%[0.00%, 88.89%]
H5 → P-CE30/9033.33%[12.22%, 55.56%]
Overall88/27232.35%[22.43%, 42.05%]

③ Justification of the 0.5% Effectiveness Threshold. Each seed’s correction evaluation set contains ∣Qeval∣|Q_{eval}| = 500 queries (Appendix A.4). Under a naive ΔRecall@10>0%\Delta \mathrm{Recall}@10 \gt 0\% criterion, a single additional successful query (1/500 = 0.2%) would count as an effective correction despite single-query noise. Setting the threshold to 0.5% requires a net gain of at least three successful queries (3/500 = 0.6%) on at least two of the three seeds, ensuring that counted hits reflect non-trivial, reproducible improvements.

④ Selection of Fusion Weights Without Using Evaluation Observations. In our joint correction experiment (Section 4.3, Figure 3(b)), we did not select or tune fusion weights using evaluation observations. Instead, we fixed a uniform, predefined grid wQE∈{0.25,0.50,0.75}w_{\mathrm{QE}} \in \{0.25, 0.50, 0.75\} in advance and reported the full curve across all weights—explicitly showing both the gain at wQE=0.75w_{\mathrm{QE}} = 0.75 (+0.56%) and the degradation at wQE=0.25w_{\mathrm{QE}} = 0.25 (-1.32%) across the four pairs where both individual methods were effective (10 pair–seed observations). This sensitivity is precisely the basis of Insight 2: existing methods lack a mechanism to set fusion weights adaptively from diagnostic scores alone.

In response to this comment, we have made the corresponding revisions in Appendix A.4, which are marked in Orange and Blue.

Reviewer_wr5bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GeoRx, a reference-based benchmark for diagnosing geometric deviations in cross-modal retrieval and guiding existing correction methods. It uses an isotropic spherical reference matched to embedding dimension and corpus size, five diagnostic metrics, and a jointly calibrated conformal threshold. Its contribution is a framework that separates geometric diagnosis, controlled fault validation, and downstream correction evaluation. Experiments cover nine frozen encoders across ten M-BEIR workloads, with three random seeds and disjoint query subsets for fitting, calibration, diagnosis, and correction assessment. The reported results show 98.93%–100% target detection rates under controlled injection and a 32.35% diagnosis–correction mapping hit rate.

Critical strengths

(1)Clear diagnostic focus. GeoRx complements retrieval accuracy benchmarks with five interpretable metrics that characterize distinct geometric patterns without relevance labels.

(2)Separated validation stages. Disjoint fitting, calibration, fault-evaluation, and correction-evaluation subsets reduce leakage and distinguish detection accuracy from downstream correction utility.

(3)Broad empirical coverage. Evaluation spans 90 encoder–workload pairs, three random seeds, and individual and joint correction methods, supporting assessment across diverse retrieval settings.

Critical weaknesses

(1)Reference validity is insufficiently established. Deviations from an isotropic spherical reference may reflect legitimate semantic structure rather than harmful geometry. Conformal calibration does not resolve this distinction.

(2)Controlled injections provide limited attribution evidence. High detection rates demonstrate sensitivity to predefined perturbations, but do not establish that diagnoses identify the causes of naturally occurring retrieval failures.

(3)Correction guidance remains weakly supported. The reported 32.35% mapping hit rate needs comparison with random, fixed-method, and frequency-based selection baselines. Its practical value also depends on mapping precision, missed effective corrections, and actual Recall gains.

Questions for authors

1.How do you distinguish harmful geometric deviations from legitimate semantic clustering, candidate popularity, or multiple relevant results when using the isotropic spherical reference?

2.Under what exchangeability assumptions does the joint conformal threshold control false positives, and do these guarantees extend to real semantic workloads?

3.How well do the controlled injections represent naturally occurring retrieval failures? Do injections selectively affect their target metric, or also trigger other diagnoses?

4.How is the 32.35% diagnosis–correction mapping hit rate defined, and how does it compare with random selection, the most frequently effective correction, and applying a fixed correction universally?

5.Does diagnosis-guided correction deliver consistent Recall improvements across seeds, and what computational overhead does diagnosis add relative to retrieval and correction?

3 comments
AuthorsOpenReview ↗

Response to W3 and Q4

4. W3 & Q4: Definition of the 32.35% Mapping Hit Rate, Baseline Comparisons, Precision, and Actual Recall Gains

A4. Thank you for raising this point. Below, we clarify the definition of the 32.35% diagnosis–correction mapping hit rate (mapping precision), quantify missed effective corrections, compare against random, fixed-method, and frequency-based selection baselines, and report actual Recall gains:

① Definition of Mapping Hit Rate (Mapping Precision) and Missed Effective Corrections. A correction is effective if it improves Recall@10 over the cosine baseline by ≥ 0.5% on at least two of the three seeds (Appendix A.4, marked in Orange). The diagnosis–correction mapping hit rate (mapping precision) is the proportion of individual corrections recommended by GeoRx that meet this effectiveness criterion (counting each recommendation separately when multiple deviations are detected for the same <encoder, workload> pair). Across the 90 pairs (450 total pair–method evaluations across the five methods), CSLS, α-QE, Diffusion, L-CE, and P-CE were effective on 55, 3, 6, 5, and 30 pairs, respectively, yielding 99 effective pair–method combinations in total. GeoRx recommended 272 corrections and 88 were effective, giving a 32.35% hit rate (88/272, with H1 → CSLS reaching 64.63%, 53/82). Out of all 99 effective pair–method combinations, GeoRx recommended 88 (88.89% effective correction coverage). The only 11 missed effective corrections comprised 2 for CSLS, 6 for Diffusion, and 3 for L-CE.

② Comparison with Random, Universal Fixed-Method, and Frequency-Based Baselines. For single-method selection baselines, the hit rate is the proportion of the 90 <encoder, workload> pairs where the selected method is effective. To compare GeoRx (which recommends multiple corrections when multiple deviations coexist) and single-method baselines on a common denominator, we also report the proportion of all 99 effective corrections captured by each strategy (effective correction coverage):

Strategy / BaselineHit rate (Mapping Precision)Effective correction coverage
GeoRx (Diagnosis-guided)32.35% (88/272)88.89% (88/99)
Uniform random selection22.00% (expected)20.00% (expected)
Fixed CSLS (Universal)61.11% (55/90)55.56% (55/99)
Fixed α-QE (Universal)3.33% (3/90)3.03% (3/99)
Fixed Diffusion (Universal)6.67% (6/90)6.06% (6/99)
Fixed L-CE (Universal)5.56% (5/90)5.05% (5/99)
Fixed P-CE (Universal)33.33% (30/90)30.30% (30/99)
Frequency-based (most frequently effective, CSLS)61.11% (55/90)55.56% (55/99)

③ Actual Recall Gains and Role of the Correction Audit. Across the 816 seed-level diagnosed deviations, actual Recall gains (ΔRecall@10) ranged from -51.0% to +23.4% and improved recall in 355 cases (e.g., on OVEN seed 1, H1 → CSLS boosts SigLIP by +11.2 points and OpenCLIP-FFT by +12.0 points; full per-workload results at https://anonymous.4open.science/r/GeoRx_rebuttal-0516/diagnosed_mapped_repairs.csv). Finally, we emphasize that the correction mapping serves primarily as an empirical audit rather than a standalone routing algorithm: existing correction methods assume that an embedding space exhibits the geometric deviation they target, and our results reveal that when multiple deviations coexist, standalone corrections do not always recover the lost recall (Insight 2), motivating adaptive joint correction.

In response to this comment, we have made the corresponding revisions in Appendix A.4, which are marked in Orange.

AuthorsOpenReview ↗

Response to Q5

5. Q5: Cross-Seed Consistency and Computational Overhead

A5. Thank you for this question. ① Consistent Recall improvements across seeds. By definition, a recommended correction counts as a hit only if it improves Recall@10 by at least 0.5% on a majority (≥ 2/3) of the seeds (Appendix A.4). The mapping hit rate (e.g., 64.63% for H1 → CSLS) therefore directly measures how often diagnosis-guided correction delivers consistent Recall improvements across seeds. The table below illustrates Recall@10 before and after applying CSLS under the H1 → CSLS mapping on three representative <encoder, workload> pairs (complete per-seed results for all correction methods are provided at https://anonymous.4open.science/r/GeoRx_rebuttal-0516/diagnosed_mapped_repairs.csv):

<encoder, workload>seed1seed2seed3Effective seeds
<CLIP, OVEN>0.6% → 4.2% (+3.6)0.0% → 2.2% (+2.2)0.2% → 2.6% (+2.4)3/3
<SigLIP, OVEN>2.4% → 13.6% (+11.2)3.2% → 15.6% (+12.4)2.2% → 12.6% (+10.4)3/3
<OpenCLIP-FFT, OVEN>14.4% → 26.4% (+12.0)15.8% → 26.2% (+10.4)14.8% → 23.4% (+8.6)3/3

② Computational overhead. Relative to retrieval and correction, diagnosis adds minimal computational overhead because H1–H4 are computed directly from the already-retrieved top-n (n=50n = 50) candidate lists and H5 uses a small fixed subset of ∣A∣=min(1024,∣C∣)|A| = min(1024, |\mathcal{C}|) candidates (Appendix A.1, Table 4). Moreover, by identifying targeted corrections upfront, diagnosis avoids the substantial computational overhead of exhaustively running every correction method (such as full-graph Diffusion or neural Cross-Encoder reranking) in turn.

In response to this comment, we have made the corresponding revisions in Appendix A.4, which are marked in Orange.

We look forward to your feedback and would be happy to clarify any remaining questions. Thank you for the time and effort you have devoted to reviewing our paper. We wish you all the best.

Dear Reviewer,

Thank you again for your feedback. As the discussion period is coming to a close, we would greatly appreciate it if you could let us know whether our responses and revisions have adequately addressed your concerns. Please let us know if there’s anything further we can provide or discuss. We are thankful for your time and consideration and look forward to your response.

Best regards,

The Authors

Reviewer_ost4OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GeoRx, a benchmark for diagnosing geometric problems in cross-modal embedding spaces. It uses five metrics to detect hubness, local crowding, graph fragmentation, ranking ambiguity, and weak cross-modal interaction. Experiments cover 9 encoders and 10 retrieval workloads. GeoRx achieves high detection rates under controlled single- and dual-deviation injections, but the overall hit rate of diagnosis-guided correction is only 32.35%.

Critical strengths

1、The problem is important. Existing benchmarks mainly report end-to-end metrics such as Recall, which provide limited information about why retrieval fails. GeoRx aims to offer more fine-grained geometric diagnosis. 2、The diagnosis and correction stages are largely separated. Controlled injection is used to evaluate the diagnostic metrics independently of correction performance. 3、The experimental coverage is reasonably broad, including 9 encoders and 10 M-BEIR workloads, resulting in 90 encoder–workload pairs. 4、The paper reports useful negative results. The mapped correction methods do not always improve retrieval, and joint correction is sensitive to fusion weights.

Critical weaknesses

1、The reference model may be overly idealized. GeoRx uses an isotropic spherical distribution without semantic structure. Real embedding spaces contain semantic clusters, class imbalance, and multiple relevant candidates. Therefore, deviation from the spherical reference may not necessarily indicate a real retrieval problem. 2、The controlled injection experiments may overestimate diagnostic ability. The injection operations are designed around the target metrics, such as directly reducing the top-1/top-2 score gap. High detection rates therefore show sensitivity to predefined deviations, but do not fully establish real-world attribution ability. 3、The false-positive analysis on real data is limited. The joint-threshold experiment uses only a few spherical configurations and does not sufficiently demonstrate low false-positive rates across all real encoder–workload pairs. 4、H3 is never detected in the 90 real encoder–workload pairs. As a result, its corresponding correction method cannot be evaluated, and the practical usefulness of this metric remains unclear. 5、The correction results lack more complete statistical analysis. The paper mainly reports average Recall gains and hit rates under a fixed threshold, without confidence intervals, significance tests, or detailed per-workload results.

Questions for authors

How do you distinguish deviation from the random spherical reference from a genuine geometric defect that causes retrieval failure? A candidate may be retrieved by many queries because it is genuinely relevant to multiple queries. How does H1 distinguish legitimate multi-positive behavior from hubness? Could you report detection and false-positive rates under different injection strengths to evaluate whether GeoRx remains effective for weaker deviations? Why is H3 never detected in the real workloads? Are the results sensitive to the kNN value and the 15% component-size threshold? Why does the reference model match only the embedding dimension and corpus size? Do diagnostic window size, query distribution, and top-k settings also affect the results? The overall correction hit rate is relatively low at 32.35%. Does this mainly indicate that existing correction methods are inadequate, or that the diagnosed deviations are not strongly aligned with real retrieval errors?

1 comment

Dear Reviewer,

Thank you again for your feedback. As the discussion period is coming to a close, we would greatly appreciate it if you could let us know whether our responses and revisions have adequately addressed your concerns. Please let us know if there’s anything further we can provide or discuss. We are thankful for your time and consideration and look forward to your response.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Remarks

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair (KiE9) for organizing the discussion and all three reviewers (ost4, wr5b, and C9tM) for their valuable feedback. We are encouraged that all three reviewers noted only "minor or fixable concerns" on soundness, and recognized that GeoRx addresses an "important" problem beyond end-to-end accuracy (ost4), "complements retrieval accuracy benchmarks with five interpretable metrics that characterize distinct geometric patterns without relevance labels" (wr5b), separates diagnosis from correction so that it "avoids an obvious circular evaluation loop" (C9tM) and "reduce[s] leakage" (wr5b), adopts a scale-matched spherical reference as "a sensible and practically motivated design choice" (C9tM), and provides "broad" empirical coverage across 90 encoder–workload pairs that "reports useful negative results" (ost4, wr5b, C9tM).

Reviewer ost4 (Geometric Diagnosis, Sensitivity, Statistical Analysis, and Hit rate): To distinguish geometric defects from semantic structure and multi-positive relevance, we held relevance structure strictly fixed on MSCOCO (5 captions/image). Under this control, CLIP exhibits strong hubness (zH1∈[15.38,22.36]z_{H1} \in [15.38, 22.36] and Recall@10 ≤\le 68.4%) whereas CLIP-SF stays below threshold on 2/3 seeds (zH1∈[0.98,5.12]z_{H1} \in [0.98, 5.12] and Recall@10 85.0%--89.6%). On OVEN, xH1x_{H1} strongly tracks retrieval degradation (ρ=−0.90\rho = -0.90). Across 1,350 single-target vector-space injections (t∈{0.25,0.50,0.75,1.00}t \in \{0.25, 0.50, 0.75, 1.00\}), GeoRx achieves 99.83% detection (594/595) even at t=0.25t=0.25 and 100.0% at t≥0.50t \ge 0.50. Once formed, H4 injections drop Recall@10 from 97.71% to 0.00% on <GME, WebQA>. We also verified H3's negative-filter role across k∈{3,5,8,10}k \in \{3, 5, 8, 10\}: real fragments stay ≤\le 4.34% with 0/270 exceeding τ\tau at k=10k=10 and a mean Diffusion ΔRecall@10\Delta\text{Recall@10} of -0.09%. Our diagnoses remain consistent across window sizes (∣Q∣∈{128,192,256,384}\vert{}Q\vert{} \in \{128, 192, 256, 384\}), query seeds (78/90 identical), and evaluation cutoffs (k∈{1,5,10}k \in \{1, 5, 10\}). We further added 95% CIs for all hit rates. Although the overall correction hit rate is 32.35% (95% CI [22.43%, 42.05%]), individual mappings achieve high hit rates: H1$\to$CSLS reaches 64.63% (53/82 pairs, 95% CI [53.66%, 75.31%]) and H5$\to$P-CE reaches 33.33% (30/90 pairs).

Reviewer wr5b (Reference Assumptions, Attribution, and Correction Utility): We explicitly formalized the conformal exchangeability assumption over ncal=32n_{cal}=32 spherical calibration windows (Pr⁡(Unew>τ)≤1/33\Pr(U_{new} > \tau) \le 1/33; empirical FPR 1.04% vs. 32.29% independent) and clarified in Section 4.6 and Appendix A.2 that this theoretical bound applies to spherical null windows rather than real semantic workloads, where zmz_m acts as a fixed reference ruler. We defined the 32.35% hit rate as micro-averaged mapping precision (88/272), quantified 88.89% effective correction coverage (88/99 across all 450 pair–method evaluations; 11 missed: 2 CSLS, 6 Diffusion, 3 L-CE), benchmarked against uniform random (22.00% hit rate / 20.00% coverage), universal fixed methods, and frequency-based selection (Fixed-CSLS: 61.11% / 55.56%), and documented cross-seed consistency (≥2/3\ge 2/3 seeds, e.g., +8.6+8.6 to +12.4+12.4 pts on OVEN across 3/3 seeds) and minimal diagnostic overhead on top-5050 lists and ∣A∣≤1024\vert{}A\vert{} \le 1024.

Reviewer C9tM (Injection Formulas, Selection Baselines, and Follow-Up Discussion): We provided detailed formulas and pseudocode for all five injection operators (H1–H5, Appendix A.3), and then reconciled the H3 formation vs. diagnostic criteria. We demonstrated the effectiveness of the 0.5% threshold on correction evaluation set with ∣Qeval∣=500\vert{}Q_{\text{eval}}\vert{}=500, in that ≥3\ge 3 net queries are required on ≥2/3\ge 2/3 seeds. We confirmed that fusion weights (wQE∈{0.25,0.50,0.75}w_{QE} \in \{0.25, 0.50, 0.75\}) were fixed a priori without evaluation leakage. In our follow-up on selection baselines, we and Reviewer C9tM noted that comparing multi-method GeoRx against single-method Fixed-CSLS is not fair due to different search spaces. Under matched-space comparisons, GeoRx reduces the evaluated set in both settings: it cuts five-method evaluations by 39.6% (450 to 272) while improving hit rate (32.35% vs. 22.00%) with 88.89% (88/99) coverage, and narrows H1$\to$CSLS (82 vs. 90) while outperforming Fixed-CSLS (64.63% vs. 61.11%) with 96.36% (53/55) coverage. Meanwhile, multi-method mapping tests whether standalone correction fixes improve recall under coexisting deviations (Insight 2).

In summary, we have carefully addressed all reviewer concerns point by point and updated all clarifications and related experiments into the revision. We respectfully hope these clarifications and related experiments will be considered in the final decision.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.