- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GeoRx is a reference-based benchmark for attributing geometric patterns in cross-modal embedding spaces. For each encoder-workload pair, it constructs an isotropic unit-sphere reference matched to the embedding dimension and corpus size, then compares five unsupervised statistics against that reference: global hubness (H1), local-neighborhood concentration (H2), local graph fragmentation (H3), top-1/top-2 score-margin ambiguity (H4), and a query-specific interaction-energy ratio (H5). Robust centers and scales are estimated from reference windows, and a joint conformal threshold is used to report a set of detected deviations. The experiments cover nine encoders and ten M-BEIR workloads, yielding 90 encoder-workload pairs with three random seeds. The authors validate the diagnostic metrics using controlled single- and dual-deviation injections, report 100.00% target detection for single injections and 98.93% for dual injections, and evaluate mappings from H1-H5 to CSLS, alpha-QE, diffusion, listwise cross-encoders, and pointwise cross-encoders. They also report a 32.35% overall diagnosis-to-correction hit rate and ablations on reference matching and joint threshold calibration.
Critical strengths
The paper separates diagnosis of embedding-space statistics from end-to-end retrieval scores and does not use correction outcomes to define the diagnostic set, which avoids an obvious circular evaluation loop. Broad and reasonably structured evaluation. The study spans nine encoders, ten workloads, three seeds, disjoint fitting/calibration/evaluation subsets, and an explicit threshold ablation. Matching the reference to both dimension and corpus size is a sensible and practically motivated design choice.
Critical weaknesses
-
A deviation from an isotropic null can reflect legitimate semantic structure, task imbalance, or multimodal data properties rather than a retrieval pathology. The paper does not establish that H2, H4, or H5 deviations predict retrieval failures, nor that the reported false-positive control transfers from held-out spherical windows to real encoder spaces. The 1.04% false-positive result may be therefore much narrower than the practical claim suggested by the benchmark framing.
-
The main validation is metric-aligned and the correction utility is not yet demonstrated. The controlled injections are designed around the same five statistics and use a single shared strength of 0.75; exact operators for H1-H4 are not specified in Appendix A.3. In addition, the 32.35% mapping hit rate has no always-one-method, random, or best-single-method baseline, uses an arbitrary Delta Recall@10 >= 0.5% criterion, reports no uncertainty, and the joint experiment covers only four pairs and ten pair-seed observations. H3 cannot be evaluated because it is never diagnosed in the real benchmark.
Questions for authors
- Please provide exact transformation formulas or pseudocode for all five injection operators, especially H1-H4, together with the number of successfully formed targets and sensitivity to injection strength and random seed. How is the H3 formation criterion (nonzero second-largest component in Appendix A.3) reconciled with the diagnostic requirement x_H3 >= 0.15 in Section 3.2?
- Can you evaluate the diagnostic set on independently generated non-spherical embedding spaces or naturally occurring perturbations with known ground truth, and report its correlation with retrieval degradation? How do you distinguish a sphere-relative deviation from beneficial semantic structure in a real workload?
- Please report per-pair and per-seed results with confidence intervals, and compare diagnosis-conditioned correction against no correction, always-CSLS, the best fixed single method, and a random mapping. Please justify the 0.5% effectiveness threshold and state how fusion weights are selected without using the reported evaluation observations.
9 comments
Response to W1 and Q2.
2. W1 & Q2: Distinguishing Sphere-Relative Deviations from Beneficial Semantic Structure, False-Positive Transfer, and Correlation with Retrieval Degradation
A2. Thank you for raising this point. In the embedding spaces we study, queries and candidates lie on the unit hypersphere after L2 normalization (Section 3.1, marked in Blue). Thus, we use the isotropic spherical distribution as a scale-calibrated null baseline (matched to d and ) rather than an ideal semantic space, and evaluate non-isotropic perturbations via both controlled injections (with known ground truth) and unperturbed real workloads:
① Transfer of False-Positive Control to Real Encoder Spaces. Under the spherical reference, the ground-truth diagnostic set is strictly empty, allowing Table 2 to isolate false alarms (1.04% for joint conformal thresholding vs. 32.29% for independent thresholding). As noted in Appendix A.2 (marked in Orange), this 1.04% false-positive rate applies to spherical reference windows and does not claim a false-positive rate for real encoder spaces where the ground-truth deviation set is unknown. Instead, in real spaces, measures how far the space deviates from the spherical reference, and we validate these diagnoses by their strong association with retrieval degradation.
② Distinguishing Harmful Deviations from Beneficial Semantic Structure and Task Imbalance (MSCOCO). Within any workload, the dataset, class balance, and annotated relevance structure are held strictly fixed across encoders. On MSCOCO, each candidate image is genuinely relevant to five caption queries. If a sphere-relative deviation merely reflected beneficial semantic structure or multi-positive imbalance, all encoders on MSCOCO would show high . Instead, with this relevance structure held fixed, CLIP-SF has much lower and higher Recall@10 than CLIP on all three seeds (GeoRx detects H1 for CLIP on 3/3 seeds, but for CLIP-SF only on seed 3):
| Encoder | , seed1 | Recall@10, seed1 | , seed2 | Recall@10, seed2 | , seed3 | Recall@10, seed3 |
|---|---|---|---|---|---|---|
| CLIP | 22.363 | 68.2% | 16.789 | 68.0% | 15.380 | 68.4% |
| CLIP-SF | 1.593 | 88.2% | 0.975 | 89.6% | 5.119 | 85.0% |
③ Correlation with Retrieval Degradation Across H1–H5 (OVEN & Real Workloads). Across the 270 pair–seed observations (90 encoder–workload pairs × 3 seeds), the 95th percentile of Recall@10 is 86.4% and the median is 41.1%; because multiple deviations (e.g., H1, H2, H5) coexist within a single pair, GeoRx detects 816 deviation instances in total across these low-recall settings (complete data at https://anonymous.4open.science/r/GeoRx_rebuttal-0516/diagnosis_z_recall.csv). On OVEN, the Spearman rank correlation between and Recall@10 across the nine encoders is -0.92 when averaged over three seeds (-0.93 on seed 1, and -0.90 for raw ). Moreover, deviations on H4 and H5 also directly track actionable retrieval errors: H5 P-CE is effective on 33.33% (30/90) of diagnosed pairs (boosting Recall@10 by up to +7.6% on OVEN), and H4 L-CE is effective on 20.00% (2/10) while avoiding severe L-CE degradation (-23.16% mean recall loss) on the remaining 80/90 unflagged pairs.
| Encoder | , seed1 | Recall@10, seed1 |
|---|---|---|
| SigLIP | 5710.83 | 2.4% |
| BLIP-2 | 1369.36 | 8.2% |
| CLIP-SF | 93.06 | 50.6% |
In response to this comment, we have made the corresponding revisions in Section 3.1, which are marked in Blue, and in Appendix A.2, which are marked in Orange.
Response to W2 and Q3
3. W2 & Q3: Baseline Comparisons, Confidence Intervals, the 0.5% Threshold, and Fusion Weight Selection
A3. Thank you for these constructive suggestions. Below, we compare diagnosis-conditioned correction against no correction, always-CSLS, the best fixed single method, and a random mapping, provide 95% confidence intervals and per-pair/per-seed results, justify the 0.5% effectiveness threshold, and clarify how fusion weights are evaluated:
① Comparison Against Selection Baselines. A correction is effective if it improves Recall@10 over the cosine baseline by ≥ 0.5% on at least two of the three seeds (Appendix A.4). Across all 450 pair–method evaluations (90 pairs × 5 methods), CSLS, α-QE, Diffusion, L-CE, and P-CE were effective on 55, 3, 6, 5, and 30 pairs, respectively, yielding 99 effective pair–method combinations in total. GeoRx recommended 272 individual corrections across the 90 pairs and 88 were effective (32.35% hit rate), capturing 88 of the 99 total effective combinations (88.89% effective correction coverage; missing only 11: 2 CSLS, 6 Diffusion, 3 L-CE). To compare multi-label GeoRx recommendations and single-method baselines on a common denominator, the table below reports both Hit rate and Effective correction coverage:
| Strategy / Baseline | Hit rate | Effective correction coverage |
|---|---|---|
| GeoRx (Diagnosis-conditioned) | 32.35% (88/272) | 88.89% (88/99) |
| Uniform random mapping | 22.00% (expected) | 20.00% (expected) |
| Fixed-CSLS (best fixed single method) | 61.11% (55/90) | 55.56% (55/99) |
| Fixed α-QE | 3.33% (3/90) | 3.03% (3/99) |
| Fixed Diffusion | 6.67% (6/90) | 6.06% (6/99) |
| Fixed L-CE | 5.56% (5/90) | 5.05% (5/99) |
| Fixed P-CE | 33.33% (30/90) | 30.30% (30/99) |
② 95% Confidence Intervals and Per-Pair/Per-Seed Results. We emphasize that the correction mapping serves as an empirical audit of whether existing methods recover lost recall when their targeted deviation is present (Insight 2). For each <encoder, workload> pair, we determine the diagnosis by majority vote across three seeds. The table below reports the 95% confidence intervals for the hit rates, and complete per-pair and per-seed results ( across all 90 pairs and 3 seeds) are provided at https://anonymous.4open.science/r/GeoRx_rebuttal-0516/diagnosed_mapped_repairs.csv:
| Diagnosis–correction mapping | Effective / recommended | Hit rate | 95% CI |
|---|---|---|---|
| H1 → CSLS | 53/82 | 64.63% | [53.66%, 75.31%] |
| H2 → α-QE | 3/90 | 3.33% | [0.00%, 8.89%] |
| H4 → L-CE | 2/10 | 20.00% | [0.00%, 88.89%] |
| H5 → P-CE | 30/90 | 33.33% | [12.22%, 55.56%] |
| Overall | 88/272 | 32.35% | [22.43%, 42.05%] |
③ Justification of the 0.5% Effectiveness Threshold. Each seed’s correction evaluation set contains = 500 queries (Appendix A.4). Under a naive criterion, a single additional successful query (1/500 = 0.2%) would count as an effective correction despite single-query noise. Setting the threshold to 0.5% requires a net gain of at least three successful queries (3/500 = 0.6%) on at least two of the three seeds, ensuring that counted hits reflect non-trivial, reproducible improvements.
④ Selection of Fusion Weights Without Using Evaluation Observations. In our joint correction experiment (Section 4.3, Figure 3(b)), we did not select or tune fusion weights using evaluation observations. Instead, we fixed a uniform, predefined grid in advance and reported the full curve across all weights—explicitly showing both the gain at (+0.56%) and the degradation at (-1.32%) across the four pairs where both individual methods were effective (10 pair–seed observations). This sensitivity is precisely the basis of Insight 2: existing methods lack a mechanism to set fusion weights adaptively from diagnostic scores alone.
In response to this comment, we have made the corresponding revisions in Appendix A.4, which are marked in Orange and Blue.
Response to Q1.
A1. We sincerely thank the reviewer for these detailed questions. Rather than editing similarity scores directly, our operators perturb L2-normalized embeddings (unit(v)=v∣∣v∣∣2) with strength t∈[0,1] and recompute full-corpus retrieval from scratch. We provide detailed pseudocode (Appendix A.3, marked in Orange) for all five injection operators (H1–H5):
Let cpool=unit((1/∣C∣)∑jcj) be the mean corpus direction:
H1. Select hub chub=argmax(cj)[cjTcpool]; update queries via qi′=unit((1−t)qi+tchub).
H2. Select support set S closest to cpool with center cS=unit(∑j∈Scj); update cj′=unit((1−t)cj+tcS)(∀j∈S) and qi′=unit((1−t)qi+tcS).
H3. Split candidate subset into G+, G− along axis u, shift via unit(cj±tu), contract toward group centers g+, g−, and align queries with midpoint m=unit(g++g−) orthogonal to u.
H4. Select top-2 candidates ca, cb closest to cpool with midpoint o=unit(ca+cb); update cj′=unit((1−t)cj+to) for j∈{a,b}, and project out the query component parallel to δ=unit(ca′−cb′) from v=unit((1−t)qi+to) via qi′=unit(v−(vTδ)δ).
H5. Let qˉ=unit(∑q∈Qtrainq); update queries via qi′=unit((1−t)qi+tqˉ).
Formed Targets and Sensitivity to Injection & Random Seed. We evaluated 1,350 single-target injections across different injection strengths t∈{0.25,0.50,0.75,1.00} (90 pairs × 5 metrics × 3 seeds). Results are highly consistent across random seeds (e.g., 100% formation across all 3 seeds at t≥0.75, and 78/90 uninjected pairs yield identical diagnostic sets across all 3 seeds). Even at t=0.25, GeoRx detects 99.83% (594/595) of formed targets:
Here, the detection rate is the proportion of formed targets that GeoRx detects, while the false-positive rate measures detections among targets that did not yet meet the strict 90%-query formation cutoff. Moreover, formed injections directly cause end-to-end Recall@10 degradation: under H4 injection (averaged over 3 seeds below), recall changes little when H4 fails to form (t=0.25), but drops sharply once formed (t≥0.50):
Reconciling the Two H3 Criteria (Appendix A.3 vs. Section 3.2) and Evaluating H3. The formation criterion (Appendix A.3) tests whether a structural split has occurred after injection: for at least 90% of queries, the kNN graph on Ln(q) must split into ≥ 2 connected components (i.e., nonzero second-largest component). The diagnostic criterion (Section 3.2) additionally requires the split to be non-trivial in size (xH3≥0.15) along with zH3>τ to filter out tiny peripheral fragments in real spaces. Across the 270 seed-level evaluations on the 90 real pairs, 136 had a nonzero secondary component (xH3>0), but the maximum was only 4.34% and no zH3 ever exceeded τ (0/270)—meaning even without the xH3≥0.15 filter, H3 is 0/270 at k=10. Evaluating its mapped correction (Diffusion) across all 90 pairs yields a near-zero mean ΔRecall@10 of -0.09% (Figure 3(a)), confirming H3's utility in ruling out non-existent fragmentation.