- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies embedding translation for mixed-index retrieval. Instead of re-encoding a corpus after switching a model, a map fitted on anchor documents that are encoded by both models converts the remaining old vectors into the new space, where they are searched together with the native ones. The main claim is that the usual alignment metrics don't predict well how such a mixed index retrieves. The authors prove a necessary and sufficient condition based on per-document error bounds and query-dependent ranking gaps, and show two equally well-aligned maps that return different results. The authors introduce ETBench to evaluate alignment methods for embedding translation (24 encoder pairs, 4 BEIR corpora, 9 translators). The paper also includes several side studies.
Critical strengths
The evaluation protocol catches failures that aggregate scores hide, which I think is the paper's most useful contribution. Procrustes has a recovery of 90% but keeps nDCG@10 under 30% on translated documents, and the identity control scores a respectable-looking 0.66 overall while its translated documents are essentially invisible (Table 1). Anyone migrating an index should run exactly this kind of check.
The study is broad and transparent, every per-pair result is in the appendix. Fig. 2a has a good practical point: alignment quality is mostly a property of the encoder pair, while retrieval quality depends heavily on the corpus, so good alignment on one dataset tells you little about another.
Critical weaknesses
The experiments do not really follow the theory. Theorem 1(a) is the standard interval-separation argument (the appendix proof says so), its "necessary" direction only holds for an adversarially chosen map rather than a fitted one, and the construction in (b) relies on anchors that don't span the space. In practice the condition seems out of reach. Even the best held-out cosine in Table 1 (0.978, for relative) corresponds to an error of about 12 degrees, while neighboring results around rank 10 are usually less than a degree apart. So the guarantee doesn't apply to any method tested whether it retrieves well or badly. And the benchmark measures relevance retention, not what the theorem is about, projected errors against ranking gaps, or overlap with the native top-k. So the paper shows that procrustes and GW fail, but doesn't show why, and doesn't offer diagnostic for it.
Questions for authors
- Do the theorem's own quantities (say, the share of queries satisfying Eq. (2), or projected error relative to the rank-10 gap) predict translated-document retention better than Hits@1 or cosine? A clear yes would make the theory a usable diagnostic and raise my score.
- How much of the procrustes and GW loss goes away with a simple per-query score offset estimated on held-out anchors? If most of it does, this is largely a calibration problem and I'd find the paper less informative; if not, the main claim would get stronger
8 comments
Reviewer 74vd : do you find your concerns addressed (particularly the gap between the experiments and the theory)?
Thank you for running both analyses.
Q2 is resolved for me, and the result strengthens the main claim. Q1 is only partly resolved. You used full-index retention for the correlation, but my question was about translated documents. After pairing your certificate ratios with the D1 nDCG@10 retention column of Table 1, I got Spearman 0.65, against 0.70 for held-out cosine. So across these 10 methods, the condition doesn’t predict retention of translated documents better than cosine.
Could you compute the same correlation against D1 retention across the 96 pair-corpus cells for each method? I think 10 method averages are too few to settle this.
Dear reviewer 74vd,
Thank you for the thoughtful review and the helpful suggestions. We are glad that you consider our evaluation protocol's ability to catch failures hidden by aggregate scores the paper's most useful contribution. We address your concerns and questions below.
Thank you for highlighting this gap in the manuscript. Theorem 1(a) was introduced to clarify when embedding alignment suffices for embedding translation. We didn’t include certification in the evaluation since it additionally requires target queries and corpus-wide query-document comparisons, making its computational cost comparable to exhaustive retrieval.
However, we do find strong correlation between the condition of Theorem 1(a) and retention metrics in evaluation. Specifically, we now examine the connection between the certification rate ( the percentage of evaluated queries satisfying Eq. (2) of Theorem 1(a) with k=10) and the real top-10 answer preservation as well as nDCG@10 retention used in the evaluation, as shown in the table blow. We can see that Theorem 1(a) certificate ratio correlates significantly more strongly with nDCG@10 retention (Spearman ρ=0.903) than held-out alignment metrics such as Hits@1 (0.309) or cosine (0.212), and positively with observated top-10 preservation (0.685). Consistent with Theorem 1(a), these results highlight the importance of query-specific ranking gaps in assessing retrieval quality beyond alignment scores alone.
Per-query offset calibration does not fully resolve the retrieval failures of Procrustes and GW. For each query, we estimate the offset as the mean translated-minus-native target score difference over held-out anchors, then subtract it from all translated-document scores while leaving native scores unchanged. As shown below, this correction does not consistently improve retrieval across methods. We also evaluate oracle calibration, which uses test relevance labels to select the offset maximizing each query’s full-index nDCG@10. We can see that Procrustes and GW achieve only 68.72% and 67.92% retention, respectively, indicating that a query-specific score bias alone cannot fully explain their retrieval losses.
Best regards,
The authors from Submission4958