- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates how alignment between text embedding models behaves under two deployment regimes: homogeneous indexes where all documents are encoded by the same model and mixed-source indexes where documents are encoded by different models. Using seven frozen text encoders, the authors show a consistent reversal: a shared nonlinear correction on top of Procrustes alignment (GCPA) degrades retrieval in homogeneous settings but improves it in mixed-source settings across four freshly encoded BEIR datasets and a large Natural Questions index of historical vectors. They further disentangle within-source ordering versus cross-source score comparability via a controlled “score transplant” diagnostic and explore a practical, query-adaptive per-source score calibration that partially recovers the mixed-index gains but with additional serving cost.
Critical strengths
- Identifies and rigorously substantiates a deployment-dependent reversal: a method that harms homogeneous retrieval can improve mixed-source retrieval when cross-source score competition is required.
- Introduces a careful score-exchange diagnostic that isolates the effect of cross-source score comparability from within-source ordering without using relevance labels.
Critical weaknesses
- Whether this mixed-retrieval scenario actually exists in real-world use remains uncertain, which limits the significance of this study.
- The shared-space dimension reduction (to 768) in GPA/GCPA may confound conclusions about geometry preservation; sensitivity to hub dimension is not fully explored.
Questions for authors
- What is the starting point for studying this problem, and are there concrete application scenarios behind it?
- How sensitive are the homogeneous and mixed-source findings to the shared-space width h? Could a larger h reduce the homogeneous penalty for GCPA while retaining mixed-source gains?
Thank you for your review and the questions!
On Q1
The starting point is that an index outlives its encoder. Hosted embedding models can be retired because of number of reasons, and re-embedding a large collection takes long enough that vector-database vendors publish migration procedures for it. Their current advice is to keep two complete sets of vectors until re-embedding finishes. A mixed index is the alternative to that cost, and in three situations it is not optional:
The first case is what our new time-ordered experiment simulates. Documents are assigned by creation order, the newest to the new encoder. With 25% of an Android corpus on the new encoder (averaged over six encoder pairs), linear alignment alone gives nDCG@10 0.228, although the same index scores 0.487 when all documents use the old encoder. GCPA gives 0.355, direct conversion 0.385, and the two corrections introduced in the revision together 0.473. So the mixed state is where alignment quality is lost and where the choice of method matters most.
On your Q2
A larger width does not reduce the homogeneous penalty. GCPA − GPA in nDCG@10 (H = homogeneous cross-encoder, X = mixed):
Widths above 768 cannot include EmbeddingGemma, whose native width is 768, so they use smaller cohorts, each with its own 768 baseline. Key results are:
Finally, the question of a correction that keeps the mixed gain without the homogeneous penalty has a positive answer by a different route. A per-source affine corrector exceeds GCPA's mixed gain and does not lose on cross-encoder homogeneous retrieval.
On presentation
We have shortened the main text by moving audit and provenance detail to the appendix and stating each experiment's purpose before its procedure.