acceptodds
Under review as a conference paper at ICLR 2027

When Alignment Becomes an Interface: Deployment-Dependent Interoperability of Text Embeddings

Abstract

Search indexes can outlive the embedding models that created them, leaving documents represented by different encoders. Alignment makes these vectors comparable, but does a method that works well for one document encoder also work when several encoders' documents compete? We study seven frozen text encoders, comparing linear alignment with a shared nonlinear correction across four freshly encoded retrieval datasets and stored vectors for 2.68 million Natural Questions documents. Each document contributes only one encoder's vector. The same correction lowers retrieval quality when all documents use one encoder, yet raises it when documents use a balanced mixture of encoders. On a prospective duplicate-question test, the top-10 relevance score falls by 0.016 in the first setting and rises by 0.103 in the second, on a 0-1 scale. Exchanging score values while preserving each source's document order shows how better competition between sources can outweigh worse ordering within them. The prospective test supports both patterns after adjustment for multiple comparisons. A simpler alternative, standardizing each query's scores against reference texts, improves linear alignment on all four fresh datasets. However, it generally trails nonlinear correction in quality and takes more search time locally. These findings show why alignment should be evaluated on the index it will serve: preserving order within one source and comparing scores across sources are distinct requirements with different quality and cost tradeoffs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Text Embeddings, Representation Alignment, Model Interoperability, Information Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QynSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how alignment between text embedding models behaves under two deployment regimes: homogeneous indexes where all documents are encoded by the same model and mixed-source indexes where documents are encoded by different models. Using seven frozen text encoders, the authors show a consistent reversal: a shared nonlinear correction on top of Procrustes alignment (GCPA) degrades retrieval in homogeneous settings but improves it in mixed-source settings across four freshly encoded BEIR datasets and a large Natural Questions index of historical vectors. They further disentangle within-source ordering versus cross-source score comparability via a controlled “score transplant” diagnostic and explore a practical, query-adaptive per-source score calibration that partially recovers the mixed-index gains but with additional serving cost.

Critical strengths
  1. Identifies and rigorously substantiates a deployment-dependent reversal: a method that harms homogeneous retrieval can improve mixed-source retrieval when cross-source score competition is required.
  2. Introduces a careful score-exchange diagnostic that isolates the effect of cross-source score comparability from within-source ordering without using relevance labels.
Critical weaknesses
  1. Whether this mixed-retrieval scenario actually exists in real-world use remains uncertain, which limits the significance of this study.
  2. The shared-space dimension reduction (to 768) in GPA/GCPA may confound conclusions about geometry preservation; sensitivity to hub dimension is not fully explored.
Questions for authors
  1. What is the starting point for studying this problem, and are there concrete application scenarios behind it?
  2. How sensitive are the homogeneous and mixed-source findings to the shared-space width h? Could a larger h reduce the homogeneous penalty for GCPA while retaining mixed-source gains?
1 comment

Thank you for your review and the questions!

On Q1

The starting point is that an index outlives its encoder. Hosted embedding models can be retired because of number of reasons, and re-embedding a large collection takes long enough that vector-database vendors publish migration procedures for it. Their current advice is to keep two complete sets of vectors until re-embedding finishes. A mixed index is the alternative to that cost, and in three situations it is not optional:

  • New documents arrive after the old encoder is retired. They can only be encoded with the new model, while older documents still carry old vectors.
  • Some documents can never be re-encoded, because the text was not retained.
  • Collections embedded by different teams or vendors are searched together.

The first case is what our new time-ordered experiment simulates. Documents are assigned by creation order, the newest to the new encoder. With 25% of an Android corpus on the new encoder (averaged over six encoder pairs), linear alignment alone gives nDCG@10 0.228, although the same index scores 0.487 when all documents use the old encoder. GCPA gives 0.355, direct conversion 0.385, and the two corrections introduced in the revision together 0.473. So the mixed state is where alignment quality is lost and where the choice of method matters most.

On your Q2

A larger width does not reduce the homogeneous penalty. GCPA − GPA in nDCG@10 (H = homogeneous cross-encoder, X = mixed):

EncodershSciFact H / XFiQA H / XNFCorpus H / XAndroid H / X
all seven256−0.024 / +0.108+0.009 / +0.079−0.024 / +0.009+0.005 / +0.092
all seven512−0.038 / +0.120−0.000 / +0.105−0.025 / +0.016−0.008 / +0.106
all seven768−0.044 / +0.120−0.010 / +0.112−0.029 / +0.017−0.016 / +0.108
six (no Gemma)768−0.018 / +0.119+0.010 / +0.091−0.016 / +0.025+0.007 / +0.100
six (no Gemma)1024−0.020 / +0.123+0.005 / +0.098−0.018 / +0.022+0.010 / +0.110
three largest768−0.005 / +0.077+0.044 / +0.087+0.001 / +0.019+0.026 / +0.083
three largest2560−0.019 / +0.096+0.004 / +0.087−0.019 / +0.015−0.003 / +0.090

Widths above 768 cannot include EmbeddingGemma, whose native width is 768, so they use smaller cohorts, each with its own 768 baseline. Key results are:

  • The mixed gain is positive in all 36 cells we ran, on both metrics.
  • Increasing h never improves the homogeneous difference. From 768 to 2560 in the three-encoder cohort it falls on all four datasets (by 0.014 to 0.040).
  • Dimension reduction is not the cause of the penalty. EmbeddingGemma is not truncated at all at h = 768, and GPA leaves its same-encoder SciFact score unchanged at 0.787, yet GCPA lowers it to 0.743. The penalty comes from the corrector.
  • The penalty depends on the encoder cohort, which we had not reported. With six encoders, or the three largest, GCPA is better than GPA homogeneously on FiQA and Android. The revision states the finding in the form that holds in every cell. The correction helps mixed retrieval far more than homogeneous retrieval.

Finally, the question of a correction that keeps the mixed gain without the homogeneous penalty has a positive answer by a different route. A per-source affine corrector exceeds GCPA's mixed gain and does not lose on cross-encoder homogeneous retrieval.

On presentation

We have shortened the main text by moving audit and provenance detail to the appendix and stating each experiment's purpose before its procedure.

Reviewer_2XztOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies retrieval over indexes where documents are embedded by different models. Using seven frozen text encoders, the authors compare Procrustes alignment (GPA) with GCPA, which adds a shared nonlinear corrector. GCPA is slightly worse than GPA when all documents come from one encoder, but it's better when documents are spread across all seven, often by a big margin. This holds on SciFact, FiQA, NFCorpus, a prospective CQADupStack test and a 2.7M-document NQ index. GCPA's advantage seems to come from making scores from different models easier to compare. Rescaling GPA's scores gets part of the way there but it's slower.

Critical strengths

The central result is solid. The flip shows up in every condition tested, including the prospective Android test with corrected thresholds and the large NQ index, and the matched-weight comparison on NFCorpus rules out an averaging artifact. The numbers I spot-checked are consistent. It's careful empirical work.

The score-swapping experiment in Section 5 is a good way to separate ranking within an encoder from competition between encoders. Keeping GPA's order but using GCPA's score values gives the best of the 4 combinations on all 4 fresh datasets. That is a simple and convincing way to show where GCPA's gain comes from.

Critical weaknesses

The contribution is narrow, and the main lesson isn't new. By the authors own framing contribution is the "observed reversal". One empirical finding about one corrector against one linear baseline, with no explanation of why GCPA's scores compete better. The general message that doing well on a single-source index doesn't mean doing well on a mixed one is what the image-retrieval backfilling literature found when old and new embeddings share an index. Hot-Refresh (Zhang et al. ICLR 2022) attributes the regression during backfilling to new-to-old positive pairs scoring below new-to-new negatives. FastFill (Jaeckle et al. ICLR 2023) and Seo et al. (WACV 2025) study mixed indices during partial backfilling. These works are not cited.

I'm not convinced by the practical takeaways. All headline numbers use a balanced random split over seven encoders, the most mixed setting possible. A real migration usually involves two models, with a split that follows time or topic. The only two-encoder experiment (Fig. 7, one seed, no error bars) shows a much smaller effect on FiQA: roughly 0.03 at 25%, and gone by 75%. SciFact holds up better. The paper also argues that C1 is slower than GCPA because it needs seven separate indexes. But the average C1 subtracts can be built into the stored vectors in advance so that part works in one normal index. Until this cheaper version is tested the quality-cost tradeoff in Section 6 isn't established.

Questions for authors
  1. How do two cheaper variants compare with GCPA in quality and latency: offset-only C1 in a single index, and C1 using statistics from the stored corpus rather than the Dolma bank? If either gets close to GCPA at GPA's cost, the Section 6 conclusion doesn't hold and I would lower my score. If both clearly trail GCPA this concern goes away.
  2. Could you report two-encoder mixes at several fractions, ideally with a split that follows topic or time? A clear reversal there would make me more positive; if it disappears, the headline only holds for seven random mixes.
1 comment

Thank you for your review! We spent some time exploring, thinking and running experiments related to your two questions and the answers have changed the paper a bit.

For Q1 on cheaper calibration variants:

You are right that C1's mean term folds into the stored vectors and its per-source offset is z_q · m_b, so offset-only C1 is an ordinary inner product against z_d − m_b in one index. We tested that and the corpus-statistic variant you proposed and for seven-source balanced index, the nDCG@10:

SystemIndexSciFactFiQANFCorpusAndroid
GPAone0.4610.2480.2650.251
Offset-only C1, Dolma bank (variant 1)one0.4620.2670.2650.271
Centering + fixed scale, stored-corpus statisticsone0.5720.3170.3090.338
GCPAone0.5820.3600.2810.353
C1 with stored-corpus statistics (variant 2)rescoring0.6340.3810.3210.433

So in a nutshell:

  • Variant 1 fails with the Dolma bank it recovers almost none of GCPA's gain.
  • Variant 2 beats GCPA on all four datasets, and on the NQ index (0.383 against 0.363).
  • The statistic source is what matters. 64 in-domain documents per encoder give the single-index variant within 0.003 of the full-corpus version.
  • On latency: (FiQA, median, cached native query vectors): GPA 1.32 ms, the single-index corpus variant 1.54 ms, GCPA 1.81 ms, corpus C1 by retrieving 200 from one index and rescoring 2.37 ms, the submitted seven-shard C1 8.85 ms.

So by your stated criterion the Section 6 conclusion does not hold, and we have removed it. A cheap variant gets close to GCPA at lower latency than GCPA, and the query-adaptive variant beats it without seven indexes.

There is however one important argument, which is also the answer to your Q2. Stored-corpus statistics work because under random assignment, every encoder stores a sample of the same content. When encoder identity follows topic they fail. In a seven-source topic-structured index, corpus C1 drops to 0.528 on SciFact (GCPA 0.610) and to 0.218 on FiQA (GCPA 0.383, GPA 0.286). Two corrections do not have this weakness:

  • the per-source affine corrector, which uses only the Dolma anchors (SciFact 0.635, FiQA 0.411 under topic structure)
  • C1 with a small paired in-domain bank, i.e. 512 corpus documents encoded by every encoder (SciFact 0.663, FiQA 0.403). It beats GCPA in all 18 dataset-by-assignment cells across nine datasets, by +0.018 to +0.081.

For your Q2:

We ran the experiments. Six encoder pairs, seven fractions, three fits, queries from the new encoder. GCPA − GPA in nDCG@10, averaged over pairs (intervals exclude zero except where marked †). The time split assigns the newest documents (by Stack Exchange post ID) to the new encoder. The reversal is present from 10% to 75% on three datasets and from 10% to 25% on NFCorpus, and it does not disappear under topic or time structure.

DatasetSplit10%25%50%75%90%
SciFactrandom+0.150+0.148+0.088+0.019−0.012
SciFacttopic+0.082+0.100+0.082+0.037−0.005†
FiQArandom+0.155+0.108+0.048+0.006−0.014
FiQAtopic+0.129+0.122+0.086+0.010−0.016
NFCorpusrandom+0.043+0.025−0.003†−0.021−0.031
NFCorpustopic+0.040+0.026+0.003†−0.017−0.028
Androidrandom+0.146+0.106+0.057+0.013−0.008
Androidtopic+0.138+0.114+0.065+0.019−0.007
Androidtime+0.151+0.127+0.084+0.036+0.005†

On the novelty and missing related work

We agree, and we have rewritten the related work. We now cite Hot-Refresh, FastFill and Seo et al., who show why mixed galleries need separate evaluation. Our contribution is to distinguish two effects for frozen text encoders, GCPA improves score comparisons across encoders even when GPA ranks documents better within each encoder. We predicted and confirmed both effects on Android. This distinction also led to a practical correction, calibrating scores against the same reference texts preserves GPA’s within-source rankings and outperforms GCPA even under topic-based assignment.

Why GCPA's score compete better

  • GCPA roughly halves the distance between per-encoder centroids in the shared space (ratio 0.47–0.56 on the four datasets).
  • Varying the corrector's strength moves gap closure, mixed gain and homogeneous loss together.
  • A per-source affine map reproduces and exceeds the gain, while a shared bias gives none.
Reviewer_4wHPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether an embedding alignment method behaves differently when documents come from one encoder or a mixture of encoders. Across seven frozen encoders, four freshly encoded datasets, and a large historical Natural Questions index, nonlinear correction hurts homogeneous retrieval but improves balanced mixed-source retrieval relative to linear alignment. An order-preserving score-exchange experiment separates within-source ranking from competition between sources. Query-dependent score calibration recovers part of the improvement, with additional cost in the tested implementation.

Critical strengths
  1. The main finding has a clear practical consequence. Evaluating an aligner only on homogeneous indexes can select the worse method for a mixed index. The prospective Android result changes from a −0.0162 nDCG@10 difference in the homogeneous setting to +0.1027 in the mixed setting. The large historical index and matched-weight checks strengthen the evidence. This is a valuable empirical contribution even though the alignment methods are existing methods.

  2. The analysis tests a useful explanation. The score transplant preserves within-source order while changing score competition. It shows how better competition can outweigh worse ordering, and the prospective test supports the predicted directions. The paper also avoids treating this diagnostic as a deployable algorithm or a theoretical upper bound.

Critical weaknesses
  1. The deployment lesson is broader than the main experimental coverage. Most evidence uses balanced random source assignment and one nonlinear correction family. The appendix includes a two-source partial-migration experiment, but this does not establish behavior when encoder identity is correlated with topic, language, or document age. Such correlations could change which source should dominate a query. This limits how confidently the results can guide real migrations.

  2. The practical calibration comparison covers a limited set of alternatives. C1 is useful, but its cost includes seven separate searches in the chosen implementation. The results therefore establish a local quality–cost comparison, rather than a general advantage of nonlinear correction over calibration. The related study Closing the Modality Gap for Mixed Modality Search also finds that source-related geometry can harm mixed retrieval and uses a lightweight correction. Its setting differs, but it is relevant to the scope of the insight and to alternative calibration controls.

Questions for authors
  1. Does the reversal persist for a two-encoder migration with source assignments correlated with document content or time? Results of this kind would substantially strengthen the deployment claim beyond the existing random-assignment and partial-migration controls.
  2. Can a simpler offline source-wise correction recover a similar mixed-index benefit while retaining one shared search? The answer would help determine how much of the measured quality–cost tradeoff comes from the chosen C1 implementation.
2 comments

Thank you for the careful reading. Both of your questions led to results that are now in the paper, and the concern in your first weakness turned out to be correct.

For your Q1

We ran six encoder pairs at seven migration fractions under random, topic-contiguous (encoder-neutral TF-IDF clusters) and time-ordered assignment. GCPA − GPA, nDCG@10, pair-averaged, new-encoder queries:

DatasetSplit10%25%50%75%
SciFacttopic+0.082+0.100+0.082+0.037
FiQAtopic+0.129+0.122+0.086+0.010
NFCorpustopic+0.040+0.026+0.003−0.017
Androidtopic+0.138+0.114+0.065+0.019
Androidtime+0.151+0.127+0.084+0.036

The reversal persists (the NFCorpus value at 50% is not distinguishable from zero). In a seven-source index where each encoder holds different topics it is +0.097 / +0.097 / +0.020 / +0.100 on SciFact / FiQA / NFCorpus / Android respectively. You wrote that such correlations could change which source should dominate a query. And yes they do, and it matters for calibration but not for the reversal. Per-source standardization against the stored corpus is the best system under random assignment and collapses under topic structure (FiQA: 0.382 random, 0.218 topic, below uncorrected GPA at 0.286), because a source's score distribution for a query then carries relevance signal that standardization removes. The revised draft reports this.

We also report the limits for two-model migrations. With a hub fitted on the pair alone the effect depends on the pair, for two same-family upgrades it is reduced in one and largely absent in the other and direct conversion of old documents into the new encoder's space usually beats both aligners.

On your Q2

Yes, and it is more informative than we expected. Most of the quality–cost tradeoff in the submitted Section 6 came from our C1 implementation and from using an out-of-domain bank. We have replaced that section.

  • One extra linear map per encoder, trained on the same consensus objective as GCPA from the Dolma anchors only: One index, GPA's serving cost. It exceeds GCPA in the mixed index on all nine datasets under both random and topic-structured assignment (+0.007 to +0.028 nDCG@10), and it does not lose on cross-encoder homogeneous retrieval (difference from GPA positive on seven datasets and indistinguishable from zero on two, where GCPA is negative on all nine).
  • Centering plus a fixed scale from the stored corpus: One index, 1.54 ms against GCPA's 1.81 ms on FiQA, within 0.01–0.04 of GCPA on three datasets and above it on NFCorpus under random assignment, weaker under topic structure.
  • Query-adaptive calibration without seven searches: Retrieving 200 candidates from one index and rescoring takes 2.37 ms on FiQA (the submitted seven-shard implementation: 8.85 ms) with the same quality as the sharded version. With a paired in-domain bank of 512 corpus documents it beats GCPA in all 18 dataset-by-assignment cells (+0.018 to +0.081) and is stable under topic structure.

Nonlinear correction family

We now compare six corrector families on the same objective: shared or per-source, each as a bias, an affine map or an MLP. Per-source affine is best, per-source MLP is slightly below it, shared affine recovers 0.37–0.76 of GCPA's gain and a shared bias recovers none. Nonlinearity is not the active ingredient.

Closing the Modality Gap for Mixed Modality Search

Thank you for this reference, we now discuss it. Its setting is closer to ours than we had recognised. It sweeps the mixed fraction on NFCorpus and SciFact with nDCG@10 and removes per-modality mean embeddings. We added controls of that kind. Mean removal with renormalization using stored-corpus means reaches 0.563 / 0.296 / 0.302 / 0.328 on SciFact / FiQA / NFCorpus / Android (GCPA: 0.582 / 0.360 / 0.281 / 0.353 respectively), and a variant with a separate query-side mean, the choice that paper argues for, reaches 0.566 / 0.327 / 0.310 on the three datasets with a validation split. Mean removal therefore transfers to aligned text encoders, but only with in-domain means, with Dolma means it gives almost nothing.

Coverage

Seven additional forums were tested prospectively with locked predictions and all 83 locked directions hold on nDCG@10.

Thank you for the additional experiments and for addressing most of my concerns in your response. I am keeping my rating at 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper updates

We thank the reviewers. All three accepted the central observation and asked, in different ways, whether it survives realistic conditions and whether the practical conclusions follow. We ran the experiments you requested and several that their results called for. Three things changed:

  1. Section 6 was wrong as written, and Reviewers 2Xzt and 4wHP were right to question it. Calibration does not need seven indexes, and with in-domain statistics it is not weaker than nonlinear correction. The revision replaces that section.
  2. The nonlinearity is not what helps. We trained six correctors on GCPA's own objective. A per-source affine map (one extra linear map per encoder, learned from the same Dolma anchors) gives a larger mixed-index gain than GCPA on all nine datasets under both random and topic-structured assignment (+0.007 to +0.028 nDCG@10 over GCPA), without GCPA's loss on cross-encoder homogeneous retrieval. A shared bias does nothing, a per-source bias recovers roughly half of GCPA's gain. So the cross-source problem is a per encoder offset and scale in the shared space, and the shared MLP was an indirect way of removing it.
  3. The mixed-index gain of the correction is positive in every setting we tested (36 of 36 cohort-by-width cells, two and seven encoders, random, topic and time splits). The homogeneous loss is not universal. With only the three largest encoders GCPA is better than GPA in both settings on FiQA and Android. What holds everywhere is that the correction helps mixed retrieval far more than homogeneous retrieval. The strict sign reversal holds in the paper's seven-encoder setting and on seven further forums tested prospectively.

We added seven CQADupStack forums that the project had never scored (webmasters, mathematica, physics, programmers, gis, english, stats), in three rounds. All 83 directions hold on nDCG@10 (79 of 83 on Recall@100, the four failures are one direction, discussed under limits). The directions include the reversal, the score/order transplant signs, and predictions about the new corrections, including one in which a correction we had just found was predicted to lose.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.