acceptodds
Under review as a conference paper at ICLR 2027

BreadBowl-Embed: Encode Once, Retrieve and Rerank

Abstract

Modern retrieval systems typically separate candidate generation from relevance refinement: bi-encoders provide efficient retrieval, while cross-encoders recover richer query-document interactions at substantially higher inference cost. We ask whether these two stages can instead operate on the same representation. We introduce BreadBowl-Embed, a retrieval model built around a Slot Encoder, which represents each query and document as a compact set of routing-value slots. The routing vectors provide an indexable representation for first-stage retrieval, while the same routing vectors subsequently define attention over a candidate's stored value vectors to produce query-dependent readouts for reranking. Retrieval and reranking therefore share not only an encoder, but the representation that determines both which documents are retrieved and how their contents are compared. Documents are encoded once and reused across both stages, so reranking requires neither a separate cross-encoder nor an additional backbone pass over candidate text. Across 15 BEIR tasks, value-based reranking raises macro nDCG@10 from 48.47 to 51.54 (scores scaled by 100) over routing alone on the same retrieved candidates. These results suggest that indexed retrieval and query-dependent relevance refinement can be supported by a single precomputed representation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Information Retrieval, Text Embeddings, Dense Retrieval, Late Interaction, Multi-Vector Representations, Neural Ranking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yaWvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates whether retrieval and reranking can share a single precomputed representation without requiring an additional backbone pass over candidate documents. The authors propose BreadBowl-Embed, which independently encodes queries and documents into a fixed set of routing–value slots. The routing vectors support indexed candidate retrieval and also determine attention over cached document value vectors for query-dependent reranking. The model is trained using a contrastive retrieval objective and reranking distillation, with periodic retrieval replay to mitigate interference between the two objectives. Experiments on 15 BEIR tasks show that adding value-based reranking improves macro nDCG@10 from 48.47 to 51.54 over routing-only scoring on the same retrieved candidates.

Critical strengths
  1. The paper addresses a practical problem in retrieval systems by exploring how to reuse precomputed document representations for both candidate retrieval and reranking. Avoiding an additional backbone pass over candidate documents is an appealing direction for reducing inference costs.

  2. The proposed routing–value architecture is an interesting design. Using the same routing vectors for both indexed retrieval and query-dependent attention provides a unified way to retrieve candidates and refine their relevance without re-encoding the document text.

  3. The experiments cover 15 BEIR tasks and demonstrate improvements from value-based reranking on 10 of the 15 tasks. The retrieval replay ablation also supports the usefulness of periodically incorporating retrieval supervision during reranking adaptation.

Critical weaknesses
  1. The novelty over existing late-interaction methods remains unclear. ColBERT already supports retrieval and reranking using cached multi-vector representations, while MVA explores attention-based scoring over stored value vectors. Direct comparisons under matched settings would help establish the specific advantages of the proposed unified architecture.

  2. The efficiency claims are not sufficiently supported by system-level evaluations. Each document requires storing 16 routing vectors and 16 value vectors, introducing additional memory and indexing costs, while the evaluated implementation also relies on exact flat search. End-to-end latency, throughput, and memory comparisons with existing retrieval and reranking methods would help determine whether the proposed approach offers practical efficiency advantages at scale.

  3. The benefits of value-based reranking are inconsistent across datasets, with performance declining on five of the 15 BEIR tasks and a particularly large drop on ArguAna. Moreover, the QK-versus-QKV comparison uses the same trained model rather than separately trained routing-only and combined architectures. Further analysis would help clarify when the value pathway is beneficial and whether its gains justify the additional representation costs.

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer yaWv (1/2)

Thank you for the careful reading. Your third point led to the most important experiment of this rebuttal, so we answer it first. Numbers are nDCG@10 × 100.

W3, first part: QK and QKV come from one trained model.

We ran exactly the control this asks for. It cuts both ways: a routing-only model recovers most of the QK-to-QKV gap, and it is never significantly ahead of QKV. From the same starting checkpoint, with the same data, order, teacher labels, seed and schedule, we continued training with every ranking loss applied to the routing score alone, so the value vectors receive no ranking gradient. "Combined model" below is the model trained with both scores, as in the paper. Intervals are 95%, from a paired bootstrap.

Development panel (5,000 queries, nine sources, fixed candidate lists, source-macro nDCG@10):

UpdatesRouting-only model, QKCombined model, QKCombined model, QKVRouting-only QK − combined QKRouting-only QK − combined QKV
6464.4162.8365.70+1.58 [+1.30, +1.87]−1.29 [−1.69, −0.88]
12865.9763.8566.46+2.11 [+1.75, +2.47]−0.49 [−0.86, −0.12]
25666.9964.8767.21+2.11 [+1.72, +2.50]−0.22 [−0.56, +0.13]
38467.6665.5867.71+2.08 [+1.71, +2.46]−0.05 [−0.39, +0.28]
51268.1466.2968.24+1.85 [+1.47, +2.21]−0.11 [−0.44, +0.22]

Held-out queries, both models at 512 updates (full MS MARCO corpus, 8.8M passages):

QueriesRouting-only model, QKCombined model, QKCombined model, QKVRouting-only QK − combined QKRouting-only QK − combined QKV
MS MARCO dev (6,980)40.3437.9740.69+2.36 [+2.00, +2.74]−0.35 [−0.70, −0.00]
TREC DL19 (43)72.1271.2373.65+0.89 [−1.05, +2.84]−1.53 [−3.51, +0.52]
TREC DL20 (54)71.2066.5771.10+4.63 [+2.41, +7.13]+0.11 [−1.62, +1.96]

What this shows:

  • The QK row is not a routing-only baseline. At 512 updates the routing-only model's routing score is 1.85 above the combined model's on the panel and 2.36 above it on MS MARCO dev.
  • The routing-only model catches up with QKV and does not pass it. On the panel QKV leads at 64 and 128 updates, and the two are level from 256 on. The comparison at 256 updates was fixed before the run. After seeing it we extended the control to 512 updates, to check whether the combined model pulls ahead with more training. It does not: the two stay level.
  • Held-out queries give the same picture. QKV keeps 0.35 points on MS MARCO dev, with an interval that only just excludes zero. No difference is shown on DL19 or DL20. On MS MARCO dev the routing-only model recovers 2.36 of the 2.72 points between QK and QKV.
  • QKV is never significantly behind. In the eight comparisons above (five panel checkpoints, three held-out sets) QKV is significantly ahead in three and level in the other five.

So most of the QK-to-QKV gap is not a gain over a routing-only model: the combined model's routing score is weaker, and its value score makes up the difference. Beyond the 0.35 points on MS MARCO dev, a gain of the value vectors over a model trained for routing alone is not shown. The revised paper will say this: the QK row will be labelled as an ablation of the scoring rule inside one model, and the control will be reported next to it.

Limits: one seed; both models continue from a checkpoint that was trained jointly; 512 of the 1,024 updates of the reported checkpoint; not run on BEIR.

W3, second part: tasks where the value score hurts. With the new checkpoint the value score lowers nDCG@10 on three of the 15 BEIR tasks instead of five: ArguAna (−2.76 [−3.54, −1.98], from −5.20), Quora (−0.63 [−0.82, −0.44], from −2.20) and CQADupStack (−0.25, from −0.08). We rescored our own candidates on ArguAna and Quora with both cross-encoders, and the two tasks turn out to be different cases.

All test queriesQKQKV0.6B cross-encoder4B cross-encoder
Quora (10,000)87.8987.2685.9487.24
ArguAna (1,406)53.3150.5568.1871.74
  • Quora. The routing score beats both cross-encoders: they are 1.94 [1.57, 2.33] (0.6B) and 0.65 [0.30, 1.00] (4B) below it. QKV is 1.32 [0.93, 1.70] above the 0.6B cross-encoder and equal to the 4B one (difference 0.02 [−0.33, +0.37]). For duplicate-question matching the teacher has nothing to add, and the value score inherits that.
  • ArguAna. The opposite. The cross-encoders are 14.87 and 18.43 points above the routing score on identical candidates, so the teacher does rank counter-arguments well and the distilled value score did not learn it.

The revision will state this as a limitation: the value score approaches cross-encoder quality on the query-to-passage tasks it was distilled on, and not on argument retrieval, where the queries are whole documents.

AuthorsOpenReview ↗

Response to Reviewer yaWv (2/2)

W1: novelty over ColBERT and MVA. We agree with both precedents, and the revision will position the model against them precisely. The routing score is a fixed-budget late-interaction scorer of the ConstBERT kind, and we do not claim it as new. MVA is the closest precedent for the value score; it reranks candidates supplied by a separate retriever. The contribution is the coupling: the vectors indexed for retrieval are the keys of the attention over the stored values, so one encoding, one index and one set of query vectors serve both stages.

The matched comparison you ask for is the control above: the routing-only model is a fixed-budget late-interaction model trained on identical data. Given its result, we no longer present QKV − QK as the advantage of this design over late interaction. The claim we make is the measured one: a single encoder and index whose own reranking is level with a cross-encoder of similar size on the same candidates, at about a fiftieth of its rerank time (comment to all reviewers, sections 1 and 4). We have no same-protocol ColBERT run. Published ColBERTv2 and ConstBERT numbers are in our reply to Reviewer ok2o.

W2: efficiency, storage and exact search. All three are measured in the comment to all reviewers (sections 3 and 4). Approximate search (IVF-PQ, one GPU, 5.7 GB index) replaces exact search at a cost of at most 0.54 nDCG@10 and brings a query to about 43 ms. We did not measure throughput under concurrent queries.

On storage you are right that 32 vectors per passage is the method's price, and it compresses well: 32,768 bytes in float32, 8,256 as int8 with no measurable loss, and 4,160 as int4 with at most 0.12 loss, about the size of one 1,024-dimensional float32 vector. Whether that is justified depends on the alternative:

  • Against a retriever plus cross-encoder, it buys about 50 times less rerank time at the quality of a cross-encoder of similar size. A 4B cross-encoder is more accurate on MS MARCO dev and takes about 230 times our rerank time.
  • Against a single-vector retriever alone, it is more storage, for ranking that is significantly better than Qwen3-Embedding-0.6B's and level with Jina v5 text-small's on MS MARCO (reply to Reviewer ok2o).
  • Against a routing-only model, which needs only the 16 routing vectors, the control above applies: at equal training it is within 0.4 points of the combined model on MS MARCO dev and not distinguishable from it on DL19 and DL20. The revised paper will put this next to the storage numbers.

Thank you for the additional experiments. The efficiency results address my concerns about exact search and representation storage. However, the routing-only control shows that it recovers most of the QK-to-QKV gap, leaving the additional benefit of the value pathway insufficiently demonstrated. The lack of a corresponding BEIR comparison further limits the evidence. While I appreciate the authors’ efforts, I do not find the additional results sufficient to change my assessment and therefore maintain my original score.

AuthorsOpenReview ↗

Two new experiments: the value pathway is ahead on BEIR and on TREC DL19

Thank you for the follow-up. You named two gaps: the benefit of the value pathway over the control was not demonstrated, and the control had no BEIR comparison. We ran the two experiments that answer them. Both favor the value pathway.

  1. The control at full length: 1,024 updates, the same as the reported model (512 in our first reply).
  2. The control on all 14 BEIR tasks.
nDCG@10 × 100Routing-only controlFull model (QKV)Difference, 95% interval
BEIR, mean of 14 tasks55.4156.08+0.68 [+0.45, +0.91]
TREC DL1972.9874.63+1.66 [+0.04, +3.37]
TREC DL2072.1373.14+1.01 [−0.50, +2.58]
MS MARCO dev41.6341.71+0.07 [−0.24, +0.38]
  • Task by task on BEIR, the full model is significantly ahead on five: Touché-2020 +4.22, ArguAna +1.56, HotpotQA +0.72, Climate-FEVER +0.66, NQ +0.57. The control is significantly ahead on one: Quora, by 0.34.
  • Twice the training did not put the control ahead. After 1,024 updates it leads on none of the four sets.
  • The comparison is matched. Same starting checkpoint, data, data order, teacher labels, seed and number of updates. Only the objective differs: the control's ranking losses train the routing score alone.

You were right that a routing-only model, not QK, is the yardstick. The revision uses it and labels QK an ablation. Against that yardstick the value pathway is not redundant: for a second set of 16 vectors per passage and 2.9 ms per query, it buys a significant lead on DL19 and on the BEIR mean.

In the revision: Section 4.4, and every BEIR task with the training trend (Figure 6) in Appendix C.4. Scope: one training seed; the intervals are over queries.

Reviewer_jKjuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BreadBowl-Embed, a retrieval model built around a shared Slot Encoder that independently maps each query and document to a fixed number of routing–value slot pairs. The routing vectors support ColBERT-style indexed retrieval via a log-mean-exp smoothed MaxSim score (sr), and the same routing vectors define attention over the candidate's stored value vectors to produce a query-dependent reranking score (sv), without re-encoding candidate documents. A two-stage training scheme combines weak contrastive training with cross-encoder distillation plus periodic retrieval replay. On 15 BEIR tasks, adding the value score on top of the routing score over identical retrieved candidates improves ranking performance.

Critical strengths

S1: The proposed method yields a single unified representation that supports both retrieval and reranking and avoids reprocessing document text with the backbone during reranking, which should substantially improve the retrieve-and-rerank pipeline's efficiency.

S2: The paper is well written and easy to follow; the polymorphic routing design and inference procedure are clearly described and illustrated.

Critical weaknesses

W1: Section 3.4 describes the training procedure too briefly, and much of the information essential to understanding the two-stage training (loss definitions, schedules, gradient paths, replay mechanism) is deferred to the appendix, making it hard for readers to grasp the purpose and design of the two training stages.

W2: The experimental evaluation is relatively weak: the main results only compare the full model (QKV) against its routing-only ablation (QK) on the same retrieved candidates, without comparing against existing retrieval/reranking baselines, so the effectiveness of the proposed method in realistic retrieval and ranking settings is not sufficiently validated; moreover, ranking evaluation is limited to BEIR, and results on standard benchmarks such as TREC DL19/DL20 are missing.

W3: The main claimed advantage of the method is efficiency—avoiding a backbone pass over candidate documents during reranking—yet the paper reports no evaluation of model efficiency, query latency, throughput, or storage cost to substantiate this claim.

Questions for authors

Q1: How does BreadBowl-Embed compare to representative existing methods (e.g., single-vector bi-encoders, ColBERT-style late interaction models, and cross-encoder rerankers such as the teacher) under the same evaluation protocol, both for retrieval recall and final ranking quality?

Q2: Could the authors provide concrete efficiency measurements (e.g., per-query latency, reranking throughput, index/storage size) comparing BreadBowl-Embed with bi-encoder + cross-encoder pipelines and late-interaction models, to quantify the claimed efficiency advantage?

3 comments
AuthorsOpenReview ↗

Response to Reviewer jKju

Thank you for the concrete list. We ran it. The baselines and the efficiency measurements are below, and the two items we could not run (a late-interaction system of our own, throughput under concurrent queries) are named as such. The shared tables are in our comment to all reviewers. Numbers are nDCG@10 × 100, in the order MS MARCO dev / DL19 / DL20.

W2 / Q1: baselines and TREC DL19/DL20.

  • DL19, DL20 and MS MARCO dev on the full corpus are in the comment to all reviewers (section 1): 41.7 / 74.6 / 73.1 for the new checkpoint, up from 34.3 / 69.0 / 59.6 at submission.
  • Cross-encoders, including both teachers, on the same candidates. The model's reranking is level with Qwen3-Reranker-0.6B, the submitted model's teacher, on dev and DL19 and significantly above it on DL20 (−0.7, +1.8, +3.7). Only Qwen3-Reranker-4B, the new checkpoint's teacher and five times the size of our encoder, is ahead (−2.9, −0.7, −2.2; significant on dev only).
  • Single-vector bi-encoders, same protocol (tables in our reply to Reviewer ok2o, Q2). Qwen3-Embedding-0.6B alone scores 38.0 / 66.8 / 67.6, significantly below our 41.7 / 74.6 / 73.1 on all three sets. Jina v5 text-small alone scores 42.1 / 70.6 / 72.0, with no significant difference from ours. Both have higher recall@100 at our default search width: 87.9 / 62.8 / 67.5 and 90.1 / 65.1 / 70.8 against 84.5 / 57.0 / 63.7 (ours is 88.6 / 66.6 / 73.2 with a four times wider search).
  • Bi-encoder plus cross-encoder. Adding a 0.6B cross-encoder to either retriever does not get past our single model: Jina then scores 42.0 / 72.6 / 66.8 and Qwen3-Embedding 41.9 / 72.0 / 67.6, level with ours on dev and DL19 and significantly below on DL20. It takes the 4B cross-encoder to move ahead (44.5 / 73.9 / 74.4 and 44.4 / 74.5 / 74.6): above ours on dev, level on the DL sets.
  • Late interaction. We could not run a ColBERT system in time. Published ColBERTv2, PLAID and ConstBERT numbers, stated as such, are in our reply to Reviewer ok2o. On the 13 BEIR datasets they report, ours is ahead (56.9 against 49.9 and 46.8, with the caveat that our training data is broader). On MS MARCO dev ColBERTv2 and ConstBERT are ahead (MRR@10 39.7 and 39.0 against our 35.3).
  • The routing-only ablation. You note that the paper compares QKV only with its own ablation. We trained a separate routing-only model as a control. At equal training it is 0.35 below the combined model's QKV on MS MARCO dev and not distinguishable from it on DL19 and DL20 (comment to all reviewers, section 2). So the QK row is an ablation of the scoring rule, not a baseline, and the revised paper will say so.

W3 / Q2: efficiency. You name the method's main advantage as avoiding a backbone pass over candidate documents during reranking. Measured, that advantage is a factor of about 50 against a cross-encoder of similar size and about 230 against a 4B one. Per-query latency by stage, index size and storage are in the comment to all reviewers (sections 3 and 4). In short:

  • Reranking against cross-encoders. About 3 ms per query against 150 ms (0.6B) and 660 ms (4B) on the same candidates and GPU. The cross-encoders were timed in batches (479 and 109 pairs per second), which favours them.
  • Latency. About 43 ms per query end to end on one GPU with a 5.7 GB approximate index, of which 34 ms is query encoding. The whole query takes less than a third of the time the 0.6B cross-encoder needs for reranking alone.
  • Storage. 8,256 bytes per passage at int8 with no measurable loss, 4,160 at int4 with at most 0.12 loss.
  • Against late interaction. Published index sizes and latencies for ColBERTv2, PLAID and ConstBERT are in our reply to Reviewer ok2o. Their indexes are smaller than ours.

We did not measure throughput under concurrent queries for any system.

W1: Section 3.4. The revision will move the loss definitions, the schedule, the gradient paths and the replay mechanism from the appendix into Section 3.4, and state the purpose of each stage: the first trains routing for retrieval; the second distils a cross-encoder into the value score while replay keeps retrieval from drifting. It will also describe the changed recipe of the new checkpoint and the two matched comparisons that motivated it: tuning the full backbone instead of keeping it frozen (+0.98 [0.68, 1.28] on the development panel at 64 updates), and letting the distillation losses train the routing vectors (+0.41 [0.18, 0.64] at 128 updates).

Thanks for the responses. While I think the efficiency advantage is supported by the additional experimental results, the improvement in effectiveness is mixed and marginal. The comparison with two rerankers still seems insufficient. Therefore, I'd like to keep my original assessment.

AuthorsOpenReview ↗

Against systems of its size: six significant wins, nine level, no loss

Thank you for the follow-up, and for confirming the efficiency result. On effectiveness, the same-setting record is not mixed.

Fifteen comparisons in one setting: five released systems of similar size on MS MARCO dev, TREC DL19 and TREC DL20 (nDCG@10 × 100, 95% paired bootstrap intervals, Section 4.2). Six significant wins, nine level, no loss.

  • The wins are not marginal. On DL20: +3.70 [+0.38, +6.98] over the 0.6B cross-encoder on identical candidates, +5.50 [+1.85, +9.19] and +6.36 [+2.58, +10.22] over the two retriever-plus-cross-encoder pipelines. Over Qwen3-Embedding-0.6B alone: +3.75, +7.82 and +5.51 on the three sets.
  • Level is the claim. One encoder and one index match a two-model pipeline, and rerank in 2.9 ms where the pipeline takes 200 ms.
  • It takes a cross-encoder five times its size to get ahead: by 2.9 points on MS MARCO dev, at 230 times the rerank time, and level on DL19 and DL20.
  • The student passes its first teacher. The 0.6B cross-encoder was its teacher in stage 2; on DL20 the student is ahead of it.

New since our first reply: a matched late-interaction model. We trained the routing-only control for the full 1,024 updates (same starting checkpoint, data and teacher labels) and ran it on all 14 BEIR tasks. The full model is significantly ahead on the BEIR mean (+0.68 [+0.45, +0.91]) and on TREC DL19 (+1.66 [+0.04, +3.37]), and level on MS MARCO dev and DL20 (Section 4.4).

On "two rerankers". They are the two that test the claim: one of our model's size and one five times larger, and both taught this model. Around them stand two released retrievers, four two-model pipelines and now the matched late-interaction model. On the two BEIR tasks where we also ran them, our model is ahead of the 0.6B on Quora (+1.32 [+0.93, +1.70]) and behind both on ArguAna, where queries are whole arguments.

Reviewer_ok2oOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose BreadBowl-Embed, which provides a shared representation for first-stage text retrieval and reranking. Encoding the query and document independently, BreadBowl-Embed produces16 routing-value slots per input. Similarities with the routing slots are used for candidate discovery, while the values are used for reranking. Across 15 BEIR datasets, the authors show that routing and value scoring (i.e., QKV) improves nDCG@10 from 48.47 to 51.54 when compared to routing only scoring (i.e., QK).

Critical strengths

S1. The authors tackle interesting problem of using the same embeddings for candidate retrieval and reranking. Using the same embedding, we have chance to save memory (e.g., index storage size).

S2. The paper is easy to follow. Figures are easy to understand and the paper is well written.

Critical weaknesses

W1. Experimental section is not comprehensive.

Although the paper discusses related late-interaction retrieval methods, such as ColBERT and ConstBERT, it does not experimentally compare BreadBowl-Embed against them. It is unclear how the proposing method is superior to existing method in terms of retrieval accuracy, index cost, or retrieval latency.

W2. The superiority over other retrieval methods is unclear.

The proposed method combines candidate retrieval and reranking to provide richer query–document interactions than conventional single-vector retrievers. However, Table 9 shows that BreadBowl-Embed achieves lower nDCG@10 (63.42 vs. 66.14) and recall@100 (87.19 vs. 90.73) than Jina v5 text-small on the prepared development datasets. The authors should clarify the intended advantage of their method.

Questions for authors

Q1. How does the BreadBowl-Embed perform when compared to existing two stage retrieval methods in terms of retrieval accuracy, index storage size, and search latency?

Q2. What is the advantage of the proposed method when compared to existing single-vector retrievers?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ok2o (1/2)

Thank you for the review. Your two questions are the right ones. The submission did not answer them; the new experiments do, and the answers favour the method. The shared tables are in our comment to all reviewers. Numbers are nDCG@10 × 100 unless stated.

Q1 / W1: accuracy, index size and latency against two-stage retrieval.

Retriever plus cross-encoder, same candidates. On MS MARCO dev / DL19 / DL20 the model's own reranking scores 41.7 / 74.6 / 73.1. A 0.6B cross-encoder on the same candidates scores 42.4 / 72.8 / 69.4: level with ours on dev and DL19, and significantly below ours on DL20. Only a 4B cross-encoder is ahead (44.6 / 75.3 / 75.3, significantly on dev only). Our rerank step takes about 3 ms per query. The cross-encoders take 150 ms and 660 ms on the same GPU, about 50 and 230 times longer, because they run a backbone pass over every candidate text and we run none. The price is storage: 8,256 bytes per passage at int8 (no measurable loss) or 4,160 at int4 (37 GB for the 8.8M passages), plus a 5.7 GB search index. A cross-encoder pipeline stores less and computes far more. The revised paper will state that trade-off with these numbers. Pipelines built on two single-vector retrievers are in the table under Q2.

Late interaction. We could not run a ColBERT system ourselves in time. What follows are published numbers for ColBERTv2 (Santhanam et al., NAACL 2022), PLAID (CIKM 2022) and ConstBERT (MacAvaney et al., ECIR 2025), not a same-protocol comparison. In short: against published numbers we are behind on MS MARCO dev, level with ConstBERT on TREC DL and ahead on BEIR, with a larger index.

  • MS MARCO dev. ColBERTv2 reports MRR@10 39.7, and ConstBERT with 32 vectors 39.0. Ours is 35.3 (32.8 with the routing score alone), so both are ahead in-domain.
  • TREC DL19 / DL20. ConstBERT with 32 vectors reports nDCG@10 73.1 / 73.3. Ours is 74.6 / 73.1.
  • BEIR. On the 13 datasets both papers report, nDCG@10 averages 49.9 for ColBERTv2 and 46.8 for ConstBERT (our averages of their tables). Ours is 56.9 (53.8 with the routing score alone). This is not like for like: ColBERTv2 was trained on MS MARCO only, and ConstBERT follows its training, while our training mixture also contains FEVER, HotpotQA and NQ data; ColBERTv2's Touché numbers use a newer version of that dataset; and the backbones differ (BERT-base against a 0.8B model). On the eight datasets left when those four and Climate-FEVER (which uses FEVER's corpus) are removed (ArguAna, DBPedia, FiQA, NFCorpus, Quora, SCIDOCS, SciFact, TREC-COVID) ColBERTv2 averages 50.5 and ours 55.3 (54.2 with the routing score alone), with QKV higher on each. Our mixture does contain scientific and StackExchange text from other sources.
  • Cost. The ColBERTv2 MS MARCO index is 16 to 25 GiB after residual compression (22 GiB under PLAID), and ConstBERT's is 11 GB with 32 vectors. Both are smaller than our int4 store (37 GB). PLAID reports 12 to 38 ms per query on an older GPU, without query encoding. Our search and rerank steps take about 9 ms after a 34 ms query encoding.

The revision will give these as published results, with the same caveats.

Q2 / W2: advantage over single-vector retrievers, given Table 9.

You were right about the submitted model. Table 9 now reads differently: the new checkpoint is above Jina v5 text-small on nDCG@10 there (69.69 against 66.14, in-distribution), though still below it on recall@100. The larger advantage is structural. A single-vector retriever needs a second model to rerank. Ours reranks its own candidates at the level of a cross-encoder of similar size, in about 3 ms and without reading the candidate texts again. On held-out MS MARCO queries this one model is significantly better than Qwen3-Embedding-0.6B alone, level with Jina v5 text-small alone, and level with or better than both of them followed by a 0.6B cross-encoder. We do not claim a better first-stage retriever than Jina.

Table 9 again. We reran the development table end to end (each source corpus encoded and searched again, same settings). The submitted checkpoint reproduces its row. The two baseline rows are the paper's and were not rerun, so we have no intervals against them.

ModelnDCG@10Recall@100
Qwen3-Embedding-0.6B (Table 9)62.2587.42
Jina v5 text-small (Table 9)66.1490.73
Submitted checkpoint, QKV (rerun)63.4287.19
New checkpoint, QK67.4389.69
New checkpoint, QKV69.6989.69

The new checkpoint is above both baselines on nDCG@10 and still below Jina on recall@100. QKV is above Jina on 8 of the 9 sources (MS MARCO is the exception, −0.84). This panel shares its sources with our training data and the checkpoint was selected on it, so it is in-distribution evidence only.

AuthorsOpenReview ↗

Response to Reviewer ok2o (2/2)

Held-out comparison. We ran both retrievers on the full MS MARCO corpus under our protocol (8.8M passages, exact search, same queries and metric code), alone and with each cross-encoder rescoring their top 100.

PipelinedevDL19DL20Recall@100 (dev / DL19 / DL20)
Ours, QK39.371.668.784.5 / 57.0 / 63.7
Ours, QKV41.774.673.1same candidates
Jina v5 text-small42.170.672.090.1 / 65.1 / 70.8
Jina + Qwen3-Reranker-0.6B42.072.666.8same candidates
Jina + Qwen3-Reranker-4B44.573.974.4same candidates
Qwen3-Embedding-0.6B38.066.867.687.9 / 62.8 / 67.5
Qwen3-Embedding + Qwen3-Reranker-0.6B41.972.067.6same candidates
Qwen3-Embedding + Qwen3-Reranker-4B44.474.574.6same candidates

Differences below are QKV minus the pipeline, with 95% bootstrap intervals, in the order dev, DL19, DL20.

  • Jina alone: −0.4 [−1.0, +0.2], +4.1 [−0.2, +8.8], +1.2 [−1.9, +4.3]. Level on all three sets.
  • Jina + 0.6B cross-encoder: −0.3 [−0.9, +0.4], +2.0 [−0.8, +5.0], +6.4 [+2.6, +10.2]. Level on dev and DL19, QKV significantly higher on DL20.
  • Jina + 4B cross-encoder: −2.8 [−3.4, −2.1], +0.7 [−2.6, +4.1], −1.3 [−3.9, +1.3]. Above QKV on dev, level on the DL sets.
  • Qwen3-Embedding alone: +3.8 [+3.1, +4.4], +7.8 [+3.1, +12.8], +5.5 [+3.0, +8.0]. QKV significantly higher on all three sets.
  • Qwen3-Embedding + 0.6B cross-encoder: −0.2 [−0.9, +0.5], +2.7 [−0.2, +5.7], +5.5 [+1.8, +9.2]. Level on dev and DL19, QKV significantly higher on DL20.
  • Qwen3-Embedding + 4B cross-encoder: −2.7 [−3.3, −2.0], +0.2 [−2.9, +3.3], −1.4 [−4.1, +1.2]. Above QKV on dev, level on the DL sets.

Both retrievers' candidates have higher recall@100 than ours at our default search width. With a four times wider search ours is 88.6 / 66.6 / 73.2.

So one model does the work of two. It matches Jina v5 text-small alone, matches or beats both retrievers followed by a 0.6B cross-encoder, and beats Qwen3-Embedding-0.6B alone outright. Only a 4B cross-encoder moves either pipeline ahead of ours, significantly only on MS MARCO dev, and it costs about 230 times our rerank time. What we do not show is an accuracy advantage over Jina alone.

Informativeness. The revised introduction will lead with this measured result, not with the mechanism: one model and one index at the accuracy of a retriever plus a cross-encoder of similar size, for about a fiftieth of the rerank time, with storage as the price.

Thank you for the additional experiments and explanation. I appreciate the authors work. However, my concerns of the paper still remains.

Regarding W1.

As the authors mentioned in their response, the experimental setting is different between the compared models (e.g., ColBERTv2). Thus, it's difficult to understand the superiority of the proposed method to the others. I think the paper could be much improved if the the retrieval accuracy, index size, and the retrieval latency are compared in the same fair setting.

Regarding W2.

The authors have introduced a new experimental result with a new checkpoint. It's unclear how this new checkpoint is different from the initial checkpoint that was used, and why the retrieval accuracy could be improved so much. Also, I think the paper could be further improved if the authors can show that the proposed method can be applied to other backbone models (e.g., Jina v5 text-small) and show benefits in retrieval accuracy.

AuthorsOpenReview ↗

One fair setting, every number measured: in nDCG@10 on MS MARCO and TREC DL, no system of our size is significantly ahead

Thank you for the follow-up. You asked for accuracy, index size and latency in one fair setting, and for what the new checkpoint is. Here are both.

W1. One setting: the same 8.8M passages, queries, protocol and GPU (nDCG@10 × 100)

SystemMS MARCO devDL19DL20Rerank per query
Ours: one model, one index41.774.673.12.9 ms
Late interaction, 16 vectors, trained in our setting (new)41.673.0▽72.1none
Qwen3-Embedding-0.6B + Qwen3-Reranker-0.6B41.972.067.6▽200 ms
Jina v5 text-small + Qwen3-Reranker-0.6B42.072.666.8▽200 ms
Our candidates + Qwen3-Reranker-0.6B42.472.869.4▽150 ms
Our candidates + Qwen3-Reranker-4B44.6▲75.375.3660 ms

▽/▲: significantly below/above ours (95% paired bootstrap interval).

  • Accuracy. In this table, every significant difference against a system of our size is in our favor. It takes a cross-encoder five times our size to get ahead, on MS MARCO dev only, at 230 times our rerank time.
  • Latency. 43 ms for the whole query on one GPU: 34 ms to encode it, 5.7 ms to retrieve from 141M vectors, 2.9 ms to rerank. Timed with float16 routing vectors in memory and the value vectors read from disk.
  • Index size. A 5.7 GB search index and a 37 GB vector store at 4 bits, which costs at most 0.12 points of nDCG@10. A pipeline stores less and runs 72 to 100 more backbone passes per query.

The late-interaction row is the same-setting comparison. We did not run ColBERTv2 or ConstBERT. They differ from our model in backbone and training data, so no number of theirs isolates the method. We trained a late-interaction model inside our setting instead: 16 vectors per query and per passage, the same starting checkpoint, data, teacher labels and 1,024 updates. Only the objective differs. It stores half as much and has no rerank step. Our model is significantly ahead on TREC DL19 (+1.66 [+0.04, +3.37]) and on the mean of 14 BEIR tasks (+0.68 [+0.45, +0.91]), and level on MS MARCO dev and DL20 (Section 4.4).

W2. The new checkpoint is the submitted model after one more training stage (Section 3.4): same architecture, no new parameters. Stage 2 kept the ranking gradient away from the backbone and the routing score. Stage 3 trains the whole encoder for both scores: MS MARCO dev 34.3 → 41.7, DL19 69.0 → 74.6, DL20 59.6 → 73.1, BEIR 15-task mean 51.5 → 55.1. What drives it, in matched comparisons on the development panel:

  • tuning the backbone: +0.98 [+0.68, +1.28] after 64 updates (frozen: +0.03);
  • letting distillation train the routing vectors: +0.41 [+0.18, +0.64] after 128 updates;
  • more of the old training instead: +0.06 [−0.08, +0.20] after 128 updates.

Stage 3 also adds two loss terms and a 4B teacher, and leaves out 6,634 SQuAD v2 questions, nearly all annotated as unanswerable (1.3% of the pool). Appendix A.4 gives every comparison.

Other backbones. Not run. The layers we add read only the backbone's token states, so the design is not tied to this backbone.

Comments on the paper

3 comments
AuthorsOpenReview ↗

To all reviewers: one encoder now matches a retriever plus a cross-encoder of its size, at a fiftieth of the rerank time (1/2)

We thank the reviewers. The submission argued that one encoding can serve both retrieval and reranking, and the reviews asked for proof against real baselines. The new experiments give it: one 0.8B encoder and one index now rank as well as a retriever followed by a cross-encoder of similar size, with no cross-encoder at query time and no second pass over the candidate text. A revised PDF follows during this discussion period. Numbers are nDCG@10 × 100 unless stated. Differences are computed before rounding.

Headline results

  • It reranks like a cross-encoder of its size, without running one. On identical candidates from the full MS MARCO corpus, the model's own reranking is level with Qwen3-Reranker-0.6B on MS MARCO dev (41.7 against 42.4) and DL19 (74.6 against 72.8) and beats it on DL20 (73.1 against 69.4, significant). On these sets only Qwen3-Reranker-4B, five times the size of our encoder, is ahead, and significantly so only on MS MARCO dev.
  • About 50 times less rerank time. Our reranking takes about 3 ms per query: 0.6 ms of arithmetic on stored vectors, the rest reading them. The cross-encoders take 150 ms (0.6B) and 660 ms (4B) on the same candidates and GPU, about 50 and 230 times longer, because they run a backbone pass over every candidate and we run none. Our whole query, encoding and search included, takes 43 ms: less than a third of the 150 ms the smaller cross-encoder needs for reranking alone.
  • One model does the work of two. Jina v5 text-small and Qwen3-Embedding-0.6B, each followed by Qwen3-Reranker-0.6B, are matched on MS MARCO dev and DL19 and beaten on DL20 (+6.4 and +5.5, both significant). Qwen3-Embedding-0.6B alone is significantly behind our model on all three sets.
  • Large gains since submission, same architecture. MS MARCO dev from 34.3 to 41.7, DL19 from 69.0 to 74.6, DL20 from 59.6 to 73.1, and the 15-task BEIR mean from 51.5 to 55.1.
  • Small footprint. 4,160 bytes per passage at int4 with at most 0.12 loss, about an eighth of float32, and a 5.7 GB search index on one GPU.
  • Small budget. The new training took about 200 GPU-hours on rented RTX 5090s, with teacher labelling and every development run included.

What the experiments do not show. (i) A gain of the value vectors over a routing-only model beyond 0.35 points: the control Reviewer yaWv asked for is 0.35 below the combined model on MS MARCO dev at equal training and not distinguishable from it on DL19 and DL20, so we no longer present the QKV − QK gap as that gain. (ii) A better first-stage retriever than Jina v5 text-small, which alone is level with our nDCG@10 on the three sets and has higher recall@100. (iii) A lead over late interaction in-domain: ColBERTv2 and ConstBERT report a higher MRR@10 on MS MARCO dev.

What we claim. Encode once, retrieve and rerank is now a measured operating point: the ranking of a retriever plus a cross-encoder of similar size, from one model and one index, in 43 ms end to end. The model already beats a cross-encoder of its own size on DL20 and on Quora. We believe stored-vector reranking can overtake small cross-encoders more widely. That is an expectation, not a result: on ArguAna the cross-encoders remain far ahead.

What changed since submission. The architecture is unchanged. Training continued from the paper's line of checkpoints for 1,024 updates (256 queries each) with a revised recipe: the whole text backbone is tuned, the distillation losses also train the routing vectors, labels come from a larger teacher (Qwen3-Reranker-4B), and a top-order loss and an InfoNCE term are added. These updates took about 200 GPU-hours of rented RTX 5090 time, roughly 120 US dollars, counting the pilot, teacher labelling, a discarded second arm and all development scoring. "Submitted" below is the paper's checkpoint and "new" is the checkpoint after this training. The new checkpoint was frozen before any test set was run on it, and each test set was run once.

1. MS MARCO dev and TREC DL19/DL20 (full corpus, 8.8M passages). Every column ranks the routing candidates: at most 100 per query, on average 72 for the new checkpoint and 92 for the submitted one at the paper's search width. The cross-encoders rescore the new checkpoint's own candidates, so the last four columns share them.

QueriesSubmitted QKSubmitted QKVNew QKNew QKVReranker-0.6BReranker-4B
MS MARCO dev (6,980)31.934.339.341.742.444.6
TREC DL19 (43)64.969.071.674.672.875.3
TREC DL20 (54)56.259.668.773.169.475.3

Paired differences for the new checkpoint (95% bootstrap intervals over queries; dev, DL19, DL20):

  • QKV − 0.6B cross-encoder: −0.7 [−1.3, 0.0], +1.8 [−0.9, +4.7], +3.7 [+0.4, +7.0]
  • QKV − 4B cross-encoder: −2.9 [−3.6, −2.3], −0.7 [−3.6, +2.2], −2.2 [−4.8, +0.4]
  • QKV − QK: +2.4 [+2.0, +2.9], +3.1 [+0.6, +5.6], +4.4 [+2.1, +6.7]
AuthorsOpenReview ↗

To all reviewers: one encoder now matches a retriever plus a cross-encoder of its size, at a fiftieth of the rerank time (2/2)

2. Routing-only control (asked for by Reviewer yaWv). QK and QKV above come from one model trained with both scores. We trained a second model from the same starting checkpoint with the same data, labels, seed and schedule, with every ranking loss on the routing score alone. At 512 updates it scores 40.34 / 72.12 / 71.20 (dev / DL19 / DL20), against 37.97 / 71.23 / 66.57 for the combined model's QK and 40.69 / 73.65 / 71.10 for its QKV. Routing-only minus QKV: −0.35 [−0.70, −0.00], −1.53 [−3.51, +0.52], +0.11 [−1.62, +1.96]. Its routing score is stronger than the combined model's, whose value score makes up the difference, and it is never significantly ahead of QKV. Tables and limits are in our reply to Reviewer yaWv.

3. Storage. Both vector sets stored at lower precision. Full corpus, all 7,077 queries, QKV.

StorageBytes per passageMS MARCO devDL19DL20
float3232,76841.7174.6373.14
float1616,38441.7174.6373.14
int88,25641.7074.6473.14
int44,16041.6474.5173.12

Int8 changes no score by more than 0.01, and int4 loses at most 0.12. At int4 a passage costs about as much as one 1,024-dimensional float32 vector (4,096 bytes).

4. Latency with approximate search. One query at a time on RTX 5090 GPUs, on the first 1,000 dev queries plus all DL queries (so the dev column differs slightly from section 1). Times are medians. The IVF-PQ rows use nprobe 256.

SearchGPUsSearch indexSearch (ms)End to end (ms)MS MARCO devDL19DL20
Exact (flat)472.4 GB121.9172.441.8874.7073.15
IVF-PQ, 32 bytes per vector15.7 GB1.542.741.6374.1672.81
IVF-PQ, 64 bytes per vector110.3 GB1.842.841.8174.3073.08

At nprobe 16 and 64, nDCG@10 is 0.05 to 0.54 lower than in these rows and the search is about 1 ms faster; the revision will give every setting. Of the 43 ms, 34 ms is encoding the query. Rescoring the shortlist with the routing vectors takes about 3.7 ms. Reranking with the value vectors takes 2.9 ms, of which 0.6 ms is the scoring and the rest is reading the vectors. The routing vectors were held in memory as float16 (72 GB) and the value vectors were read from disk; the lower precisions of section 3 were not timed. With a cold cache the value read took about 11 ms (as in the exact row), which gives about 52 ms end to end.

On the same candidates (72 per query on average) the cross-encoders took 150 ms per query for the 0.6B model and 660 ms for the 4B model on one of these GPUs (measured in batches, which favours them). That is about 50 and 230 times our 2.9 ms, and still more than 12 and 50 times with a cold cache.

5. BEIR, 15 tasks, both checkpoints. Same pipeline and settings as the paper's BEIR table.

TaskSubmitted QKSubmitted QKVNew QKNew QKVQKV change
ArguAna57.752.553.350.5−1.9
CQADupStack41.941.845.345.0+3.2
Climate-FEVER22.129.032.242.2+13.2
DBPedia-Entity38.441.243.145.0+3.8
FEVER77.986.687.890.5+3.9
FiQA43.243.641.242.5−1.1
HotpotQA60.964.769.273.3+8.6
MS MARCO (dev)31.934.339.341.7+7.4
NFCorpus38.538.937.337.8−1.1
NQ43.355.253.861.8+6.7
Quora87.285.087.987.3+2.3
SCIDOCS21.821.522.122.3+0.9
SciFact73.872.773.774.6+1.9
TREC-COVID66.478.375.082.1+3.7
Touché-202022.027.822.630.2+2.4
Mean, 15 tasks48.551.552.355.1+3.6
Mean without MS MARCO (14)49.752.853.256.1+3.3

The new checkpoint is higher on 12 of the 15 tasks and lower on three (ArguAna, FiQA and NFCorpus, by 1.1 to 1.9). For context, the Jina v5 paper reports 15-task means of 55.5 for Qwen3-Embedding-0.6B and 56.7 for Jina v5 text-small (settings not matched): the new checkpoint is 0.4 and 1.5 below them, where the submitted one was 4.0 and 5.1 below. The value score now lowers nDCG@10 on three tasks instead of five: ArguAna (−2.76, from −5.20), Quora (−0.63, from −2.20) and CQADupStack (−0.25, from −0.08). Mean candidate recall@100 is 68.8 against 67.9.

Limits. Recall@100 at the paper's search width is 84.5 / 57.0 / 63.7 (dev / DL19 / DL20) for the new checkpoint against 82.0 / 59.0 / 65.2 for the submitted one, so it is lower on both DL sets. With a four times wider search, where nearly every query has 100 candidates, it is 88.6 / 66.6 / 73.2 against 83.6 / 64.6 / 69.4, and the new checkpoint's nDCG@10 changes by less than 0.01. We did not run a late-interaction system ourselves, and we did not measure CPU serving or concurrent-query throughput. All results are from one training seed.

AuthorsOpenReview ↗

Summary of the final revision: two more experiments, both in favor of the architecture

To the Area Chair and all reviewers. The open question after our first replies was what the value pathway adds over a model with routing vectors alone. We ran the two experiments that answer it: a matched routing-only control at full length (a fixed-budget late-interaction model from the same starting checkpoint, data, teacher labels and 1,024 updates), and that control on all 14 BEIR tasks.

Full model minus control (nDCG@10 × 100, 95% paired bootstrap intervals)

  • BEIR, mean of 14 tasks: +0.68 [+0.45, +0.91]. Significantly ahead on five tasks, significantly behind on one (Quora, by 0.34).
  • TREC DL19: +1.66 [+0.04, +3.37].
  • MS MARCO dev (+0.07 [−0.24, +0.38]) and TREC DL20 (+1.01 [−0.50, +2.58]) are level. The control is ahead on none of the four.

What stands from the first round

  • One encoder and one index rank MS MARCO passages as well as a retriever followed by a cross-encoder of similar size: level on MS MARCO dev and TREC DL19, ahead on DL20 by 3.7 to 6.4 points.
  • Fifteen comparisons against systems of its size: six significant wins, nine level, no loss.
  • Reranking takes 2.9 ms per query, against 150 ms for that cross-encoder on the same candidates and GPU.

Scope. One backbone and one training seed. ColBERTv2 and ConstBERT were not run; we give their published numbers and, in their place, the same-setting control above. A cross-encoder five times the encoder's size is ahead on MS MARCO dev.

In the revision: Section 4.2 (the fifteen comparisons), Section 4.4 and Appendix C.4 (the control).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.