acceptodds
Under review as a conference paper at ICLR 2027

When Rerankers Work Together: Better and Faster Reranking without Joint Training

Abstract

Standalone reranker quality does not determine its value in a multistage RAG pipeline. We systematically study heterogeneous rerankers composed without joint training, spanning lexical, single-vector, late-interaction, and joint query–context scoring. Two lightweight semantic coarse scorers, Cache-Pointwise and Cache-Shapley, learn from passage relevance labels and share cacheable representations and linear-time singleton inference. Under matched candidates and budgets, we compare cascades with stage-removal references to measure evidence retention, repair and breakage, reader accuracy, and online pruning cost. On NoLiMa 32K with Gemma, inserting these scorers before CrossEncoder or Provence improves answer accuracy by 4.02–9.43 percentage points while accelerating online pruning by 1.57–1.88 relative to the corresponding reranker alone. Cache-Pointwise composed with ColBERTv2 through reciprocal rank fusion improves accuracy by 4.60 points at 1.36 speedup over direct ColBERTv2. The stage-removal comparisons reveal an asymmetry: a cascade can outperform its final reranker alone while underperforming its coarse selector alone. Two readers, a second multi-hop dataset, and three-stage extensions show that the gains depend on the reader, reranker pairing, and stage configuration. These findings demonstrate a practical route to better and faster reranking without joint training and motivate studying stage order, candidate budgets, and rank fusion as determinants of composition quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, multi-stage reranking, RAG evaluation, context selection, evidence retention, quality-efficiency trade-offs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gFW3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies whether trained rerankers can improve accuracy and efficiency when combined in a cascade. It introduces two lightweight scorers over cached embeddings, trained using pointwise or Shapley-based objectives, to filter candidates before a more expensive reranker. Both scorers use independent sentence scores at deployment. Experiments examine evidence retention, answer quality, latency, and memory. The study provides useful evidence about when cascades help, although the justification for Shapley training and the benefit of adding further stages remain less convincing.

Critical strengths
  1. Cached embeddings reduce repeated computation, while separate training allows existing rerankers to be combined without retraining the entire pipeline.

  2. The paper includes negative results, confidence intervals, and seed sensitivity. The ColBERT experiments also demonstrate promising improvements in both accuracy and pruning speed.

Critical weaknesses
  1. The architecture combines frozen embeddings, feature fusion, DeepSets, and staged filtering, while Shapley training follows SCP. The contribution is primarily the empirical study rather than a substantially new ranking method.

  2. Shapley training ranks marginal contributions, but inference ranks singleton values. No consistency constraint or analysis establishes when these two rankings should agree.

  3. Each sentence receives an independent score. Its value cannot change when another selected sentence duplicates, contradicts, or complements it, limiting the practical role of the set-based formulation.

  4. Encoding sentences alone can obscure pronouns, references, and supporting relationships. The coarse scorer cannot use neighboring text to resolve these dependencies before filtering.

  5. All unselected MS MARCO passages become negatives. Some may contain useful supporting information, but the paper does not examine how this possible annotation noise affects learned selection.

  6. Later rerankers cannot recover useful sentences discarded by the first stage. The method provides no uncertainty-based fallback for candidates whose usefulness is difficult to judge independently.

  7. The greedy procedure follows relevance scores without optimizing value relative to sentence length or joint usefulness. A highly ranked long sentence can consume budget needed for several useful shorter sentences.

  8. Increasing deployment samples from 5 to 50 lowers retention from 70.23% to 49.77% (Table 31). This substantial decline needs a clearer explanation of what the learned coalition scores capture.

  9. Pointwise–ColBERT reaches 34.14% accuracy versus 31.49% for Pointwise alone. However, the paper does not report paired uncertainty for this comparison, leaving the additional stage’s benefit insufficiently established.

  10. Both learned heads underperform frozen cosine on MuSiQue. The stronger BGE control also improves NoLiMa retention without improving QA, showing that better learned relevance scores do not reliably improve downstream answers.

  11. Pointwise–ColBERT obtains identical QA accuracy with hard selection and RRF. The retention difference is also statistically inconclusive, so the added fusion mechanism lacks clear empirical justification.

Questions for authors

See the Weaknesses

Reviewer_cahbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper trains two neural networks for cacheable coarse grain reranking to put before a precise reranker in multi-stage ranking pipelines. The evaluations use different configurations of the reranking pipelines to filter some of the irrelevant context before passing it to an LLM for answer generation under a fixed token budget. Evidence retention, token cost, and answer accuracy are used as main evaluation metrics, with latency/speed ups as secondary for colbertv2 runs.

Critical strengths
  • Easy to follow problem formulation, and experimental setup
  • Good experimental hygiene: Table one is well motivated with reporting the full context and oracle evidence only numbers [it is clear that noisy context with full evidence hurts answer accuracy], reporting separate relative deltas, breakage vs repair analysis
Critical weaknesses
  • Limited scorer novelty: SCP is an already existed method that is cited, and the feature vector used for training is heavily inspired by InferSent without citing the paper. Including the training of the coarse scorers as a novel contribution is an overstatement. The paper's contribution should be its evaluation framework.
  • Poor experiment setup has caused misleading conclusions: "adding a precise cross-encoder hurts the coarse selector's performance" this is only true because the selected cross-encoder is trivially bad; "cross-encoder/ms-marco-MiniLM-L-6-v2" is a 22M param model that according to table 1 is much poorer than the same size encoder that is used for training the coarse selectors, in fact its answer accuracy is even lower than not applying any context selection. I strongly recommend repeating the experiment with more realistic small embedder/reranker pairs that are still under 1B and relevant in today's real world applications. e.g. qwen3 0.6 embedder and cross-encoder, or BGE M3 embedder and cross-encoder reranker.
  • As stated by the authors and given that the evaluation framework is the core contribution of the paper, the impact of different target budgets, context size should be studied. Studying larger LLMs like 27B, etc could be beneficial too, budget permitting.
  • Other context selection methods are named in the related work but no comparison is done in the experiments section.
Questions for authors

see my suggestions in the above section

Reviewer_NoL2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies cascades of independently trained rerankers that select sentences from a long context before an LLM reader. A cheap coarse stage g (BM25, frozen MiniLM cosine, or one of two learned scorers) passes 50% of the sentences to a precise stage h (an MS MARCO MiniLM cross-encoder, OpenProvence, or ColBERTv2). Stage h then fills a 30% token budget for the reader. The two learned scorers, Cache-Pointwise and Cache-Shapley, are the same small MLP over frozen MiniLM embeddings, trained on MS MARCO. They differ only in the per-candidate score used in the pairwise margin loss during training: the singleton score or a Monte-Carlo Shapley value. Both use the singleton score at inference.

The claimed contributions are (1) a systematic study of heterogeneous rerankers composed without joint training, (2) the two cacheable coarse scorers, and (3) an evaluation protocol that compares each cascade with each of its stages alone and reports evidence "repair" and "breakage" rates. The main validation is NoLiMa 32K with Gemma 3 12B. On it, adding a learned coarse stage raises the precise stage's QA accuracy by 4 to 9 points and makes the pruning step 1.6 to 1.9 times faster. Secondary results cover Llama 3.1 8B, 2WikiMultiHopQA, ColBERTv2 endpoints, and three-stage cascades.

Critical strengths

S1. A carefully controlled empirical analysis.

  • Lexical, single-vector, late-interaction, and cross-encoder scorers composed into two- and three-stage pipelines on matched inputs (same 870 contexts, reader, and budget), with paired bootstrap intervals, stage-removal references, cost, and openly reported negative results.

S2. Main change over SCP[1]: training signal decoupled from the inference score.

  • Cache-Shapley is trained with Monte-Carlo Shapley credits but scored with singleton values at inference, so inference stays O(n)O(n), unlike SCP [1]. Singleton readout of the same network is both cheaper and better than Monte-Carlo readout (76.4% vs. 49.8% retention, Table 31).

Reference numbers refer to the list at the end of Critical Weaknesses.

Critical weaknesses

W1. The main gain is not convincing.

  • On NoLiMa, the "precise" rerankers keep fewer needles than frozen MiniLM cosine (CrossEncoder 50.7%, Provence 50.3%, MiniLM 72.2%), and Cache-Pointwise alone beats its own CrossEncoder cascade (31.5% vs. 28.5% QA; both Table 1). All reader results also use only seed 42, the best of three training seeds (86.1 / 78.7 / 68.5% retention, Table 34; L421–422), so the title claim rests on weak precise stages and one favorable seed, which questions generalizability.

W2. Limited novelty and narrow scope.

  • Scorer: follows SCP [1] closely, including its embeddings, value network, Shapley credits, margin loss, and training scheme; SCP already proposes the cascade as future work. The fusion head is the InferSent matching head [2], and cheap sentence-level pruning with a sentence encoder already exists [3].
  • Composition and protocol: cascades of independently trained rankers [4], RAG pipeline studies that combine rerankers and compressors [5], and helped/hurt metrics like repair/breakage [6] are established, and the paper cites none of them.
  • Scope: (most concerning) one main benchmark (NoLiMa, an LLM needle test, Not really representative or suitable for RAG tasks, and even uses a subset -- 870 of its 7,540 tests), one budget per dataset, two small readers with Gemma-only headline results, and no joint-training baseline.

W3. Writing issues.

  • Presentation: many terms were newly introduced but poorly explained, or used before definition (Cache-, online pruning, admitted fraction, credit training, coalition, g-only/h-only). The main text never names the CrossEncoder checkpoint. Shapley variant has no motivation (why test with "Shapley" variant? no explanation), and no average gain over Pointwise across seeds (Table 34).

References

[1] Chen et al., Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruning, arXiv:2607.16209

[2] Conneau et al., Supervised Learning of Universal Sentence Representations from Natural Language Inference Data, arXiv:1705.02364

[3] Liskavets et al., Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference, arXiv:2409.01227

[4] Nogueira et al., Multi-Stage Document Ranking with BERT, arXiv:1910.14424

[5] Wang et al., Searching for Best Practices in Retrieval-Augmented Generation, arXiv:2407.01219

[6] Yan et al., Positive-Congruent Training: Towards Regression-Free Model Updates, arXiv:2011.09161

Questions for authors
  1. What does this paper add over SCP [1] beyond the feature-fusion head, singleton inference, and the cascade evaluation?
  2. What are the coarse-only results on 2WikiMultiHopQA?
  3. What are Llama's full-context and oracle-evidence QA on NoLiMa?
  4. These are the questions that came into my mind while I was reading the paper: (not asking to answer these, just to show that the paper is very hard to read, with many questions:)
  • what is cache-pointwise? (what does cache- mean? there's even no reference on introduction)
    • is it your-defined wording or is it something that actually already exists? -> oh so this is your-defined wording.
  • what do you mean by online pruning? what did you prune?
  • what is h-only and g-only in figure 1?
  • what is admitted fraction alpha?
  • what is credit training?
  • L.85 MiniLM: which MiniLM? where's the reference?
    • L.262 finally found MiniLM-L6-v2 (also without any reference). Please unify
  • what is/why Shapley? any references? to show what?
  • when i read all context i notice there are more benchmarks like 2wikimultihopqa. But only nolima32k was emphasized in the abstract.

Comments on the paper

1 comment

Dear reviewers, as the deadline is approaching, please check the rebuttal and consider participating in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.