acceptodds
Under review as a conference paper at ICLR 2027

BoundaryMORPH: Budgeted Cross-Encoder Document Selection for Diffuse Retrieval

Abstract

Open-ended queries in modern Retrieval-Augmented Generation (RAG) are increasingly "diffuse," requiring a large set of documents to be assembled into a finite LLM context window. To ensure retrieval quality, systems use fast dual-encoders and more expensive cross-encoders (CEs) to score candidates. However, the CE budget is strictly bounded by latency and is often smaller than the context window capacity . This mismatch makes standard reranking structurally flawed: it wastes compute verifying obvious top candidates while ignoring relevant documents further down the initial ranking. To address this, we introduce BoundaryMORPH, a novel algorithm that allocates CE budget specifically for the LLM's context capacity . Using a Gaussian Process, BoundaryMORPH treats the initial dual-encoder ranking as a structural prior and intelligently spends CE calls on resolving top- set membership at the boundary, rather than seeking a single most-relevant document. Information from each CE call propagates to unscored documents, maximizing the utility of the budget. We demonstrate that BoundaryMORPH achieves state-of-the-art set retrieval quality across multiple models and datasets with open-ended queries ( pp nCG@100 over the strongest baseline).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
information retrieval, RAG, active learning, reranking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yvQuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to better budget document selection when cross-encoder budget B is smaller than LLM context capacity. The author observe that previous reranking wastes calls on documents already safely inside the top-k, and propose BoundaryMORPH, which is a GP-based selector that spends each CE call on the most contested pair around the k boundary. Experiments show consistent gains over single-stage reranking. The paper is overall well motivated and good to follow.

Critical strengths
  1. The problem framing is novel. The B < k regime is common in real RAG systems but rarely addressed before.
  2. The results are strong and ablation settings are meaningful. The experiments are overall carefully designed and ablations show both paarts of the method matter.
Critical weaknesses
  1. An end 2 end result of whether delivering more relevant but edge documents actually improves the final answer would strengthen the motivation.
  2. Adding a simple ablation of just scores MIPS ranks around the k-th position would show how much the GP actually adds to the gain.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer yvQu

We thank the reviewer for the thoughtful review, which has already helped us improve the paper. We are glad you found the problem framing novel, and agree that the B < k regime is common in real RAG systems but rarely addressed before. We also appreciate that you found the results strong, the experiments carefully designed, and the ablations meaningful. Below, we address end-to-end evaluation and the ablation you suggested.

1. End-to-end evaluation

"An end 2 end result of whether delivering more relevant but edge documents actually improves the final answer would strengthen the motivation."

This is an important point, and we agree that showing how these edge documents affect the final answer would strengthen the motivation. We believe evaluating open-ended answers to these queries is a research problem in its own right; even in long-form QA, it is not yet clear how to evaluate answers reliably [1, 2], and research-style answers must also be deep, factual, and comprehensive. Because of this, budgeted reranking works usually evaluate with IR ranking metrics [3, 4, 5, 6], and we follow these works. We have added end-to-end evaluation as an important limitation in the revised PDF (LL [860-865]).

2. How much does the GP add?

"Adding a simple ablation of just scores MIPS ranks around the k-th position would show how much the GP actually adds to the gain."

Thank you for this suggestion! This is a great baseline to have, since it isolates exactly what the GP contributes. Per your suggestion, we spend the CE budget on a window of B documents centered at rank k in the MIPS ranking: the bottom B/2 documents inside the MIPS top-k, and the top B/2 documents just outside it. Everything ranked above the window is automatically included in the context, and everything below it is automatically excluded, so only documents in the window can change between included and excluded; the best-scored half of the window fills the remaining slots.

NeuCLIRRobust04TravelDest
Single-stage rerank (MIPS top-B)46.237.128.4
Window centered at k (your baseline)49.140.131.8
BoundaryMORPH59.251.543.1

nCG@100 (%) at B = 50; all three methods use the same number of CE calls.

The window already helps, which confirms that the boundary is where CE calls are most impactful. However, most of BoundaryMORPH's gain comes from the GP and active selection. The window must spend its entire budget on the same fixed ranks, and can only change the documents it scored. The GP instead lets each CE score update the estimates of unscored documents across the whole pool, so BoundaryMORPH can spend calls on documents far above or below rank k whenever their inclusion is in doubt. We commit to adding this ablation to Section 5.2.


We hope that the new ablation and our discussion of end-to-end evaluation address your concerns. If any questions remain that would help in reassessing your score, we would be more than happy to address them.


[1] Hurdles to Progress in Long-form Question Answering

[2] A Critical Evaluation of Evaluations for Long-form Question Answering

[3] Bayesian Active Learning with Gaussian Processes Guided by LLM Relevance Scoring for Dense Passage Retrieval

[4] Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval

[5] AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking

[6] Adaptive Re-Ranking with a Corpus Graph

Reviewer_yvQuOpenReview ↗

Response to rebuttal

Thanks for your response. It solved part of my concerns, and I will keep my score at weak acceptance

Reviewer_6CgGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BoundaryMORPH, a budget-aware document selection method for diffuse retrieval. The paper focuses on a practical setting where the number of cross-encoder (CE) calls is much smaller than the number of documents that can be included in the LLM context. The main idea is to allocate CE calls to documents whose inclusion in the final top-k set is uncertain, rather than repeatedly scoring highly ranked documents. Specifically, BoundaryMORPH uses a Gaussian Process (GP) with the initial dense retrieval ranking as its prior mean to estimate CE scores for unscored documents. It then applies a boundary-aware acquisition rule to select the next document for CE evaluation. Experiments on three retrieval datasets show consistent improvements over existing methods, including BAGEL and RGS, under the same CE budget.

Critical strengths
  • This paper is well-motivated. The paper identifies an important mismatch between the limited CE budget and the potentially large number of documents needed for RAG. Formulating the problem as budgeted top-k set selection, rather than conventional reranking, is a useful perspective.

  • The proposed method is effective & efficient. With only 0.25% CE calls, it can achieve 6080% of the performance of full reranking, which is encouraging. This method significantly outperforms baseline methods.

  • The paper includes useful ablation studies, evaluations with different embedding and CE models, and latency measurements.

Critical weaknesses
  • This paper is a little bit difficult to follow for people who don't have the background on GP. I suggest expand the method section and put the algorithm into the body.

  • The method requires sequential CE evaluations and GP updates. As pointed out in the limitation, CE inference can be massively paralleled in real-world production settings, which hinders BoundaryMORPH's efficiency advantage. It would be useful to further discuss how the method could be adapted to batch selection.

Questions for authors
  • The method assumes that documents with similar embeddings tend to have similar residuals between CE scores and MIPS priors. How sensitive is the method to this assumption? Are there query types where the GP estimates become less reliable?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 6CgG

We thank the reviewer for the thoughtful review, which has already helped us improve the paper. We are glad you found the paper well-motivated, that it identifies an important mismatch between the limited CE budget and the many documents RAG needs, and that framing the problem as budgeted top-k set selection is a useful perspective. We also appreciate that you found our ablations, evaluations with different embedding and CE models, and latency measurements useful. Below, we address the GP background, batch selection, and your question about the smoothness assumption.

1. GP background

"This paper is a little bit difficult to follow for people who don't have the background on GP. I suggest expand the method section and put the algorithm into the body."

This is a fair point. Without a GP background, it may be hard to see how our method fits into a standard GP, and an external reader's view on presenting this background is genuinely helpful to us. Based on your feedback, we've moved a simplified version of our pseudocode from the appendix into Section 3 (see revised pdf). We've also added a high-level intuition of GP-based Bayesian optimization to the start of Section 3.1 (LL [161-192]):

Bayesian optimization with a GP is an iterative loop. At each step, a GP is fit to the evaluations made so far, and it predicts the value of every candidate...An acquisition rule then uses these predictions to choose the next candidate to evaluate, and the loop repeats until the budget is spent.

2. Sequential execution and batch selection

"CE inference can be massively paralleled... It would be useful to further discuss how the method could be adapted to batch selection."

We agree, and we raise this in our Limitations as well. Parallel hardware can reduce latency, although every CE call still has a cost, and on-device or shared hardware cannot always run many calls at once. Per your suggestion, we adapted BoundaryMORPH to batch selection, combining our selection criterion with the standard hallucinated observations trick from the GP bandit literature [1]:

Method (B = 50, avg. of 3 datasets)Docs per roundRoundsnCG@100 (%)
Single-stage rerank50137.2
BoundaryMORPH25250.2
BoundaryMORPH10551.3
BoundaryMORPH (sequential)15051.3

Medium-size batches greatly reduce the number of rounds while matching the sequential result, and even the largest batches keep most of BoundaryMORPH's gain over single-stage reranking. We commit to adding batch selection to the appendix and to our Limitations discussion, and we see batch selection as a promising direction for future work.

3. Sensitivity to the smoothness assumption

"The method assumes that documents with similar embeddings tend to have similar residuals... How sensitive is the method to this assumption? Are there query types where the GP estimates become less reliable?"

This is a good question! After seeing your question, for each query, we measured how well this assumption holds using centered kernel–target alignment [2], a standard measure of how well a kernel's notion of similarity matches a target (here, the true residuals). The assumption holds on every query, well above a purely random baseline. This lets us ask your question directly: when the assumption is weaker, does BoundaryMORPH perform worse? We split the queries into quarters by this alignment measure:

Queries grouped by alignmentLowest quarterSecond quarterThird quarterHighest quarter
Alignment0.080.100.130.19
Alignment of purely random residuals0.000.000.000.00
Improvement over single-stage reranking (pp)+10.9+14.8+14.5+16.5

Higher alignment means the assumption holds better. Improvements are in nCG@100 at B = 50.

Yes, which supports the intuition: BoundaryMORPH's improvement over single-stage reranking is smallest where the assumption is weakest and largest where it is strongest, though it remains large throughout. We commit to adding this analysis to the appendix.

Finally: asking for which types of queries this assumption holds least/most is an interesting open question. We hypothesize that it holds least for narrow, event-specific queries (e.g., a single wildfire) and for attribute queries (e.g., "cities known for traditional crafts"), where documents close in embedding space may share a topic but still differ in relevance. We see this question as a direction for future work.


We hope that the revised presentation, the batch selection results, and the alignment analysis address your concerns. If any questions remain that would help in reassessing your score, we would be more than happy to address them.


[1] Parallelizing Exploration-Exploitation Tradeoffs in Gaussian Process Bandit Optimization

[2] Algorithms for Learning Kernels Based on Centered Alignment

AuthorsOpenReview ↗

Response to Reviewer 6CgG

We believe that we have addressed all the concerns as raised. As the reviewing period comes to an end, we would be happy to provide any additional clarifications if helpful. Please let us know if there are any remaining questions we can address to assist in your reassessment.

Reviewer_H7FHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The problem the paper is trying to solve is this: Using embedding models for RAG is cheap but not accurate. Rhis is why we usually use a more expensive reranker to obtain the final candidates that goes to the decoder. CE is the most accurate but also the most expensive. Assume we need to output k candidates but we have budget B for the cross encoder where B<k how can we use B to output k suggestions? BoundaryMORPH uses a Gaussian process with a retriever-percentile prior to propagate observed cross-encoder scores to unscored documents. At each step, it picks one document inside the current top k and one outside whose predicted scores are closest relative to their uncertainty, then scores the more uncertain document with the cross-encoder. They do experiments that show improvements over reranking and active retrieval baselines on three datasets.

Critical strengths

The problem of bringing the cost of reranking down is a relevant and interesting problem. The paper is easy to understand and follow. The experiments show consistent improvement over the baseline and ablations show both retriever-based prior and boundary-based selection rule contribute. retriever-based prior or the boundary-based selection rule.

Critical weaknesses

my main concerns with the paper are as follows:

  1. The setup itself does not make a lot of sense. the papers go into the decoder which is usually a larger more expensive LLM. The paper focus on the case where B<k which means that in this regime the prefill of the B docs to the CE is much lower than the prefill of the k documents into the LLM and this is before generating a single answer token. This meant that the cost saving is marginal at best.
  2. Why going directly to CE? there is a middle ground which is late interaction i.e. ColBert. There is no comparison for late interaction which might give the same improvement cheaper. To me this paper method and setup fits much better to active learning then RAG. I active learning setting this problem makes more sense but probably also a much more saturated research area.
  3. I dont see much novelty in the method either. BAGEL already uses a GP to propagate relevance observations across document embeddings, while Mean Prediction (MP) in Active Set Ordering already constructs a posterior-mean top-k set and selects evaluations through comparisons across its boundary. BoundaryMORPH adds a retriever-percentile prior and replaces MP’s regret-bound-based pair selection with a standardized gap. the experiments do not directly compare against MP using the same prior, kernel, initialization, and evaluation budget.
Questions for authors

How does BoundaryMORPH compare with the original MP acquisition rule when both use the same retriever-percentile prior, kernel, initialization, and budget?

What is the teoretical motivation behind the standardized-gap rule beyond intuition? Have you compared it with a covariance-aware pairwise criterion?

What evidence supports the practical importance of B<k when generator costs are included? Does the method improve final answer quality at a fixed end-to-end budget?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Note to Reviewers and AC about Revision

Thank you to all reviewers and the AC. We've uploaded a revised pdf of our paper, with all writing changes incorporated in a blue/green color for readability, and referenced in our responses to the reviews. The font color will of course be changed to normal in the camera ready version. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.