- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to better budget document selection when cross-encoder budget B is smaller than LLM context capacity. The author observe that previous reranking wastes calls on documents already safely inside the top-k, and propose BoundaryMORPH, which is a GP-based selector that spends each CE call on the most contested pair around the k boundary. Experiments show consistent gains over single-stage reranking. The paper is overall well motivated and good to follow.
Critical strengths
- The problem framing is novel. The B < k regime is common in real RAG systems but rarely addressed before.
- The results are strong and ablation settings are meaningful. The experiments are overall carefully designed and ablations show both paarts of the method matter.
Critical weaknesses
- An end 2 end result of whether delivering more relevant but edge documents actually improves the final answer would strengthen the motivation.
- Adding a simple ablation of just scores MIPS ranks around the k-th position would show how much the GP actually adds to the gain.
Questions for authors
See weaknesses.
2 comments
Response to rebuttal
Thanks for your response. It solved part of my concerns, and I will keep my score at weak acceptance
Response to Reviewer yvQu
We thank the reviewer for the thoughtful review, which has already helped us improve the paper. We are glad you found the problem framing novel, and agree that the B < k regime is common in real RAG systems but rarely addressed before. We also appreciate that you found the results strong, the experiments carefully designed, and the ablations meaningful. Below, we address end-to-end evaluation and the ablation you suggested.
1. End-to-end evaluation
This is an important point, and we agree that showing how these edge documents affect the final answer would strengthen the motivation. We believe evaluating open-ended answers to these queries is a research problem in its own right; even in long-form QA, it is not yet clear how to evaluate answers reliably [1, 2], and research-style answers must also be deep, factual, and comprehensive. Because of this, budgeted reranking works usually evaluate with IR ranking metrics [3, 4, 5, 6], and we follow these works. We have added end-to-end evaluation as an important limitation in the revised PDF (LL [860-865]).
2. How much does the GP add?
Thank you for this suggestion! This is a great baseline to have, since it isolates exactly what the GP contributes. Per your suggestion, we spend the CE budget on a window of B documents centered at rank k in the MIPS ranking: the bottom B/2 documents inside the MIPS top-k, and the top B/2 documents just outside it. Everything ranked above the window is automatically included in the context, and everything below it is automatically excluded, so only documents in the window can change between included and excluded; the best-scored half of the window fills the remaining slots.
nCG@100 (%) at B = 50; all three methods use the same number of CE calls.
The window already helps, which confirms that the boundary is where CE calls are most impactful. However, most of BoundaryMORPH's gain comes from the GP and active selection. The window must spend its entire budget on the same fixed ranks, and can only change the documents it scored. The GP instead lets each CE score update the estimates of unscored documents across the whole pool, so BoundaryMORPH can spend calls on documents far above or below rank k whenever their inclusion is in doubt. We commit to adding this ablation to Section 5.2.
We hope that the new ablation and our discussion of end-to-end evaluation address your concerns. If any questions remain that would help in reassessing your score, we would be more than happy to address them.
[1] Hurdles to Progress in Long-form Question Answering
[2] A Critical Evaluation of Evaluations for Long-form Question Answering
[3] Bayesian Active Learning with Gaussian Processes Guided by LLM Relevance Scoring for Dense Passage Retrieval
[4] Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval
[5] AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking
[6] Adaptive Re-Ranking with a Corpus Graph