acceptodds
Under review as a conference paper at ICLR 2027

What Makes Retrieval Optimal for Context-Level RAG? A Joint Top-k Approach

Abstract

Large language models can answer user question effectively, but they still suffer from hallucinations and outdated knowledge. Retrieval-Augmented Generation (RAG) has emerged as a popular solution by retrieving accurate and up-to-date reference documents from external databases and incorporating them into the generation context. Most existing theoretical formulations of RAG focus on the document-level setting, where candidate answers are generated for each retrieved document and then ranked. In practice, however, the more commonly deployed paradigm is context-level RAG, which directly generates a final answer conditioned on the top- retrieved documents. This creates a notable gap between existing theory and practical systems. To bridge this gap, we analyze context-level RAG through a probabilistic marginalization framework and derive a joint top- retrieval objective whose ranking criterion depends on the question, answer, and comprehensive context. We show that this joint top- objective is optimal for downstream answer generation. However, directly retrieving according to this criterion is infeasible in practice, since the answer and comprehensive context are unknown at retrieval time. To address this challenge, we propose JBR-C (Joint-Based Retriever for Context-level RAG), a practical retriever that uses listwise learning-to-rank to approximate the latent joint ranking signal with respect to the best answer and context. Extensive experiments on four question-answering benchmarks show that JBR-C consistently outperforms representative retrievers.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Retrieval-Augmented Generation, Document Representation, Listwise Learning-to-Rank

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dLNTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a retrieval objective for context-level RAG, where a final answer is generated directly from the top-k retrieved documents. The authors argue that existing document-level RAG formulations do not adequately account for interactions among multiple documents when generating an answer.

To address this limitation, the paper proposes JBR-C, a method for training a dense retriever using joint ranking supervision. For each question in the training set, a candidate set consisting of gold documents, hard-negative documents, and randomly sampled negative documents is constructed. Joint ranking scores are then calculated by combining document relevance estimated by an embedding model with the probability of generating the correct answer estimated by an LLM. The documents are ranked according to these scores, and the resulting rankings are used to train a dense retriever through listwise learning-to-rank (LL2R). Experiments on four QA datasets generally demonstrate improvements over the evaluated baselines in both retrieval quality and downstream answer generation performance.

Critical strengths

S1. Generation-aware Ranking Supervision: The paper presents a systematic approach to constructing ranking supervision by combining document relevance and the probability of generating correct answers. The improvements in retrieval and generation performance across four QA datasets demonstrate the practical effectiveness of the proposed approach.

Critical weaknesses

W1. Limited Baseline Coverage: The evaluated baselines are relatively conventional, and comparisons with relevant retriever training and ranking methods are missing. Prior studies have explored answer-aware supervision for retriever training [1, 3], shared-context attribution for estimating document utility [2], and layer-wise contrastive ranking [4]. Comparisons with these methods would help establish the effectiveness and novelty of the proposed RI-based supervision.

  • [1] Atlas: Few-shot Learning with Retrieval Augmented Language Models (JMLR, 2023)

  • [2] Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Models (EMNLP, 2025)

  • [3] ReSCORE: Label-free Iterative Retriever Training for Multi-hop Question Answering with Relevance-Consistency Supervision (ACL, 2025)

  • [4] Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS, 2025)

W2. Limited Evidence of Document Complementarity: Although JBR-C constructs ranking supervision using a comprehensive context, the trained retriever relies on independent query–document similarities during inference. It remains unclear whether this approach effectively retrieves complementary documents. Moreover, the lack of multi-hop QA evaluations (e.g., HotpotQA, 2WikiMultiHopQA, and MuSiQue) makes it difficult to assess whether the proposed method addresses its central motivation.

Questions for authors

Q1. Effectiveness of Ranking Indicator (RI)-based Supervision: What evidence shows that RI-based supervision improves the retrieval of complementary documents rather than merely improving individual document rankings? A comparison with +LL2R on multi-hop QA benchmarks would help validate the effectiveness of RI-based supervision in retrieving complementary evidence.

Reviewer_rn4tOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a probabilistic formulation of context-level RAG and a document-ranking score combining context-conditioned retrieval probability with reference-answer likelihood. It argues that selecting the largest joint contributions minimizes a sufficient truncation threshold for preserving answer ordering. The practical method, JBR-C, constructs context-aware teacher rankings and trains a Contriever-based retriever with a listwise likelihood loss. Experiments evaluate retrieval and generation on TREC, WebQ, NQ, and TriviaQA with several generators.

Critical strengths
  1. Aligning retrieval with downstream generation is an important problem, and moving beyond single-positive supervision is a reasonable practical direction.
  2. The paper evaluates both retrieval and answer quality across four datasets and three generators. The ranking-indicator and listwise-loss ablations help separate components, while the appendices include context construction, preparation cost, cross-dataset transfer, and reranking experiments.
  3. JBR-C shows useful empirical improvements over DPR on NQ and TriviaQA in the main table, and it retains the standard dual-encoder inference structure.
Critical weaknesses
  1. The central theoretical claim does not follow for the deployed context-level generator. Equation 3 truncates a sum whose terms remain conditioned on the full comprehensive context T*. The actual system generates from P(a | q, Tk), where only the retrieved subset is available. The paper does not establish a relationship between this distribution and the truncated quantity. Maximizing retained mass while still conditioning on T* therefore does not establish optimality for generation from Tk, particularly when documents are complementary. The statement that the approximation is exact when Tk = T* also needs a support assumption ensuring that documents outside T* have zero contribution.
  2. Theorem 1 establishes a narrower property than general optimality for answer ordering. Its threshold compares the retained mass of one target answer against the full probability of a competing answer. This is a sufficient certificate, rather than the actual ordering condition between two truncated probabilities in Definition 2. The top-k ordering is also target-answer dependent. The theorem should be stated at this narrower scope, and its relevance to a fixed retrieval budget and the learned query-only retriever needs justification.
  3. The generation-aware baseline is not sufficiently reliable to support the claimed comparison. REPLUG reaches 0.000 EM on NQ and 0.002 on TriviaQA with Llama, and 0.008 on TriviaQA with Qwen. Appendix C states that its supervision is prepared only once, changing the original training procedure. These extreme failures require diagnosis or a validated reference configuration; they do not by themselves demonstrate that generation-aware retrieval is intrinsically unstable. Moreover, Table 4’s context-conditioned relevance-only indicator is already close to JBR-C and slightly better on TriviaQA, which makes the incremental value of the answer-likelihood factor less clear.
  4. The evaluation split protocol is ambiguous. Appendix C explicitly describes sampling test instances for hyperparameter tuning, but does not clearly identify a separate untouched final evaluation set or provide its sizes. Since some reported gains are modest and the smaller benchmarks have limited data, this distinction is necessary to interpret the results. The text should clarify the protocol rather than leave the reader to infer whether test data influenced model selection.
Questions for authors
  1. What assumptions connect the truncated expression in Equation 3 to P(a | q, Tk)? Can the claimed downstream optimality be established under those assumptions, or should it be restricted to retaining mass in the oracle-conditioned mixture?
  2. Can you state precisely whether Theorem 1 concerns a sufficient threshold for one target/competitor pair or the minimum k for Definition 2? How does the learned ranking inherit that property after replacing T* and the scoring distributions with heuristics?
  3. What causes the near-zero REPLUG results? Please provide a sanity-checked configuration and clarify whether the main conclusions hold against a properly functioning generation-aware baseline.
  4. Please give the exact training, validation/model-selection, and final evaluation splits and sizes. Were the instances described as test data for hyperparameter tuning also used in any reported final results?
1 comment

Dear reviewers,

Will you please check the authors' response and reply it?

Best, AC

Reviewer_mH4EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on RAG. The proposed method addresses the retrieval portion of RAG. It first combines an embedding model and an LLM to rank given candidate documents, then trains a small retriever using a common ranking loss to mimic and learn this ranking. During inference, this trained small retriever is directly used for retrieval without additional LLM calls. Experiments on multiple question-answering datasets and downstream generation models demonstrate that the proposed method improves both document retrieval and final question-answering performance.

Critical strengths

I think a critical strength of this paper is its label construction method when training the small retriever. So it leaves the use of the large models in the label construction phase and avoids LLM calls during inference. And the trained retriever also brings consistent downstream performance improvements.

Critical weaknesses

I feel confused about the use of the LLM in the retriever label construction phase. When scoring each candidate document, the paper also inputs the gold documents into the LLM. A possible consequence is that even in documents with no useful information, the LLM might still give high scores. So the use of the LLM might be redundant and potentially introduce noise. Experimental results also suggest that LLM scoring doesn't seem to substantially improve performance. If the LLM doesn't play a role, the proposed method may have limited innovation, as it appears to be simply a distillation of a large embedding model.

Questions for authors

In the experimental setup in the appendix, I saw that you mentioned using some test set examples for hyperparameter tuning. Why use test set examples for hyperparameter tuning, and were these examples excluded from the actual test set?

3 comments

Dear reviewers,

Will you please check the authors' response and reply it?

Best, AC

Thanks for the reply. It addressed my questions about hyperparameter tuning. But I still have concerns about the LLM scoring part. As mentioned in the authors' reply, it may add noise, and there is no additional evidence showing that it adds additional value over embedding-only scoring. So I plan to keep my score.

Additional Value of LLM Scoring. Thank you for the follow-up. We would like to clarify that Table 4 in the original manuscript provides empirical evidence for the incremental value of LLM scoring. All experimental settings (e.g., backbone, candidate documents, and training procedure) are held fixed, with only the supervision signal varied. Specifically, comparing our joint scoring with embedding-only scoring Pr⁡(d∣q,T∗)\Pr(\boldsymbol{d}\mid\boldsymbol{q},\mathcal{T}^{*}), which uses the same context, isolates the contribution of the LLM signal. Our joint scoring improves EM@5/R-L@5 on TREC, WebQ and NQ, while achieving comparable performance on TriviaQA. Moreover, joint scoring consistently outperforms the conventional baseline embedding-only signal Pr⁡(d∣q)\Pr(\boldsymbol{d}\mid\boldsymbol{q}) across all four datasets.

Supervision SignalTREC EM@5 / R-L@5WebQ EM@5 / R-L@5NQ EM@5 / R-L@5TriviaQA EM@5 / R-L@5
Ours joint: Pr⁡(d∣q,T∗)Pr⁡(a∗∣q,d,T∗)\Pr(\boldsymbol{d}\mid \boldsymbol{q},\mathcal{T}^{*} )\Pr( \boldsymbol{a}^{*} \mid\boldsymbol{q}, \boldsymbol{d},\mathcal{T}^{*} )0.310 / 0.4580.158 / 0.3100.241 / 0.3670.540 / 0.629
Baseline1: Pr⁡(d∣q)\Pr(\boldsymbol{d}\mid\boldsymbol{q})0.267 / 0.4000.151 / 0.3000.236 / 0.3620.519 / 0.604
Ablation1: Pr⁡(d∣q,T∗)\Pr(\boldsymbol{d}\mid\boldsymbol{q},\mathcal{T}^{*})0.302 / 0.4530.151 / 0.3040.233 / 0.3600.543 / 0.630
Baseline2: Pr⁡(a∗∣q,d)\Pr(\boldsymbol{a}^{*}\mid\boldsymbol{q},\boldsymbol{d})0.267 / 0.4060.155 / 0.3030.176 / 0.2700.301 / 0.353
Ablation2: Pr⁡(a∗∣q,d,T∗)\Pr(\boldsymbol{a}^{*}\mid\boldsymbol{q},\boldsymbol{d},\mathcal{T}^{*})0.224 / 0.3700.129 / 0.2580.153 / 0.2280.068 / 0.095

While LLM scoring might introduce noise, these results suggest that the LLM signal provides complementary information to embedding-based scoring, yielding modest but consistent improvements on three datasets without materially degrading performance on the fourth.

Comments on the paper

1 comment

We sincerely thank all reviewers for their constructive feedback. We have responded to the concerns and suggestion point by point, providing clarifications, additional evidence, and corresponding revisions. We would like to highlight the following key strengths and contributions of our work.

Novelty and Theoretical Contribution. Our work bridges the gap between document-level RAG theory and practical context-level RAG through a probabilistic marginalization framework. We derive an explicit context-conditioned, answer-aware joint top-k ranking criterion and propose JBR-C, which integrates joint ranking supervision with listwise learning-to-rank. Unlike existing approaches, JBR-C transfers this supervision into a standard dense retriever without additional LLM calls during inference.

Practical Effectiveness and Reviewer Feedback.

  • Reviewer mH4E identifies our label construction method as a "critical strength," highlighting that it avoids LLM calls during inference and brings consistent downstream performance improvements. In our point-by-point response, we further clarify that joint scoring outperforms embedding-only scoring.
  • Reviewer rn4t acknowledges the importance of aligning retrieval with downstream generation, the comprehensive evaluation across four datasets and three generators, and the practical advantage of retaining standard dual-encoder inference. We have addressed the reviewer's questions point by point, clarifying the theoretical scope, practical approximation, experimental comparisons, and evaluation protocol.
  • Reviewer dLNT recognizes our systematic generation-aware ranking supervision and its practical effectiveness. In response, we further distinguish our novelty from related work and conduct additional experiments on MultiHopRAG and HotpotQA, where JBR-C achieves the highest EM among dense retrievers. Our ablation also demonstrates that combining joint RI supervision with listwise learning increases the average relative improvement from 5.5% to 12.5%.

Overall, our theoretical formulation, practical inference efficiency, consistent improvements across four QA benchmarks, and additional multi-hop evaluations demonstrate the novelty and practical value of JBR-C. We sincerely hope these contributions and our detailed rebuttal will be considered in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.