- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a retrieval objective for context-level RAG, where a final answer is generated directly from the top-k retrieved documents. The authors argue that existing document-level RAG formulations do not adequately account for interactions among multiple documents when generating an answer.
To address this limitation, the paper proposes JBR-C, a method for training a dense retriever using joint ranking supervision. For each question in the training set, a candidate set consisting of gold documents, hard-negative documents, and randomly sampled negative documents is constructed. Joint ranking scores are then calculated by combining document relevance estimated by an embedding model with the probability of generating the correct answer estimated by an LLM. The documents are ranked according to these scores, and the resulting rankings are used to train a dense retriever through listwise learning-to-rank (LL2R). Experiments on four QA datasets generally demonstrate improvements over the evaluated baselines in both retrieval quality and downstream answer generation performance.
Critical strengths
S1. Generation-aware Ranking Supervision: The paper presents a systematic approach to constructing ranking supervision by combining document relevance and the probability of generating correct answers. The improvements in retrieval and generation performance across four QA datasets demonstrate the practical effectiveness of the proposed approach.
Critical weaknesses
W1. Limited Baseline Coverage: The evaluated baselines are relatively conventional, and comparisons with relevant retriever training and ranking methods are missing. Prior studies have explored answer-aware supervision for retriever training [1, 3], shared-context attribution for estimating document utility [2], and layer-wise contrastive ranking [4]. Comparisons with these methods would help establish the effectiveness and novelty of the proposed RI-based supervision.
-
[1] Atlas: Few-shot Learning with Retrieval Augmented Language Models (JMLR, 2023)
-
[2] Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Models (EMNLP, 2025)
-
[3] ReSCORE: Label-free Iterative Retriever Training for Multi-hop Question Answering with Relevance-Consistency Supervision (ACL, 2025)
-
[4] Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS, 2025)
W2. Limited Evidence of Document Complementarity: Although JBR-C constructs ranking supervision using a comprehensive context, the trained retriever relies on independent query–document similarities during inference. It remains unclear whether this approach effectively retrieves complementary documents. Moreover, the lack of multi-hop QA evaluations (e.g., HotpotQA, 2WikiMultiHopQA, and MuSiQue) makes it difficult to assess whether the proposed method addresses its central motivation.
Questions for authors
Q1. Effectiveness of Ranking Indicator (RI)-based Supervision: What evidence shows that RI-based supervision improves the retrieval of complementary documents rather than merely improving individual document rankings? A comparison with +LL2R on multi-hop QA benchmarks would help validate the effectiveness of RI-based supervision in retrieving complementary evidence.
Dear reviewers,
Will you please check the authors' response and reply it?
Best, AC