- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper shows that the effectiveness of evidence selection in RAG depends on the reader’s reasoning effort.
Critical strengths
N/A
Critical weaknesses
- The paper is hard to follow; after reading the abstract and introduction sections, I cannot figure out the main points/claims/results presented in the paper.
- The authors use saturated benchmarks (back to 2018) and a small GPT-5-mini, which does not reflect the current development of frontier models.
- The evaluation setup is limited.
Questions for authors
N/A
3 comments
Response to Reviewer auZB (2/2)
W2. Older benchmarks and gpt-5-mini. These benchmarks are not saturated in our setting, where the reader sees only five retrieved passages: the best accuracy of any policy and effort is 66.4% EM for gpt-5-mini on held-out 2Wiki (Table 1) and 50.3% for Qwen3-32B over 1,500 questions (Table 37). We use them because they annotate which passages support each answer, which the removal and restoration interventions need. Every comparison is also paired on the same questions, so memorization that is common to all policies and efforts cancels out of the interaction.
We chose gpt-5-mini because it exposes discrete effort settings and reports reasoning tokens, which the accuracy and cost analysis needs. To reduce the dependence on one model, the revision adds two open-weight reasoning models as readers, and the dependence on effort holds with both on 2Wiki, without the reversal (Table 2). We agree that a larger frontier reader is a natural next step, and Section 6 lists broader readers among the limitations.
W3. The evaluation setup is limited. The submitted paper tested this dependence with one reader. The revision adds two readers whose effort is set differently (a thinking budget and a thinking switch), selection by the reader's own model in place of the released selector, 250 more questions each for HotpotQA and MuSiQue with Qwen3-32B, the removal intervention, and a confirmation of answer reuse on separate questions. The dependence on effort is now tested with three readers on 2Wiki and two readers on all three benchmarks (Table 2). The benefit of selection falls with effort for all three readers on 2Wiki, and for Qwen3-32B also on MuSiQue; on HotpotQA the differences in Table 2 are positive, but their 95% intervals include zero. Details are in our replies to Reviewers yvd7 and vzoj.
We hope the revised abstract and introduction make the main claim clear, and we would be grateful if you would take them into account in your assessment. We are glad to answer any further questions.
Thank you for pointing out that the original introduction made the main finding hard to identify. The question is whether passages chosen for a reader at one reasoning setting remain a good choice when that reader is given more reasoning effort. On 2Wiki, the same passage selection helped GPT-5-mini at minimal effort and hurt it at medium effort. In our first reply, we described testing a frontier reader as a next step. We have now run that comparison: on the same questions and passages, the interaction was 0.2 points for GPT-6-astra (Astra), against 7.4 for GPT-5-mini (Section 5.2). We also extended the evaluation beyond Wikipedia. On 600 held-out MultiHop-RAG news questions, giving Astra the first three retrieved passages cost 62% less than asking it to choose three. Exact-match accuracy was 84.3% and 83.3%, respectively (difference +1.0 points [−0.7, 2.7], Section 5.6). A selector should therefore be compared with spending the same budget on the reader.
In response to the three concerns you raised, we revised the introduction to state the main finding more directly, tested a frontier reader, and extended the evaluation beyond Wikipedia to news articles. We hope these changes address your concerns and would be happy to answer any remaining questions about the results or their interpretation.
Response to Reviewer auZB (1/2)
Thank you for the feedback on clarity, the benchmarks and reader, and the scope of the evaluation. We revised the abstract and introduction so that the main claim and results appear earlier, and we extended the evaluation.
W1. Clarity of the main claim. The main claim is that the value of evidence selection depends on the reader's reasoning effort. We call this dependence the interaction. The paper shows three results.
In the revision, the first paragraph of the introduction ends with the main finding, the three contributions carry these labels (interaction, interventions, calibration), and we simplified dense sentences in the abstract and introduction.