- Rating
- 2
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates when retrieval can harm questions that language models already answer correctly. The authors propose an evaluation protocol combining independent screening, fresh closed-book controls, and statistical bounds to measure negative retrieval. Experiments across five QA benchmarks show that retrieval can improve overall accuracy while causing previously correct answers to become incorrect. The paper further investigates the effects of different distractors and evaluates several existing methods for mitigating negative retrieval.
Critical strengths
-
The research question is important, and the evaluation protocol is rigorous. Understanding when retrieval harms previously correct answers is valuable for developing reliable RAG systems. The proposed independent screening, fresh controls, and statistical analysis provide a careful way to measure this phenomenon.
-
The paper is clearly organized with a logical experimental progression. It first establishes an evaluation protocol and quantifies negative retrieval, then investigates the effects of different distractors, and finally evaluates existing mitigation methods.
Critical weaknesses
-
The paper provides limited new insights beyond existing studies. As acknowledged in Table 6, prior work such as Astute RAG [1] has already investigated correct-to-incorrect transitions caused by retrieval, while The Distracting Effect [2] has examined the impact of distractors even when supporting evidence is available. Although this paper introduces a more rigorous evaluation protocol, its main empirical findings largely confirm existing observations, and the additional insights enabled by the proposed protocol appear limited.
-
The controlled distractor experiments are not directly connected to the negative retrieval observed in the main evaluation. Section 4.1 identifies substantial preservation loss under natural retrieval, whereas Section 4.2 constructs a separate setting that always includes complete supporting evidence and varies only one distractor. While these experiments demonstrate the effects of different distractors, they do not explain what actually causes the observed preservation failures. A direct analysis of the actual harmful cases from Section 4.1 would provide more convincing insights into the causes of negative retrieval.
[1] Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models. ACL 2025.
[2] The Distracting Effect: Understanding Irrelevant Passages in RAG. ACL 2025.
Questions for authors
How sensitive are the results to the number of closed-book screening calls? The paper uses three successful screening calls to identify questions that the model can reliably answer, but it is unclear whether three calls are sufficient to establish such reliability. Have the authors investigated how varying the number of screening calls affects the selected question population and the measured preservation loss?