- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how retrieved context affects question-answering performance in retrieval-augmented generation (RAG). Using 468 filtered Natural Questions examples, the authors compare closed-book and retrieval-augmented question answering across multiple models. They further conduct answer-bearing passage replacement, answer-span masking, and non-answer-span masking experiments to examine the role of answer information in retrieved documents. The paper also investigates how prompt instructions, correctness metrics, and refusal detectors affect the interpretation of experimental results. The findings show that retrieval can improve overall accuracy while also causing models to lose answers they could originally provide correctly, and that different experimental and evaluation settings can influence how this phenomenon is assessed.
Critical strengths
- The paper addresses the phenomenon that retrieval can cause models to lose previously correct answers, and measures both retrieval-induced answer loss and recovery. Compared with reporting aggregate accuracy alone, this analysis provides a more comprehensive understanding of how retrieval affects model behavior.
- The authors replace answer-bearing passages within genuine retrieval rankings and use answer-span masking and non-answer-span masking as controls to examine the impact of answer information on model performance. The findings are consistent across multiple model configurations, while paired statistical analyses and confidence intervals further strengthen the reliability of the results.
- The paper compares different correctness metrics and refusal detection methods, showing that the same model outputs can lead to different evaluation results. The authors also validate their main findings through semantic annotation and robustness analyses, making the experimental evaluation more comprehensive.
Critical weaknesses
- Prior studies have already examined how irrelevant retrieved content interferes with model responses and how retrieval can undermine answers that models could originally provide correctly. Although this paper uses more detailed intervention and control experiments to demonstrate the importance of answer information, its main findings remain relatively close to existing observations. The paper does not sufficiently clarify what new insights these experiments provide beyond stronger empirical evidence.
- The paper conducts answer-bearing passage replacement, answer-span masking, and matched non-answer-span masking experiments. However, these experiments mainly examine the effects of artificially removing answer information and provide limited explanations for why answer loss occurs under normal retrieval conditions. For example, why might a model lose a previously correct answer even when the retrieved documents contain the correct information? Is this caused by conflicting evidence, failure to effectively use retrieved information, or overreliance on retrieved context? The paper does not systematically distinguish between these factors, leaving the underlying mechanisms of retrieval-induced errors insufficiently explained.
- The experimental scope and generalizability of the findings are limited. The paper primarily uses 468 filtered single-hop Natural Questions examples, with the core intervention experiments further restricted to questions containing replaceable answer-bearing passages. Although multiple models are evaluated, the main experiments still rely on the same set of questions, and the full comparison of prompt instructions is conducted on only one model. The dataset is relatively small, and the paper lacks evaluations on other datasets, more complex question-answering tasks, and different retrieval systems. It therefore remains unclear whether the findings generalize to broader RAG scenarios.
Questions for authors
n/a