acceptodds
Under review as a conference paper at ICLR 2027

When Does Retrieval Hurt? Characterizing Negative Retrieval in RAG

Abstract

Retrieval can supply knowledge that a language model lacks, but it can also overturn answers that the model already gives reliably without context. We study the second effect, answer preservation under retrieval, using an evaluation protocol that separates closed-book screening from fresh, paired closed-book and retrieval evaluation. On questions screened as reliably answered, the difference between retrieval error and fresh closed-book error measures net preservation loss; marginal error rates also give sharp bounds on harmful and beneficial answer changes. The protocol reports overall utility and preservation on explicitly defined question populations. On 3,250 questions from five QA benchmarks, top-10 retrieval raises the primary reader's overall accuracy by 9.5 points, yet raises its error on the 1,878 screened questions by 12.8 points relative to the fresh control. A matched intervention on 600 screened questions keeps complete supporting evidence and replaces a single distractor: in-context RAG makes 25, 47 and 111 errors with an unrelated, topical near-miss and conflicting distractor, respectively, whereas Astute RAG makes 14, 19 and 34. Supporting evidence therefore does not prevent content-dependent errors; the conflict-induced increase is 14.3 percentage points for in-context RAG and 3.3 for Astute RAG. Elementary probability identities connect these measurements and bound the paired harm that marginal error rates can identify.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Negative Retrieval, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t663OpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates when retrieval can harm questions that language models already answer correctly. The authors propose an evaluation protocol combining independent screening, fresh closed-book controls, and statistical bounds to measure negative retrieval. Experiments across five QA benchmarks show that retrieval can improve overall accuracy while causing previously correct answers to become incorrect. The paper further investigates the effects of different distractors and evaluates several existing methods for mitigating negative retrieval.

Critical strengths
  1. The research question is important, and the evaluation protocol is rigorous. Understanding when retrieval harms previously correct answers is valuable for developing reliable RAG systems. The proposed independent screening, fresh controls, and statistical analysis provide a careful way to measure this phenomenon.

  2. The paper is clearly organized with a logical experimental progression. It first establishes an evaluation protocol and quantifies negative retrieval, then investigates the effects of different distractors, and finally evaluates existing mitigation methods.

Critical weaknesses
  1. The paper provides limited new insights beyond existing studies. As acknowledged in Table 6, prior work such as Astute RAG [1] has already investigated correct-to-incorrect transitions caused by retrieval, while The Distracting Effect [2] has examined the impact of distractors even when supporting evidence is available. Although this paper introduces a more rigorous evaluation protocol, its main empirical findings largely confirm existing observations, and the additional insights enabled by the proposed protocol appear limited.

  2. The controlled distractor experiments are not directly connected to the negative retrieval observed in the main evaluation. Section 4.1 identifies substantial preservation loss under natural retrieval, whereas Section 4.2 constructs a separate setting that always includes complete supporting evidence and varies only one distractor. While these experiments demonstrate the effects of different distractors, they do not explain what actually causes the observed preservation failures. A direct analysis of the actual harmful cases from Section 4.1 would provide more convincing insights into the causes of negative retrieval.

[1] Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models. ACL 2025.

[2] The Distracting Effect: Understanding Irrelevant Passages in RAG. ACL 2025.

Questions for authors

How sensitive are the results to the number of closed-book screening calls? The paper uses three successful screening calls to identify questions that the model can reliably answer, but it is unclear whether three calls are sufficient to establish such reliability. Have the authors investigated how varying the number of screening calls affects the selected question population and the measured preservation loss?

Reviewer_CmaZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies whether retrieval preserves answers that LLM can answer it without context. An LLM can answer correctly in three screening calls from an evaluation subset. Separate closed-book and retrieval runs then measure how retrieval changes error on that subset. The paper also derives identities and bounds connecting these error rates to correct-to-incorrect and incorrect-to-correct answer changes. Experiments use 3,250 questions from five QA datasets. For the primary reader, top-10 context improves overall accuracy by 9.5 percentage points but increases error by 12.8 points on the 1,878 screened questions.

Critical strengths
  1. The closed-book evaluation addresses a specific measurement problem. A question answered correctly during screening can still receive an incorrect answer in a later call without retrieval. Section 2 accounts for this and avoids treating the screened set as direct evidence of what the model knows. The distinction between net error increase and harmful answer changes is useful: Table 2 implies at least 240 correct-to-incorrect changes even though retrieval produces 309 more correct answers overall. Aggregate accuracy alone would miss this.

  2. Retaining the supporting evidence while changing one distractor is a useful experimental design. It makes the comparison easier to interpret than comparing unrelated retrieved bundles. The reported increase of 86 errors for in-context RAG, compared with 20 for Astute RAG, also shows that the choice of inference procedure can affect preservation. The paper acknowledges that these procedures have different computational costs.

Critical weaknesses
  1. Section 4.2 and Appendix D describe the supporting bundles as complete and audited, but do not explain how the audit was conducted or how completeness was established. This is especially relevant for questions that require several supporting facts.

  2. The paper also leaves unclear how the distractors were produced, how their incorrect claims were checked, and how context length was matched. The description of the 600-question sample gives dataset quotas and requires membership in the screened set, but does not fully explain the selection procedure.

  3. There is a similar gap in the main evaluation. Section 3 preserves the passage ordering supplied with each question without identifying where these lists came from or how they were ordered. The context-availability filter is difficult to reconstruct, and Appendix C does not provide the actual prompts.

  4. For example understanding the large top-1 accuracy drop in Table 5 requires knowing what the first passage represents and how the reader is instructed to use it. The paper needs enough detail to establish what was held fixed and what changed.

  5. Prior work already examines joint closed-book/RAG correctness and distraction when supporting evidence is present. Wang et al. (https://aclanthology.org/2025.acl-long.1476/, Figure 1) report joint correctness categories. Amiraz et al. (https://aclanthology.org/2025.acl-long.892/, Section 4.4) evaluate weak and hard distractors alongside a gold passage. The contribution here therefore depends partly on what independent screening and the fresh closed-book reference reveal beyond those analyses.

  6. The paper does not show how requiring three correct screening calls changes the selected questions or the conclusions compared with requiring one. Appendix C specifies temperature 0 and the same inference seed for operational repeats. Under that configuration, the disagreement between screening calls would help explain what repeated screening accomplishes. The fresh-control correction is 0.91 percentage points on the reported set; subtracting the same correction also leaves the method rankings unchanged.

  7. The reporting of answer changes raises another question. The protocol evaluates both conditions for each question, yet Tables 2 and 8 report bounds based on marginal counts without the observed transition counts. The bounds are valid, but the paired results would show directly how many answers changed in each direction. Appendix E checks the aggregate arithmetic without supplying the records or computation needed to reproduce the paired bootstrap intervals.

Questions for authors
  1. Where do the ordered passage lists come from, and how was their order determined?

  2. Please specify the dataset splits, context-availability filter, and exact closed-book and retrieval prompts.

  3. How were the 600 questions and their distractors selected?

  4. Please describe the support audit and length matching, and provide actual experimental examples, including a multi-hop question.

  5. Can you provide the observed correct-to-incorrect and incorrect-to-correct counts for the paired evaluations, together with the bootstrap computation? If the question-level records are unavailable, please clarify which records were used for paired resampling.

Reviewer_ni98OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how often retrieval changes answers that a model already gets right without retrieved context. The authors first find questions the model answers correctly three times without context. They then compare fresh answers with and without retrieved passages. On 3,250 questions from five QA datasets, retrieval improves overall accuracy by 9.5 points but increases errors on these reliably answered questions by 12.8 points. A controlled experiment shows that adding a conflicting passage causes many more errors than an unrelated one, even when the correct evidence is present. Astute RAG is the most robust of the five mitigation methods tested.

Critical strengths
  1. The authors transparently present what they results can and can’t show. The paper is careful to say what can be concluded from overall error rates alone. It also separates “possible ranges” from statistical confidence intervals. Splitting the questions into groups narrows the estimated harm from 0 to 31% to 7.4 to 10.2%, a good practical example of the method.
  2. The authors also have some Useful finding for practitioners. Retrieval improves overall accuracy by 9.5 points, yet at least 240 previously correct answers become wrong. This shows clearly that average accuracy can hide many changed answers.
  3. Compehensive controlled experiment: In the controlled study, the correct evidence, context length, and evidence position stay the same, and only the distracting passage changes. This makes the effect of the distractor easier to isolate than with naturally retrieved passages.
Critical weaknesses

While the problem is interesting, the space has small significance. In large enterprise companies, systems that aid retrieval and search, already have proven ranking and extraction algorithms. Very rarely it is observed that the retrieving algorithms fail in listing relevant context that can be fed to a RAG system. The paper provides no evidence that such pipelines frequently pass irrelevant or misleading passages to the reader. So the question arises, how bad is it really?

Also, the measured “harm” may partly reflect the scoring method rather than real mistakes. This is my main concern. Answers are scored by string matching against reference answers, so several kinds of acceptable answers could be counted as errors: different wording- answers written with retrieved passages may be more detailed or phrased differently from the reference, Time-dependent answers: SituatedQA answers depend on date and place, so a passage may support a valid answer for a different time. This dataset shows one of the largest losses (19.6 % points). Incomplete reference lists: NQ-Open and PopQA often lack acceptable alternative answers. Refusals: answers like “the passages don’t say” are counted as wrong.

Lastly, all three models tested come from a single company and are available only through its API, so the results may not apply to other models and could change if the provider updates them. R1 also uses a lower reasoning setting than R2 and R3, so differences between the models can’t be cleanly compared.

Questions for authors
  1. Which retriever and reranker produced the passage orderings, and how does E_k change with a stronger pipeline (e.g., a dense retriever plus cross-encoder reranking)? How often do naturally retrieved top-10 contexts contain near-miss or conflicting passages?
  2. xWhere do the contexts come from for each dataset (retriever, corpus, inclusion of gold passages), and what is the answer-recall of the top-1, top-5 and top-10 prefixes? Does E_K vary with whether the context contains the gold answer?
  3. What are the retriever prompts?
  4. Would you be open to, for reproducibility concerns, release question, passages, and model outputs? Thank you for including the scripts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.