acceptodds
Withdrawn from ICLR 2027

Two Context Regimes with Opposite Signs: Measuring the Retrieval Tax on Natural Questions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieved context can raise average question-answering accuracy while losing answers that a model gives without documents. We measure both directions on 468 Natural Questions examples with a retrieval tax and a rescue rate, each paired against a closed-book cell under the same prompt contract, the system instruction that states how documents may be used. Under exact dense retrieval over 2.68 million passages, five passages lose 14.0% of closed-book-correct answers and repair 49.1% of the incorrect ones. Inside the same kind of genuine rankings, replacing every answer-bearing passage by the next non-answer passage raises the tax from 0.133 to 0.593 for Qwen3-8B in preregistered within-question runs, and by 0.33 to 0.46 on four models and two contracts; blinded semantic labels that adjudicate each question’s reference once for all its answers give +0.313, with identification bounds above zero. Masking only the gold-answer spans in the unchanged passages produces about half of the containment-scored increase on two models and, under semantic labels, exceeds a matched non-answer mask on complete cases and on questions with an adequate reference; both edits keep their effect at a fourfold reasoning budget scored on the full answer. Uninformative placeholder context costs 43–62% of known answers, but Qwen and Mistral decline by citing the context while Llama-3.1-8B mostly answers “I don’t know”, so on identical outputs two refusal detectors rank Llama-3.1-8B lowest and highest in known-question over-refusal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval augmented generation, question answering, context effects, prompt contracts, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_i2GMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how retrieved context affects question-answering performance in retrieval-augmented generation (RAG). Using 468 filtered Natural Questions examples, the authors compare closed-book and retrieval-augmented question answering across multiple models. They further conduct answer-bearing passage replacement, answer-span masking, and non-answer-span masking experiments to examine the role of answer information in retrieved documents. The paper also investigates how prompt instructions, correctness metrics, and refusal detectors affect the interpretation of experimental results. The findings show that retrieval can improve overall accuracy while also causing models to lose answers they could originally provide correctly, and that different experimental and evaluation settings can influence how this phenomenon is assessed.

Critical strengths
  1. The paper addresses the phenomenon that retrieval can cause models to lose previously correct answers, and measures both retrieval-induced answer loss and recovery. Compared with reporting aggregate accuracy alone, this analysis provides a more comprehensive understanding of how retrieval affects model behavior.
  2. The authors replace answer-bearing passages within genuine retrieval rankings and use answer-span masking and non-answer-span masking as controls to examine the impact of answer information on model performance. The findings are consistent across multiple model configurations, while paired statistical analyses and confidence intervals further strengthen the reliability of the results.
  3. The paper compares different correctness metrics and refusal detection methods, showing that the same model outputs can lead to different evaluation results. The authors also validate their main findings through semantic annotation and robustness analyses, making the experimental evaluation more comprehensive.
Critical weaknesses
  1. Prior studies have already examined how irrelevant retrieved content interferes with model responses and how retrieval can undermine answers that models could originally provide correctly. Although this paper uses more detailed intervention and control experiments to demonstrate the importance of answer information, its main findings remain relatively close to existing observations. The paper does not sufficiently clarify what new insights these experiments provide beyond stronger empirical evidence.
  2. The paper conducts answer-bearing passage replacement, answer-span masking, and matched non-answer-span masking experiments. However, these experiments mainly examine the effects of artificially removing answer information and provide limited explanations for why answer loss occurs under normal retrieval conditions. For example, why might a model lose a previously correct answer even when the retrieved documents contain the correct information? Is this caused by conflicting evidence, failure to effectively use retrieved information, or overreliance on retrieved context? The paper does not systematically distinguish between these factors, leaving the underlying mechanisms of retrieval-induced errors insufficiently explained.
  3. The experimental scope and generalizability of the findings are limited. The paper primarily uses 468 filtered single-hop Natural Questions examples, with the core intervention experiments further restricted to questions containing replaceable answer-bearing passages. Although multiple models are evaluated, the main experiments still rely on the same set of questions, and the full comparison of prompt instructions is conducted on only one model. The dataset is relatively small, and the paper lacks evaluations on other datasets, more complex question-answering tasks, and different retrieval systems. It therefore remains unclear whether the findings generalize to broader RAG scenarios.
Questions for authors

n/a

Reviewer_ZjonOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper asks the question of how a forced retrieval affects can affect the results of an LLM in a question-answering scenario. It compares the trade-off between acquiring the necessary information to answer a question that requires extra context and the accuracy drop caused by useless text in the context.

Using 468 NQ questions, the authors evaluate various LLMs when they are presented with the question answering chunk (orcale1) and compare this with a few controls that present placeholder chunks. The authors also run a comparison in which the system prompt is changed. Answers are evaluated with LLM as judge.

Critical strengths

The experiments were executed carefully and allow understanding of the retrieval tax.

Critical weaknesses

The paper has no substantive novelty. Most of the claims that drive the paper are already known by the literature and the text does not add much to them.

  • Uninformative context decreases performance (retrieval tax) (Shi et al., 2023)
  • Changing the prompt while keeping the retrieval can also affect the result (Sclar et al., 2024)

From a presentation standpoint, the paper presents lots of results but not in a cohesive way that argument in favor of a hypotheses. A big part of the relevant results are only presented in Appendices and some elements are not properly defined (e.g. the “conservative” and “liberal” merge rules behind the two-judge estimates, the annotator and judge models behind all semantic results are not named)

Questions for authors

What would be an alternative hypothesis that this paper denies? What would be a practical contribution to RAG systems given the findings?

Reviewer_1YqyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies 468 Natural Questions examples and shows that retrieved documents can improve overall accuracy while causing models to lose answers they previously got right—a “retrieval tax.” High-recall retrieval repairs 49.1% of previously incorrect answers while losing 14.0% of correct ones, but removing answer-bearing passages raises the tax by 33–46 percentage points across tested models. It argues that RAG evaluation should report both gains and losses and account for how prompts, correctness metrics, and refusal detectors change the conclusions.

Critical strengths
  • Retrieval tax vs rescue is an important and interesting problem
Critical weaknesses
  • Poorly written abstract: Abstract should convey the overview of the paper. The abstract of this work lays out detailed experiment results without any end conclusion.
  • Lack of clear findings: Title is written as "measuring" the retrieval tax, and from reading the introduction, I do not find any findings or messages the author want to give through this work. It feels like this work merely contains the "measurement", as the title suggests. I do not think the measurement shown in this work alone suffices for a publication at ICLR. Why are these measurement interesting? (There is a chance that I have missed findings/messages the author have written in the body sections, but there was no such offered in the introduction. For such case, then the introduction is very poorly written, which is an enough reason for rejection.)
Questions for authors
  • What is the reason for only using NQ, not other QA datasets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.