acceptodds
Under review as a conference paper at ICLR 2027

RAG-Stress: Probing the Limits of Evidence Reliance in Retrieval-Augmented Generation

Abstract

We introduce RAG-Stress, a controlled diagnostic protocol for examining the limits of evidence reliance in retrieval-augmented generation. Misleading retrieved evidence can induce a model to replace an answer it previously gave correctly, a failure that aggregate accuracy alone does not distinguish from preexisting errors. The protocol holds the question and reference answer fixed, edits one assertion to support a designated incorrect answer, and crosses two source priority policies with three positions of the answer span within the evidence text. We measure misleading rate (MR) on each model's subset of questions answered correctly without retrieval, alongside clean accuracy on the full evaluation set. We evaluate fifteen systems spanning API models, open models, and search agents trained with reinforcement learning on TriviaQA-RC, HotpotQA, and SearchQA, with additional English and Chinese MedQA evaluations. Instructions that prioritize documents consistently produce higher MR than those permitting reliance on prior knowledge. Averaged over models and positions, the gap ranges from 10.9 to 13.5 percentage points across the three QA datasets. Averaged equally across models, MR follows End Beginning Middle under both policies on all three QA datasets; this ordering is not universal across individual models. A separate paired audit of 500 questions and two checkpoints supports increased harmful override under Strict relative to Soft, while the corresponding instruction contrasts in beneficial correction remain inconclusive. These findings distinguish evidence adherence from factual reliability and motivate evaluating whether retrieved evidence preserves, replaces, or corrects a model's answers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, large language models, knowledge conflicts, factual reliability, robustness evaluation, counterfactual evidence

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_GYQaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a RAG testing protocol. For a given question and reference answer, it constructs two types of passages: a "clean" passage and a "foil" passage which contains a modified assertion that supports an incorrect answer. The protocol first uses a closed-book setting to identify the subset of samples (K) that each model answers correctly by default, then measures the "Misleading Rate" (MR)—the proportion of cases where the model switches from the correct answer to the foil after being presented with misleading evidence. The experiments employ a cross-factorial design involving two types of source-priority instructions and varying the position of the answer-supporting assertion within the evidence.

Critical strengths

Factors are controlled. For a given problem, only a single assertion is altered; the instruction type and assertion position form a full factorial design.

The experiment settings include multiple LLMs. The paper compares seven API-based models, four open-source models, and four RL-search systems, replicating the protocol across TriviaQA-RC, HotpotQA, SearchQA, and medical questions in both Chinese and English.

Human auditing enhances measurement reliability. Manual verification of 1,000 edited samples showed that 88.1% met the target criteria.

Critical weaknesses
  1. The evaluation confuses factual robustness with instruction following. In the Strict setting, the model is explicitly told to trust the document. The model is not told that the document has been modified. Therefore, producing the foil answer may simply mean that the model follows the instruction correctly. It does not necessarily indicate a robustness failure.

  2. The evaluation relies on an oracle that is usually unavailable in RAG applications. Users query a RAG system because they do not know the answer and do not fully trust the model’s internal knowledge. At deployment time, they usually cannot determine whether the retrieved document or the model’s internal answer is correct. However, this benchmark can label the retrieved document as corrupted only because the evaluator already knows the ground-truth answer. The paper does not explain how a real system could decide which source to trust without such post-hoc supervision. This substantially limits the practical value of the evaluation.

  3. The proposed diagnostic protocol is too limited for the paper’s main contribution. Even as a controlled study of cases where the model initially knows the answer but receives incorrect evidence, the experiments cover too few task types and datasets. The protocol should also examine important generation settings, such as decoding temperature, prompt variations, and model scale. Instead, the main interventions are only two intuitive instructions, Soft and Strict, and three positions for the modified statement. The observed differences are largely expected, while the analysis remains descriptive and does not provide deeper explanations for the model behavior.

Questions for authors
  1. Design a mechanism to determine whether a question should be answered "closed-book" or "open-book" without knowing the ground-truth answer. And, in the case of "open-book" tasks, assess the trustworthiness of the retrieved content without supervision.

  2. The paper is difficult to follow.

2 comments

Thank you for your comments. Main-table clean accuracies have an unresolved provenance conflict with a file labeled synthetic; they do not support the responses below.

Responses to Weaknesses

W1. Instruction following versus factual robustness. A compliant response can still be incorrect relative to the reference. In the verified supplementary audit, end-position Neutral MR is 65.1%/65.6% for Llama/Qwen, showing that foil adoption is not confined to Strict.

W2. Dependence on an oracle. Evaluator reference/foil labels are hidden from the model and used only for offline scoring. The protocol does not assume that a deployed system knows which source is correct.

W3. Contribution and experimental scope. The contribution is baseline-conditioned measurement of designated-foil transitions, rather than discovering knowledge conflict. Broader claims about task coverage, decoding robustness, model scale, and internal mechanisms require corresponding verified evidence.

Responses to Questions

Q1. Oracle-free routing and trust assessment. We do not provide a validated oracle-free router or unsupervised trust estimator. The tested Verify prompt does not solve these problems. RAG-Stress instead provides a diagnostic for evaluating proposed mechanisms through answer preservation, foil adoption, and correction of baseline failures.

Q2. Readability. We will organize the explanation around closed-book, clean-evidence, and edited-evidence responses, with separate metrics, explicit denominators, and a clear distinction between the main evaluation and supplementary audit.

Thank you for your reply. The evidence reliance related topic is practical and meaningful. I believe that this topic works after carefully revision.

Reviewer_jDToOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

RAG evaluation usually only looks at whether the accuracy improves after adding retrieval or whether the answers are supported by evidence. The authors believe this is insufficient: when the evidence appears relevant and internally consistent, but supports the wrong answer, the model may "have evidence but still answer incorrectly. Core findings are close to expectations, with limited novelty, and key evidence is insufficient.

Critical strengths
  1. The paper cleanly separates evidence adherence from factual reliability via a baseline-conditioned "correct → foil" metric, and consistently avoids subtracting metrics with different denominators (MR, clean accuracy, HOR, BCR).
  2. Broad coverage (15 systems, 3 QA datasets, bilingual MedQA); the finding that RL search agents retain high foil adoption even under Soft is practically relevant.
Critical weaknesses
  1. Strict explicitly instructs the model to follow documents "even if mistaken," so higher foil adoption is largely instruction compliance rather than vulnerability; Soft is restricted to "well-known facts." There is no neutral-prompt baseline, so claims about "default evidence reliance" are unsupported.
  2. Baseline correctness comes from a single greedy answer; lucky guesses (especially in multiple-choice MedQA) count as "knowing"
  3. Word-Position construction on real passages is unspecified (sentence reordering may break discourse; recency to the question is confounded).
  4. The paper is hard to read, heavy hedging and repeated disclaimers.
Questions for authors
  1. What are the per-dataset question counts, the number and selection of distractor documents, and the per-model |K_M|?
  2. How exactly is Word Position implemented on real passages, and how do you verify span location and control length and coherence?
  3. Which LLM generated the edits? Is there same-family self-preference? Do results hold with human-written or alternative-generator edits?
  4. What are the bases and sizes of the RL agents, how are their output formats handled with search disabled, and is there a size-matched non-RL baseline?
  5. What are the exact API snapshots, query dates, and parameters of the proprietary models?
1 comment

Thank you for your comments. Main-table clean accuracies have an unresolved provenance conflict with a file labeled synthetic; the empirical evidence below concerns the separately verified supplementary audit.

Responses to Weaknesses

W1. Instruction compliance and neutral baseline. Compliance and reference-defined error can coexist. In the supplementary audit, end-position Neutral MR is 65.1%/65.6% for Llama/Qwen, versus Soft’s 53.0%/58.3%. Thus, foil adoption is not confined to Strict. We will avoid interpreting either prompt as a universal default.

W2. Lucky baseline guesses. A single correct answer does not establish stable knowledge. We will consistently describe K as the observed closed-book-correct subset.

W3. Word Position construction. The verified audit changes document slots, not within-document Word Position. The latter still requires its construction, span-location, and coherence records.

W4. Readability. We will separate metrics, state denominators explicitly, and consolidate repeated qualifications.

Responses to Questions

Q1. Counts and distractors. The supplementary audit uses 500 shared questions, with |K|=338/259 for Llama/Qwen and three answer-filtered distractors per context. Main-study denominators remain unverified.

Q2. Position implementation. The audit places the target passage in slots 1, 2, or 4 while preserving distractor order. This does not verify the main within-document transformation or its length/coherence controls.

Q3. Edit generator. The recovered implementation documents a Llama rewriter, but complete generation provenance remains unresolved. Human-written and alternative-generator robustness has not been verified.

Q4. RL agents. Exact bases, sizes, output parsing, and matched non-RL controls require original records. Search-disabled results concern fixed-evidence behavior and do not isolate an RL-training effect.

Q5. API configuration. Historical snapshots, query dates, generation parameters, and failure handling require the original request logs; display names alone cannot establish them.

Comments on the paper

1 comment

Thank you for your suggestions. Main-table clean accuracies have an unresolved provenance conflict with a file labeled synthetic; we therefore rely on the separately verified supplementary records below.

Responses to Weaknesses

W1. Ecological validity. The fixed-context audit does not establish performance under live retrieval, conflicting independently selected sources, or persistent memory. We will state this scope explicitly.

W2. Position confounds. Placement sensitivity does not identify attention, salience, or recency mechanisms. The supplementary document-slot intervention is also distinct from within-document Word Position.

W3. Uncertainty. New conditional analyses use 20,000 question-level bootstrap resamples. Their exploratory pointwise intervals do not supply uncertainty for the main tables.

W4. Matching errors. Replay verifies scoring calculations, not semantic correctness. Alias and hedge errors remain possible.

W5. Mitigation connections. We evaluated one checking-and-abstention prompt, not a trained verifier or leave-one-out defense. Broader mitigation comparisons remain open.

Responses to Questions

Q1. Edit granularity. No matched multi-assertion or distributed-edit comparison is available. The current findings cannot establish robustness across edit granularities.

Q2. Live-search agents. No verified live-search RL-agent experiment is available. Whether high Soft MR persists under agent-controlled evidence selection remains unresolved.

Q3. Compliance signals. Citation behavior was not separately analyzed. The short-answer protocol makes citation absence an unsuitable noncompliance indicator.

Q4. Release. A replay package containing fixed evidence, labels, scoring code, and 17,000 saved requests and outputs is prepared locally. It has not yet been publicly released.

Q5. Clean-correct conditional MR. Yes. Across Soft/Strict and three document slots, conditional foil adoption is 1,123/1,742=64.5% for Llama and 872/1,373=63.5% for Qwen. These are correlated question–condition pairs; bootstrap resampling preserves conditions within each question. The endpoint measures loss of preserved correctness, not net utility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.