acceptodds
Under review as a conference paper at ICLR 2027

VeriRAG: Verifying Answer Updates for Robust RAG

Abstract

In retrieval-augmented question answering, a system may revise its retrieval-free initial answer using retrieved passages. Such updates can correct model errors, but corpus poisoning and indirect prompt injection can also make them harmful. A verifier can guide this decision by checking contextual support, yet may continue to endorse a candidate even after necessary support is removed. We introduce VeriRAG, a training-free, attack-label-free method that requires two complementary verifier responses before adopting a candidate: continued endorsement when complete support is preserved, and an insufficient-evidence judgment when necessary support is withdrawn. For questions whose answers require a finite set of factual relations to hold jointly, VeriRAG derives these relations from the question alone and selects the candidate to test from the candidate answer strings, without showing the retrieved passages to this selection step. Holding the question and candidate fixed, it constructs paired contexts for each relation: one preserves complete support, while the other removes all support for that relation but retains joint support for the others. All pairs are audited and frozen before response testing. Adoption requires the expected responses on every pair, complete and unopposed support in the original context, valid citations, and no fully supported competing answer. Any failed or unresolved check leaves the initial answer unchanged. Through this process, VeriRAG allows retrieval to correct initial errors while reducing harmful answer updates. Experiments across three datasets and three language models demonstrate improved robustness under mixed corpus-poisoning and indirect prompt-injection attacks, with an average end-to-end accuracy of 71.0% and an attack success rate of 5.6%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Retrieval-Augmented Generation, Knowledge Poisoning, Robust RAG, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VfPmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VeriRAG verifies a retrieved candidate before using it to replace an initial answer. It accepts an update only when the verifier responds appropriately to controlled changes in supporting evidence.

Critical strengths
  1. The method tests whether a verifier notices missing support. This addresses a failure that checking the original context alone can miss.
  2. The evidence edits are audited and frozen before response testing. This gives the intervention procedure a clear separation from its outcomes.
  3. The Keep Initial baseline helps assess whether the method makes useful corrections. The repeated-verification control also examines the benefit of changing the evidence.
Critical weaknesses
  1. RC-RAG [1] already uses counterfactual prompting to decide whether a RAG answer should be retained. Please compare with an adaptation that returns the initial answer when RC-RAG rejects an update.
  2. The recognizer must identify a factual basis from its existing knowledge before evidence verification begins. How does this requirement affect questions about genuinely unfamiliar facts? Please evaluate newly introduced knowledge for which the retrieved evidence is sufficient.
  3. Appendix A requires a nonempty preservation edit near the removed support. This may make valid updates depend on how much expendable text surrounds a fact. Could the authors compare concise evidence with versions containing irrelevant padding while keeping the factual content unchanged?
  4. Can an attacker block a valid correction by adding one apparently supported competing answer? Please evaluate this case when the initial answer is wrong. Target-answer ASR does not capture an attack that succeeds by preventing correction.

[1] Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework.

Questions for authors

See Weaknesses.

4 comments

We thank the reviewer for four concrete requests. Each led to a new experiment. New results use Qwen3.5-9B with the queries and contexts of Table 1; percentages are ten-run means unless marked as a single run.

1. Comparison with RC-RAG adapted to return the initial answer

We agree this comparison was missing. Under the strong attack, VeriRAG is 3.8–6.9 points higher in ACC and 5.1–10.2 points lower in ASR than RC-RAG with the requested fallback.

We implemented RC-RAG + Fallback: RC-RAG's counterfactual prompting decides whether the retrieved answer is kept, and when it rejects, the system returns the same retrieval-free answer a0a_0 that VeriRAG falls back to. Our adaptation uses three counterfactual prompts (the published two, plus one that assumes the answer just given is wrong), keeps the answer only when all three reproduce it, and omits the published fusion call. Both methods share initial answers, contexts, and attack targets. They judge different answers, RC-RAG its own generated answer and VeriRAG a recognized candidate, so this is a comparison of two complete update procedures.

DatasetMethodClean ACCStrong ACCStrong ASR
NQRC-RAG + Fallback76.3856.3412.83
NQVeriRAG78.1262.042.63
HotpotQARC-RAG + Fallback75.7657.1215.47
HotpotQAVeriRAG77.9664.037.04
MS-MARCORC-RAG + Fallback65.2360.4412.04
MS-MARCOVeriRAG65.5464.286.97
  • VeriRAG leads in all six cells; the clean margin is 0.3–2.2 points of ACC.
  • On NQ, RC-RAG + Fallback ends 2.4 points below the retrieval-free answer alone (58.7%), and VeriRAG 3.3 points above it.
  • Cost per query: RC-RAG + Fallback uses 4 calls and 5.06K tokens; VeriRAG uses 34.2 calls and 22.80K tokens (8.6 times the calls, 4.5 times the tokens).

RC-RAG treats a stable regenerated answer as reliable, and an injected instruction or a repeated poisoned claim can also produce a stable answer. VeriRAG changes the evidence and requires the verifier to withdraw its endorsement once the support is removed. We will add this comparison to the main text.

2. Genuinely unfamiliar facts with sufficient retrieved evidence

The requirement does restrict such updates. On 384 questions about newly introduced facts, the submitted method reaches 12.76% ACC under the strong attack and makes 6 corrections.

We built 384 questions about fictional entities, each described in one supporting passage, with attack passages from the strong-attack procedure of Table 1 (single run). The initial answer is correct on 44 questions (11.46%). A correct candidate with sufficient evidence is available for 371 questions under clean retrieval and 318 under the strong attack.

We also evaluated an optional Evidence Admission mode. It admits any well-formed candidate with a registered source, without parametric recognition, and tests admitted candidates in passage-rank order until one passes every check. The mode changes both admission and candidate selection, and the results do not separate the two.

MethodClean ACCStrong ACCStrong ASRCorrections under strong attackTarget answers returned
Adopt candidate without verification92.7137.7657.55140221
VeriRAG as submitted (closed-book recognition)13.0212.761.5666
Evidence Admission, without P/W response tests88.5461.7222.1422885
Evidence Admission, full87.7670.313.6522614
  • Evidence Admission makes 226 corrections at 3.65% ASR.
  • The response tests lower ASR by 18.49 points, raise ACC by 8.59 points, and reduce harmful overwrites of initially correct answers from 35 to 0.
  • On the main NQ setting under the strong attack, the mode reaches 61.37% ACC and 6.84% ASR, against 62.04% and 2.63% for the closed-book gate.

We will add it as an optional mode for unfamiliar facts and state the recognition requirement as a limitation.

We thank the reviewer again for the constructive requests. This part continues our response with the third and fourth points.

3. Dependence on expendable text around the fact

The reviewer's reading of Appendix A is correct. With the facts unchanged, concise evidence yields 31.0 points less correction coverage than the same evidence with irrelevant padding.

The preservation edit must be nonempty and matched to the withdrawal edit in source, length, and position. If a passage contains little more than the supporting statement, no such edit exists and the update falls back to a0a_0.

We took 126 initially incorrect questions whose correct candidate is recognized and supported under clean retrieval, and rewrote the supporting passages in three forms with identical facts (single run): the supporting sentences only; the same with 40–60 tokens of unrelated text around them; and the same with the statement repeated once in different wording.

Evidence formQueries with all pairs audit-acceptedFailures due to the nonempty/matching constraintsCorrect adoptionsIncorrect adoptions
Concise7848740
Irrelevant padding11971130
Redundant padding108181030

Concise evidence yields 74 corrections of 126 (58.7%) and irrelevant padding 113 (89.7%). Of the 41 queries that padding recovers, 39 end in a correct adoption. The method needs passages with some text around the supporting statement, and we will state this in a Limitations paragraph and in Appendix A.

4. Blocking a valid correction with a competing answer

Yes. One completely supported competing passage blocks 93.0% of otherwise valid corrections. Every blocked query returns the initial answer, and none returns the competitor.

The adoption rule vetoes an update whenever the context gives complete support to another answer value (Eq. 5); in the submission this gate blocks 41 of 1,000 NQ queries under the strong attack (Table 11). We took 264 initially incorrect questions, of which VeriRAG corrects 129 under clean retrieval, and added one well-formed passage asserting a competing answer for the same entity (single run). The Correction Blocking Rate (CBR) is the share of those 129 corrections that no longer occur.

Added passageCorrections under clean retrievalStill correctedBlocked (CBR)Blocked at competing-answer gateBlocked at another gateCompetitor returned
None1291290000
Competitor with partial support12910029 (22.5%)6230
Competitor with complete support1299120 (93.0%)12000

With partial support, 23 of the 29 blocks occur at the original-evidence requirement, because the passage makes the candidate's support opposed. This is an availability attack: the attacker can hold the system at its retrieval-free answer but does not get the competitor adopted. We will report CBR alongside ASR and list correction blocking in the Limitations paragraph.

Dear Reviewer VfPm,

I hope this message finds you well. As the discussion period ends in less than three days, we wanted to check that our response has addressed your concerns. In brief:

  1. RC-RAG. With the requested fallback, VeriRAG is 3.8–6.9 points higher in ACC and 5.1–10.2 points lower in ASR under the strong attack on all three datasets.
  2. Unfamiliar facts. The submitted method reaches 12.76% ACC on 384 new-fact questions; an optional Evidence Admission mode reaches 70.31% at 3.65% ASR.
  3. Surrounding text. With the facts unchanged, concise evidence yields 74 corrections of 126 and padded evidence 113.
  4. Correction blocking. One competing passage blocks 93.0% of valid corrections; every blocked query returns the initial answer, never the competitor.

If anything remains unclear, please let us know and we will respond promptly. Thank you for your time and effort in reviewing our paper.

Best regards, The Authors

Sorry for the late reply, and thank you for the additional experiments. I remain concerned that the original method can block useful updates even without attacks, limiting RAG’s ability to use new knowledge. I hope the authors can further address this limitation in the revised paper.

For now, I lean toward maintaining my original score.

Reviewer_NncfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VeriRAG, a training free framework to decide whether initial model answer can be replaced using retrieved evidence with corpus poisoning and prompt injection. The approach derives facts from the question and constructs audited pairs with the verifier with two views: preservation view retains supporting evidence where withdrawal view removes. A candidate answer is accepted only if the verifier endorses in preservation and reports insufficient evidence under withdrawal. Otherwise, the initial answer is retained. The paired tests check whether the verifier endorses when support is preserved and withdraws endorsement when required support is removed. This is how the retrieval-based answer update happens in this proposed approach.

The paper presents experiments using three QA datasets and three language models. The evaluations show improved accuracy and reduced attack success compared to existing RAD defense approaches. The paper is relatively easy to read and understand. It shows positive results over baselines. However, the main building block of the approach seems incremental compared to existing literature. Counterfactual verification ideas exist and hence, VeriRAG approach seems a carefully controlled verification pipeline. For example, Eraser (ACL 2020) for preservation – withdrawal setting. Even though the paper’s approach of mitigating poison attacks is simple but clever, the technique does not deviate much from other existing approaches.

Critical strengths

• Relatively easy to follow the paper. Idea is simple and clever.

• Paper shows positive results over the selected baselines.

Critical weaknesses

• Main weakness is that the model builds on top of ideas of existing evidence removal and counterfactual verification ideas. It sounds like a more carefully controlled verification procedure except a novel concept/process. Hence, excitement is a bit limited.

• Before checking retrieved evidence, model must recognize candidate answer with existing knowledge. This can be problematic in unfamiliar domains or datasets.

• Model is computationally expensive, even though the approach is simple.

Questions for authors

• The candidate-recognition stage requires the model to identify a factual basis for a candidate without access to retrieved evidence. Could this restrict useful updates for facts absent from the model’s parametric knowledge, such as private organizational information or newly introduced domain facts? Have you evaluated cases where the initial answer is incorrect and the retrieved evidence is sufficient, but the model cannot independently recognize the correct candidate?

• Along the above question, can this approach be adapted to check whether generated answer is with the influence of poisoned corpus or not? That is, we do not have an initial model answer.

3 comments

We thank the reviewer for the careful reading. We answer the main concern about novelty first, then the two questions, then cost.

W1: What is new relative to evidence removal and counterfactual verification

We do not claim evidence removal or counterfactual verification as new. The contribution is the set of controls that make a removal test usable as an answer-update rule under contaminated retrieval.

ERASER removes or isolates a rationale and measures the change in model confidence, as a metric of explanation faithfulness. SURE-RAG compares support for a fixed answer on full and degraded evidence, as an offline diagnostic. RC-RAG regenerates the answer under the assumption that it was wrong and keeps it if unchanged. None of these decides whether a retrieved answer should replace the model's own answer when the retrieved text may be poisoned.

The finding that motivates the paper is that a verifier keeps endorsing a candidate in 21.9–29.7% of responses after audited edits remove the support it requires (Figure 3). No check of the original context can observe this. Turning removal into a decision requires the following:

RequirementERASER-style rationale removalVeriRAG
The answer under test is fixedThe prediction is fixedThe candidate is fixed before any evidence-dependent test
What must be removed is defined in advanceA given rationale spanThe relations the question requires, derived from the question alone
Removal is completeThe same claim elsewhere is untouchedEvery edit is audited against all passages for duplicate support
Removal isolates one fact—One relation is withdrawn while the others stay jointly supported
The test cannot be tuned to its outcome—Views are frozen before the verifier is queried
The result is a decisionA faithfulness scoreAdopt only if every test passes; otherwise keep the initial answer

These controls have measured effects:

  • Audited withdrawal yields a valid view for 72.5–79.0% of records, against 56.0–61.0% for deleting the cited span alone (Figure 3).
  • Requiring the withdrawal response lowers mean ASR from 9.06% to 5.63% in the same pipeline (Table 9).
  • Withdrawing one relation at a time, against treating the question as a single relation, is worth 5.5 points of ACC on NQ, 5.7 on MS-MARCO, and 9.7 on HotpotQA on multi-relation questions, and 3.2, 5.2, and 6.6 points of ASR (new ablation; table in our response to Reviewer wTgs, Q1).
  • Repeating checks of the original context at the same budget gives lower ACC in all six attacked settings and higher ASR in five (new, three datasets; on NQ, 60.8% against 62.0%, Table 12).
  • Showing the passages to the recognizer raises ASR by 3.5–3.9 points (Table 3).

We will rewrite the related-work section and the contribution list in these terms.

W2 and Q1: Does closed-book recognition restrict updates for facts absent from parametric knowledge?

Yes, and we had not measured it. On 384 questions about facts the model cannot know, the submitted method reaches 12.76% ACC under the strong attack and makes 6 corrections.

The set consists of questions about fictional entities, each described in one supporting passage (single run). The initial answer is correct on 11.46% of them, and sufficient evidence for the correct answer is in the context for 371 questions under clean retrieval and 318 under the strong attack. We also evaluated an optional Evidence Admission mode that admits candidates without parametric recognition and tests them in passage-rank order until one passes every check.

MethodClean ACCStrong ACCStrong ASRCorrections under strong attack
Adopt candidate without verification92.7137.7657.55140
VeriRAG as submitted (closed-book recognition)13.0212.761.566
Evidence Admission, without P/W response tests88.5461.7222.14228
Evidence Admission, full87.7670.313.65226

On the main NQ setting under the strong attack, Evidence Admission reaches 61.37% ACC and 6.84% ASR, against 62.04% and 2.63% for the closed-book gate, so the two are different operating points. We will add the mode as an optional setting.

We thank the reviewer again for the thoughtful questions. This part continues our response with the second question and the concern about cost.

Q2: Can the approach check an answer for poisoning influence when there is no initial answer?

Yes, as a selective screen for an already generated answer. It does not identify which passages are poisoned.

If the model can be queried, nothing changes: the method produces the initial answer itself with a retrieval-free call. If an answer has already been generated and must be audited as it stands, the decision becomes accept or abstain. In this Candidate-Only mode the answer is accepted only if it passes all evidence checks and the P/W response tests.

On NQ (1,000 queries, single run), auditing the VanillaRAG answer:

ConditionDirect output ACCDirect output ASRAccepted (correct / incorrect)AbstainedCoverageAccuracy on acceptedASR after screening
Clean65.80—579 / 4138062.00%93.39%—
Strong attack4.6092.8025 / 159604.00%62.50%1.50%

Under the strong attack, 913 of the 928 target answers are rejected, along with 21 of the 46 correct ones. We report this mode as a diagnostic. A pass means the answer is consistent with our evidence checks; an abstention means verification was insufficient, not that the answer is wrong.

W3: Computational cost

The method is expensive, and we will say so in the main text. Full VeriRAG uses 22.80K tokens and 34.2 calls per query, 3.65 times SeCon-RAG's tokens (Table 26); the response tests account for 1.60K of those tokens. The same compute spent elsewhere does not buy the same result: repeated verification of the original context costs 23.15K tokens and is less accurate in all six attacked settings, and self-consistency with 34 sampled answers costs about 35K tokens and reaches 6–14% ACC under the strong attack.

Dear Reviewer Nncf,

I hope this message finds you well. As the discussion period ends in less than three days, we wanted to check that our response has addressed your concerns. In brief:

  1. Novelty (W1). We do not claim evidence removal as new. The contribution is the set of controls that make a removal test usable as an answer-update rule under contaminated retrieval, each with a measured effect; for example, requiring the withdrawal response lowers mean ASR from 9.06% to 5.63%.
  2. Unfamiliar facts (W2, Q1). The submitted method reaches 12.76% ACC on 384 new-fact questions; an optional Evidence Admission mode reaches 70.31% at 3.65% ASR.
  3. No initial answer (Q2). A Candidate-Only mode screens an already generated answer: the attack target survives on 1.50% of queries, against 92.80% unscreened.
  4. Cost (W3). The method is expensive, and we will say so; the same compute spent on repeated verification or self-consistency does worse on all three datasets.

If anything remains unclear, please let us know and we will respond promptly. Thank you for your time and effort in reviewing our paper.

Best regards, The Authors

Reviewer_wTgsOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

VeriRAG is a training-free defense for RAG under corpus poisoning and indirect prompt injection. It frames the decision as whether to replace a retrieval-free initial answer with a candidate proposed from retrieved passages. The method derives the required factual relations from the question alone, picks a candidate from anonymous answer strings, and builds audited preservation and withdrawal views of the context for each relation. A candidate is adopted only if the verifier endorses it on preservation views, reports insufficient evidence on withdrawal views, and the original context passes support, citation and competitor checks; otherwise the initial answer is kept. Across three models, three QA datasets and 18 attacked settings, VeriRAG averages 71.0% accuracy and 5.6% attack success rate, compared with 67.9% and 11.5% for SeCon-RAG.

Critical strengths
  • Well-motivated verification principle: Checking that a verifier's endorsement disappears when necessary support is removed targets a real failure mode (endorsement driven by injected instructions or priors). Original-context checks cannot detect it, and Figure 3 shows verifiers still endorse 22–30% of audited withdrawal views.

  • Conservative, safe-by-default design: Falling back to the initial answer on any failed or unresolved check, keeping the recognizer blind to passages, and freezing views before querying the verifier are principled choices that limit how attacks can steer the decision.

  • Strong empirical comparison: The paper presents a thorough evaluation, with six prior defenses plus two more in the appendix, three models, ten runs, paired CIs and a broad set of stress tests (redundant support, verifier-targeted instructions, candidate-string injection).

  • Solid ablations: The nested policies share candidates and frozen views. This isolates the response-test contribution cleanly.

Critical weaknesses
  • The improvement is marginal: Counterfactual verification adds only +1.57 accuracy and −3.43 ASR on average over the same pipeline without it. Most of the robustness comes from the keep-initial default and original-evidence checks, which are not new ideas.

  • The running cost is high: VeriRAG needs ~34 LLM calls and ~23K tokens per query, yet improves accuracy by only ~6.4 points over never using retrieval (Keep Initial, 64.6%). Its ASR is also higher than Keep Initial's (5.6% vs. 1.2%).

  • Under clean retrieval VeriRAG trails a simple sufficient-context gate (78.7% vs. 81.4%) and often InstructRAG. On MS-MARCO with Qwen it falls well below VanillaRAG (65.5% vs. 73.6%), which limits deployment where attacks are rare.

  • Dependence on parametric recognition: A candidate can only be tested if the model recognizes it from strings alone (two order-reversed calls must agree). This structurally forgoes the long-tail questions where retrieval helps most and caps the method's benefit.

  • Limited adaptive-attack analysis: The attacker is not optimized against the audits or the withdrawal test. A white-box attacker who writes poison that survives span deletion, or that biases the rubric derivation, is not evaluated beyond a few hand-designed stress cases.

Questions for authors
  1. What share of queries actually reaches the counterfactual stage, and how often does L>1 occur on NQ and MS-MARCO? If most questions are single-relation, how much does relation decomposition matter outside HotpotQA?

  2. On a matched compute budget (e.g., 34 calls of self-consistency or repeated original-context verification with majority voting), how does VeriRAG compare on all datasets, not only NQ?

  3. Several Qwen/MS-MARCO clean entries are identical (65.5 for RobustRAG, SeCon-RAG and VeriRAG). Is this a coincidence or a table error?

  4. How does accuracy vary with the initial answer's correctness? Specifically, what fraction of correct retrieved answers is rejected because recognition or auditing fails?

4 comments

We thank the reviewer for the careful reading and the precise questions. Numbers from the submission are cited by table. New results use Qwen3.5-9B with the queries and contexts of Table 1; percentages are ten-run means unless marked as a single run.

Our position on the central concern

  1. Most of the robustness comes from the keep-initial default and the original-evidence checks, as the reviewer says. The response tests are the new part: they cost 7% of the tokens and reduce mean ASR from 9.06% to 5.63%.
  2. The same compute spent differently does not reach the same result. Repeated verification and 34-call self-consistency cost more tokens and do worse on all three datasets.
  3. The method is designed for settings where contaminated retrieval is a real risk. Where attacks are rare, the cheaper sufficient-context gate is the better choice.

Q1: Queries reaching the counterfactual stage, frequency of L>1L>1, and decomposition outside HotpotQA

40–51% of queries reach the counterfactual stage. L>1L>1 holds for 35.0% of NQ and 23.1% of MS-MARCO queries, and on those queries decomposition is worth 5.5 and 5.7 points of ACC.

On NQ under the strong attack (Table 11), 403 of 1,000 queries reach counterfactual verification; it rejects 103 and 300 are adopted. The others stop at: unresolved rubric (18), recognizer abstention (222), original-support failure (200), competing-answer block (41), construction or validity-audit failure (116). For all three datasets (HotpotQA and MS-MARCO funnel entries are new):

Dataset (strong attack)Reach CF stageRejected by CFAdoptedL=2L=2L≥3L\ge 3L>1L>1
NQ40.3%25.6% of reaching30.0%2856535.0%
HotpotQA51.3%24.4% of reaching38.8%61417879.2%
MS-MARCO48.9%25.8% of reaching36.3%1864523.1%

To measure decomposition directly, we ran a new ablation that treats every question as a single relation, with one preservation/withdrawal pair on the whole requirement and all other gates unchanged:

DatasetSubsetFull ACC / ASRSingle-relation ACC / ASRDifference
NQL>1L>1 (350 queries)63.14 / 3.1457.63 / 6.31+5.51 / −3.17
MS-MARCOL>1L>1 (231 queries)67.97 / 6.7162.29 / 11.90+5.68 / −5.19
HotpotQAL>1L>1 (792 queries)64.77 / 7.2655.09 / 13.84+9.68 / −6.58

Over all 1,000 queries this is about 1.9 points of ACC on NQ, 1.3 on MS-MARCO, and 7.7 on HotpotQA.

Q2: Comparison at a matched compute budget on all datasets

On all three datasets Full is ahead of both matched-budget controls.

Repeated Original Verification (Appendix D.2) shares Full's pipeline and replaces the 2L2L P/W calls with 2L2L further checks of the original context, adopting only if all return Supported. It costs 23.15K tokens and 34.5 calls against 22.80K and 34.2 for Full (Table 26). New, all three datasets:

DatasetConditionFull ACC / ASRRepeated ACC / ASR
NQStrong62.04 / 2.6360.83 / 3.31
NQLight78.04 / 6.1876.31 / 7.02
HotpotQAStrong64.03 / 7.0461.81 / 9.72
HotpotQALight69.03 / 4.0267.62 / 3.81
MS-MARCOStrong64.28 / 6.9762.11 / 9.14
MS-MARCOLight68.21 / 10.1866.43 / 11.66

Full has higher ACC in 6 of 6 settings and lower ASR in 5 of 6; the exception is HotpotQA under the light attack.

Self-consistency (new): 34 sampled RAG answers per query with majority voting (temperature 0.7), under the strong attack. Full averages 22.80K tokens per query.

DatasetFull ACC / ASRSelf-consistency ACC / ASRSelf-consistency tokens per query
NQ62.04 / 2.636.38 / 88.7135.06K
HotpotQA64.03 / 7.049.76 / 86.2436.28K
MS-MARCO64.28 / 6.9714.11 / 74.5233.94K

Voting over answers drawn from the same contaminated context does not remove the attack.

Q3: Identical 65.5 entries for Qwen/MS-MARCO clean

It is a coincidence of one-decimal rounding. The unrounded ten-run means are 65.47% (RobustRAG), 65.51% (SeCon-RAG), and 65.54% (VeriRAG). The three methods are correct on different queries: in a representative run they agree on 813 of the 1,000. We will report two decimals for this cell.

We thank the reviewer again for the detailed questions. This part continues our response with the fourth question.

Q4: Accuracy by initial-answer correctness, and correct answers rejected by recognition or auditing

On NQ under the strong attack, Full keeps 85.5% of initially correct answers and corrects 28.6% of initially incorrect ones.

Of the 587 initially correct queries, 502 remain correct and 85 are overwritten. Of the 413 initially incorrect queries, 118 are corrected; 315 of them have a correct answer in the candidate pool (single run, new):

Outcome for the 315 available correct answersCountShare
Lost at recognition (52 abstentions, 25 wrong selections)7724.4%
Lost at a later gate (original evidence, competition, construction, response tests)12038.1%
Adopted11837.5%

Of the 222 recognizer abstentions in Table 11, only these 52 can cost a correction.

The 100-query substudy (Tables 14–15) shows what the gates buy in exchange:

20% PR + 10% PIA10% PR + 20% PIA
Initially correct (59) kept by Full5252
Initially correct (59) kept if the candidate is adopted unverified3028
Initially incorrect (41) corrected by Full1617
Recognized correct candidates rejected by later gates9 of 259 of 26
Harmful overwrites removed by the gates22 of 2924 of 31

We thank the reviewer again for the thorough assessment. This part addresses the five weaknesses in turn.

Responses to the weaknesses

W1 (the gain from counterfactual verification is small). We agree with the decomposition and will say so in the paper. The response tests add 1.57 points of ACC and remove 3.43 points of ASR on average, a 38% relative reduction in ASR, in the same direction in all nine model–dataset pairs (Table 9). The gain is not an effect of adopting fewer updates: random retention of the same number of updates is expected to give 63.4% and 62.5% ACC and 10.8% and 8.9% ASR on the two substudy attacks, against 68% and 69% ACC and 6% and 5% ASR observed (Table 16).

W2 (cost). The cost is high and we do not dispute it: 22.80K tokens and 34.2 calls per query, 3.65 times SeCon-RAG's tokens. The response tests account for 1.60K of those tokens, and the cost above Original-Evidence Verification is 5.10K. The table places every method on one scale (Qwen3.5-9B, strong attack, mean of the three datasets; costs from Table 26, ACC and ASR derived from Tables 1 and 6 or from the new results above):

MethodTokens (K)CallsStrong ACCStrong ASR
Keep Initial0.151.058.732.00
VanillaRAG1.041.04.8793.40
RC-RAG + Fallback (new)5.064.057.9713.45
SeCon-RAG6.258.056.208.80
Sufficient-context gate8.0516.861.9310.63
Original-Evidence Verification17.7029.959.2713.87
Repeated Original Verification (new)23.1534.561.587.39
Self-consistency, 34 calls (new)35.0934.010.0883.16
Full VeriRAG22.8034.263.455.55

More compute alone does not help: both controls at or above Full's budget are worse. The nearest cheaper alternative, the sufficient-context gate, is 1.5 points lower in ACC at a third of the tokens, with nearly double the ASR.

On the comparison with Keep Initial, the 6.4-point average hides how the gain depends on contamination: Full is ahead by 14.0 points under clean retrieval, 9.7 under the light attack, and 3.1 under the strong attack (derived from Table 1). Keep Initial's lower ASR comes from never using retrieval.

W3 (clean retrieval). This is correct: the sufficient-context gate reaches 81.44% clean ACC against 78.66% for Full, and on Qwen/MS-MARCO Full's 65.5% is below VanillaRAG's 73.6%; under attack Full is 2.16 points higher in ACC and 3.72 points lower in ASR (Table 2). The cheaper gate is the better choice where attacks are rare, and we will state that scope in the main text.

W4 (dependence on parametric recognition). The restriction is real: 77 of the 315 available correct candidates are lost at recognition (Q4). An optional Evidence Admission mode that drops the requirement reaches 70.31% ACC and 3.65% ASR on 384 questions about facts the model cannot know, against 12.76% and 1.56% for the submitted method. On the main NQ setting it reaches 61.37% ACC and 6.84% ASR, against 62.04% and 2.63% (full table in our response to Reviewer VfPm, Section 2).

W5 (adaptive attacks). We agree that the paper does not evaluate an attacker optimized against the defense. The rubric is derived from the question alone, which the attacker cannot change; the audits and the verifier read attacker-written text and remain exposed. Table 21 reports hand-designed, protocol-aware passages: redundant support and verifier-targeted instructions keep ASR at 2.3–2.7% while raising fallback from 70.0% to 80.5–85.3%, candidate-string injection raises ASR to 6.5%, and coherent false evidence with conditional responses raises it to 11.8% with ACC at 56.3%. We will call these protocol-aware stress tests and list optimized attacks on the audits as a limitation.

Dear Reviewer wTgs,

I hope this message finds you well. As the discussion period ends in less than three days, we wanted to check that our response has addressed your concerns. In brief:

  1. Q1. 40–51% of queries reach the counterfactual stage; decomposition is worth 5.5 and 5.7 points of ACC on multi-relation NQ and MS-MARCO questions.
  2. Q2. At a matched budget on all three datasets, VeriRAG beats repeated verification in ACC in 6 of 6 settings and 34-call self-consistency by more than 50 points.
  3. Q3. The identical 65.5 entries are one-decimal rounding (65.47, 65.51, 65.54).
  4. Q4. On NQ under the strong attack, VeriRAG keeps 85.5% of initially correct answers and corrects 28.6% of initially incorrect ones.
  5. W1–W5. The response tests cost 7% of the tokens and reduce mean ASR from 9.06% to 5.63%; we also placed all methods on one cost scale and clarified the scope for clean retrieval and adaptive attacks.

If anything remains unclear, please let us know and we will respond promptly. Thank you for your time and effort in reviewing our paper.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.