- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VeriRAG verifies a retrieved candidate before using it to replace an initial answer. It accepts an update only when the verifier responds appropriately to controlled changes in supporting evidence.
Critical strengths
- The method tests whether a verifier notices missing support. This addresses a failure that checking the original context alone can miss.
- The evidence edits are audited and frozen before response testing. This gives the intervention procedure a clear separation from its outcomes.
- The Keep Initial baseline helps assess whether the method makes useful corrections. The repeated-verification control also examines the benefit of changing the evidence.
Critical weaknesses
- RC-RAG [1] already uses counterfactual prompting to decide whether a RAG answer should be retained. Please compare with an adaptation that returns the initial answer when RC-RAG rejects an update.
- The recognizer must identify a factual basis from its existing knowledge before evidence verification begins. How does this requirement affect questions about genuinely unfamiliar facts? Please evaluate newly introduced knowledge for which the retrieved evidence is sufficient.
- Appendix A requires a nonempty preservation edit near the removed support. This may make valid updates depend on how much expendable text surrounds a fact. Could the authors compare concise evidence with versions containing irrelevant padding while keeping the factual content unchanged?
- Can an attacker block a valid correction by adding one apparently supported competing answer? Please evaluate this case when the initial answer is wrong. Target-answer ASR does not capture an attack that succeeds by preventing correction.
[1] Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework.
Questions for authors
See Weaknesses.
4 comments
We thank the reviewer again for the constructive requests. This part continues our response with the third and fourth points.
3. Dependence on expendable text around the fact
The reviewer's reading of Appendix A is correct. With the facts unchanged, concise evidence yields 31.0 points less correction coverage than the same evidence with irrelevant padding.
The preservation edit must be nonempty and matched to the withdrawal edit in source, length, and position. If a passage contains little more than the supporting statement, no such edit exists and the update falls back to .
We took 126 initially incorrect questions whose correct candidate is recognized and supported under clean retrieval, and rewrote the supporting passages in three forms with identical facts (single run): the supporting sentences only; the same with 40–60 tokens of unrelated text around them; and the same with the statement repeated once in different wording.
| Evidence form | Queries with all pairs audit-accepted | Failures due to the nonempty/matching constraints | Correct adoptions | Incorrect adoptions |
|---|---|---|---|---|
| Concise | 78 | 48 | 74 | 0 |
| Irrelevant padding | 119 | 7 | 113 | 0 |
| Redundant padding | 108 | 18 | 103 | 0 |
Concise evidence yields 74 corrections of 126 (58.7%) and irrelevant padding 113 (89.7%). Of the 41 queries that padding recovers, 39 end in a correct adoption. The method needs passages with some text around the supporting statement, and we will state this in a Limitations paragraph and in Appendix A.
4. Blocking a valid correction with a competing answer
Yes. One completely supported competing passage blocks 93.0% of otherwise valid corrections. Every blocked query returns the initial answer, and none returns the competitor.
The adoption rule vetoes an update whenever the context gives complete support to another answer value (Eq. 5); in the submission this gate blocks 41 of 1,000 NQ queries under the strong attack (Table 11). We took 264 initially incorrect questions, of which VeriRAG corrects 129 under clean retrieval, and added one well-formed passage asserting a competing answer for the same entity (single run). The Correction Blocking Rate (CBR) is the share of those 129 corrections that no longer occur.
| Added passage | Corrections under clean retrieval | Still corrected | Blocked (CBR) | Blocked at competing-answer gate | Blocked at another gate | Competitor returned |
|---|---|---|---|---|---|---|
| None | 129 | 129 | 0 | 0 | 0 | 0 |
| Competitor with partial support | 129 | 100 | 29 (22.5%) | 6 | 23 | 0 |
| Competitor with complete support | 129 | 9 | 120 (93.0%) | 120 | 0 | 0 |
With partial support, 23 of the 29 blocks occur at the original-evidence requirement, because the passage makes the candidate's support opposed. This is an availability attack: the attacker can hold the system at its retrieval-free answer but does not get the competitor adopted. We will report CBR alongside ASR and list correction blocking in the Limitations paragraph.
Dear Reviewer VfPm,
I hope this message finds you well. As the discussion period ends in less than three days, we wanted to check that our response has addressed your concerns. In brief:
- RC-RAG. With the requested fallback, VeriRAG is 3.8–6.9 points higher in ACC and 5.1–10.2 points lower in ASR under the strong attack on all three datasets.
- Unfamiliar facts. The submitted method reaches 12.76% ACC on 384 new-fact questions; an optional Evidence Admission mode reaches 70.31% at 3.65% ASR.
- Surrounding text. With the facts unchanged, concise evidence yields 74 corrections of 126 and padded evidence 113.
- Correction blocking. One competing passage blocks 93.0% of valid corrections; every blocked query returns the initial answer, never the competitor.
If anything remains unclear, please let us know and we will respond promptly. Thank you for your time and effort in reviewing our paper.
Best regards, The Authors
Sorry for the late reply, and thank you for the additional experiments. I remain concerned that the original method can block useful updates even without attacks, limiting RAG’s ability to use new knowledge. I hope the authors can further address this limitation in the revised paper.
For now, I lean toward maintaining my original score.
We thank the reviewer for four concrete requests. Each led to a new experiment. New results use Qwen3.5-9B with the queries and contexts of Table 1; percentages are ten-run means unless marked as a single run.
1. Comparison with RC-RAG adapted to return the initial answer
We agree this comparison was missing. Under the strong attack, VeriRAG is 3.8–6.9 points higher in ACC and 5.1–10.2 points lower in ASR than RC-RAG with the requested fallback.
We implemented RC-RAG + Fallback: RC-RAG's counterfactual prompting decides whether the retrieved answer is kept, and when it rejects, the system returns the same retrieval-free answer a0 that VeriRAG falls back to. Our adaptation uses three counterfactual prompts (the published two, plus one that assumes the answer just given is wrong), keeps the answer only when all three reproduce it, and omits the published fusion call. Both methods share initial answers, contexts, and attack targets. They judge different answers, RC-RAG its own generated answer and VeriRAG a recognized candidate, so this is a comparison of two complete update procedures.
RC-RAG treats a stable regenerated answer as reliable, and an injected instruction or a repeated poisoned claim can also produce a stable answer. VeriRAG changes the evidence and requires the verifier to withdraw its endorsement once the support is removed. We will add this comparison to the main text.
2. Genuinely unfamiliar facts with sufficient retrieved evidence
The requirement does restrict such updates. On 384 questions about newly introduced facts, the submitted method reaches 12.76% ACC under the strong attack and makes 6 corrections.
We built 384 questions about fictional entities, each described in one supporting passage, with attack passages from the strong-attack procedure of Table 1 (single run). The initial answer is correct on 44 questions (11.46%). A correct candidate with sufficient evidence is available for 371 questions under clean retrieval and 318 under the strong attack.
We also evaluated an optional Evidence Admission mode. It admits any well-formed candidate with a registered source, without parametric recognition, and tests admitted candidates in passage-rank order until one passes every check. The mode changes both admission and candidate selection, and the results do not separate the two.
We will add it as an optional mode for unfamiliar facts and state the recognition requirement as a limitation.