acceptodds
Under review as a conference paper at ICLR 2027

AF-RAG: Active Falsification for Local Top-K Contamination in Literature-Grounded Question Answering

10 views

Abstract

Automated laboratories and auto-research systems increasingly use retrieved literature to generate hypotheses, plan experiments, and analyze results. However, erroneous numerical values, conclusions, or comparisons in the retrieval context can propagate into downstream research decisions. We study a constrained local retrieval failure in which erroneous or adversarially promoted evidence occupies the initial Top-*K*, while claim-relevant corrective evidence remains retrievable at deeper ranks in the same corpus. We propose Active Falsification RAG (AF-RAG), which audits verifiable claims, generates targeted falsification queries, retrieves counter-evidence beyond Top-*K*, and adjudicates claim-level conflicts. We formalize this condition as an information cocoon with stealth poisoning, where locally coherent micro-tampering changes query-relevant facts. Under this local assumption, AF-RAG conditionally prioritizes claim-relevant deep evidence. On 2,641 arXiv papers and 1,200 stealth-poisoned samples, AF-RAG lowers Poisoned Adoption Rate (PAR) from 94.75% to 42.67%, raises Truth Recovery Rate (TRR) from 25.58% to 67.50%, increases Joint Success Rate (JSR) from 3.75% to 49.83%, and achieves 79.9% accuracy on unpoisoned queries. Supplementary diagnostics evaluate a verification extension combining provenance-aware adjudication and conflict-driven evidence acquisition, showing improved joint success when initial evidence is correct and external evidence is contaminated.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, information retrieval, question answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XHjEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the retrieval contamination problem in RAG-based auto-research systems. When manipulated evidence is retrieved in the initial Top-K results while the corrective sources are ranked lower in the same corpus, existing RAG systems usually return the wrong results. To address this, the paper proposes a new mechanism, AF-RAG, to identify verifiable claims and generate targeted falsification queries. Based on these, it actively retrieves additional evidence to correct contaminated answers. The experiments show that the proposed AF-RAG improves truth recovery and reduces adoption of poisoned claims under a controlled setting.

Critical strengths
  1. The problem setting is clear and the proposed method is intuitive. The paper focuses on a concrete failure mode in which the initial Top-k retrieval results contain manipulated claims while corrective evidence remains retrievable at deeper ranks in the corpus. Under this setting, actively searching for counter-evidence to verify potentially unreliable claims is a simple and natural idea.

  2. The experiments and analysis are quite comprehensive within the proposed setting. The paper includes component ablations, experiments under different evidence-placement conditions, cross-domain evaluations on Physics and PMC, multi-model experiments, and retrieval-budget-matched comparisons.

Critical weaknesses
  1. The robustness results depend on a controlled placement of poisoned evidence. The benchmark preserves entities, metrics, and experimental conditions that AF-RAG uses to retrieve the authentic source, while placing poisoned evidence in the initial context and corrective sources outside it. This matches the stated threat model, but largely limits generalization. With clean initial evidence and a poisoned deep pool, truth accuracy is 97% for Naive RAG but 17% for AF-RAG (Tab 23). The supplementary module partly addresses this, raising JSR from 18% to 53% in the correct/poisoned condition, but uses fixed first-round evidence and a clean follow-up index, and lowers JSR from 49.5% to 42% in the poisoned/correct condition (Tab 2). What if the location of poisoned evidence is unknown?

  2. The contribution of the falsification mechanism is unclear. Claim auditing, evidence retrieval, and correction closely resemble existing verification approaches such as FIRE [1] and RARR [2], but they are not compared under the same corpus and retrieval constraints. The existing retrieval-budget and claim-based controls are useful, but do not fully separate query construction, exclusion of initial documents, and evidence-prioritized synthesis. A simple baseline could exclude the initial document IDs, retrieve with the original query or generic rewrites under matched budgets, and answer only from the new evidence. Comparing full AF-RAG with an answer generated from the same external evidence alone would help clarify whether its gains require conflict adjudication.

  • [1] FIRE: Fact-checking with Iterative Retrieval and Verification (Xie et al., NAACL 2025)

  • [2] RARR: Researching and Revising What Language Models Say, Using Language Models (Gao et al., ACL 2023)

Questions for authors
  1. How would AF-RAG decide when to trust external evidence if the location of poisoned evidence is unknown?

  2. Could you compare adapted versions of FIRE and RARR under the same corpus and retrieval constraints?

  3. Could you compare AF-RAG with a baseline that excludes the initial document IDs, retrieves using the original query or generic rewrites under matched budgets, and answers only from the new evidence? Also, how does full AF-RAG compare with answering from exactly the same external evidence alone?

Reviewer_KpRAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the constrained local retrieval failure problem in automated research systems, where the initial retrieved top-k documents contain erroneous evidence and claim-relevant corrective evidence remains retrievable at deeper ranks in the same corpus. To address this problem, the authors propose AF-RAG, a new prompt-based framework that audits verifiable claims, generates targeted falsification queries, retrieves counter-evidence beyond top-k, and adjudicates claim-level conflicts. Experiments on a dataset with 1200 instances and 2641 arxiv papers show that the proposed method can achieve better performance.

Critical strengths

The studied problem is well-motivated. The authors provide sufficient references to support the fact that scholarly literature is not fully reliable, and erroneous evidence may affect the quality of the outputs from automated research systems. How to deal with conflicts in automated research systems is interesting.

  • The paper is well-written and easy to follow.
Critical weaknesses
  • The assumption of locally identifiable retrieval contamination may be uncommon in real-world automated research systems, where determining which retrieved papers contain erroneous evidence and which are reliable can be challenging.
  • The performance comparison may be unfair. For AF-RAG, the prompt explicitly instructs the model to prioritize external evidence over the initial context, thereby leveraging prior knowledge about the experimental setup. In contrast, the baselines treat the two sources equally, leaving the model without explicit guidance on which source is more reliable when conflicting evidence arises. Table 23 shows that when this prior assumption does not hold, the performance of AF-RAG drops substantially, while a simple Naive RAG method achieves high accuracy. This suggests that the performance gains of AF-RAG may stem largely from the prior information encoded in its prompt, rather than from the proposed method pipeline itself.
Questions for authors
  • How would the baselines perform under the same prompting strategy, explicitly instructing the model to prioritize subsequently retrieved documents over the initial context rather than treating all documents equally?
  • How would AF-RAG and the baselines perform in a more realistic and general RAG setting, where the top-k retrieved documents are directly used as context without any prior assumption about their reliability?
Reviewer_tx2POpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Active Falsification RAG (AF-RAG), a method to safeguard RAG against local retrieval errors in automated research. The authors claim that standard RAG using top-K retrieval optimizes for relevance without checking sources outside top-K sufficiently for counter-evidence. They propose a method that extracts verifiable claims with an LLM, generates verification queries, retrieves outside the top-K, and generates an answer with a prompt that distrusts the initial context. They evaluate their method on 1.2K synthetic instances from arXiv.

Critical strengths

Evaluating and improving literature agents is timely, and the direction of factual corruption is an interesting, safety-centered view of the problem.

Critical weaknesses
  1. It is unclear whether and to what extent, the described failure mode of “retrieving poisoned claims, while having non-poisoned ones in its sampling tail” practically occurs. The constructed setup appears artificial, as there is no evidence that real literature agents fail this way.

  2. Although the usage case is interesting, methodologically, the paper makes few contributions in my opinion. The paper reads as a prompting pipeline over BM25, with existing components (per-claim search queries, discarding unreliable retrieval to search further) in place to support the pipeline.

  3. The method seems very strongly tailored to the proposed (synthetic) setup where the initial pool is poisoned, and the retrieval pool is right. It seems by design it distrusts the original context and prefers the additionally retrieved evidence. One could design an adversarial setup where this method fails.

  4. Finally, there are some evaluation issues:

  • All results come from single-seed runs, this is not good practice in general (although the margins seem rather large). Some models are not reported (e.g., what the “vanilla LM” is, making it unclear how to reproduce.

  • The main evaluation figure (Fig 3) suggests that “vanilla LM” performs significantly better than any RAG baselines (e.g., in terms of JSR; 28% vs. ~10%), despite being “closed-book”. While this partly follows from the adversarial setup, it raises the question of whether the chosen baselines are adequate, since a reasonable baseline should at least not be worse than using no retrieval. For instance, none of the baselines receives AF-RAG’s synthesis prompt, which seems to be an important driver of the method’s gains.

Questions for authors
  • Can the authors show robustness of the method in case the initial evidence is good, but the additionally retrieved evidence is poisoned?

  • How do the baselines perform when they receive the same synthesis prompt as AF-RAG?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.