acceptodds
Under review as a conference paper at ICLR 2027

Evidence-Gated Correction: A Controlled Study of Recursive Retrieval in Scientific QA

Abstract

Retrieval-augmented generation for scientific question answering can produce answers whose support is insufficiently grounded in retrieved evidence. We formulate recursive correction as a selective control problem: after an initial answer, a controller decides whether to preserve it, retrieve additional evidence, generate a candidate correction, or commit that candidate. SelfHeal-RAG implements this separation using hybrid dense-sparse retrieval, NLI-based evidence scoring, recursive PubMed retrieval, and a state-aware controller that adjudicates candidate corrections against an affirmative proposition. On a frozen, question-only BioASQ held-out set (n=150; 110 Yes / 40 No), evidence-gated recursion improves paired accuracy from 42.67% to 51.33% (+8.67 percentage points; exact McNemar p=0.000244), with all 13 changed decisions beneficial and none harmful. A retrieval-off condition exactly reproduces the baseline, while V5 significantly differs from retrieval-off, consistent with recursive retrieval contributing to the observed improvement. Two controls qualify this result: unconditional replacement reaches 54.0%, so this experiment does not establish an accuracy advantage for selective gating over unconditional replacement; and the benchmark has a 73.33% majority-class baseline. In a complementary diagnostic with answer-bearing context supplied directly, baseline accuracy is 74.5% and recursive correction adds 1.5 percentage points (p=0.25). Together, these results provide evidence that recursive evidence acquisition can improve scientific question answering relative to one-pass correction, while leaving the relative value of selective commitment versus simpler unconditional correction unresolved.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Retrieval-Augmented Generation, Recursive Retrieval, Evidence Verification, Scientific Question Answering, Biomedical Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_KLKHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors design SelfHeal-RAG for biomedical question answering, aiming to improve answer correction through recursive retrieval and an evidence-gated controller. The framework assesses whether an initial answer is supported by retrieved evidence. The results show improvements over the single-pass baseline. While the research question is relevant, the doubtful motivation, unclear method description and limited evaluation make the paper's contribution difficult to assess.

Critical strengths

The authors demonstrate a high level of transparency by providing almost all research materials, even including SHA-256 hashes of the Python source files.

Critical weaknesses

The motivation is unclear to me: iterative RAG methods already exist, but the paper neither clearly explains what is missing from them nor compares against them experimentally. The discussion of hallucination evaluation and the emphasis on code hashes disconnect from the main argument. The authors also do not explain why the Jaccard threshold is 0.90 or why τ=0.85\tau=0.85 in Eq. (3).

Some works may be related to the authors' work.

  1. RAG-Critic: Leveraging Automated Critic-Guided Agentic Workflow for Retrieval Augmented Generation
  2. Chain-of-Retrieval Augmented Generation
  3. Self-Critique Guided Iterative Reasoning for Multi-hop Question Answering
  4. FIRE: Fact-checking with Iterative Retrieval and Verification
Questions for authors
  1. Experimental comparisons with recent methods would help me better understand the value.
  2. Could the authors explain why the Jaccard threshold of 0.90 and τ=0.85\tau = 0.85 in Eq. (3)? How sensitive are the results to these choices?
  3. Could the authors test the method on more diverse datasets? Since both current datasets are biomedical, it is hard to tell whether the findings generalize to other domains.
  4. Could the authors clarify how the discussion of hallucination evaluation and the inclusion of code hashes relate to the paper's main argument? Their relevance is currently unclear to me.
Reviewer_WAcvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SelfHeal-RAG, a recursive correction pipeline for biomedical yes/no QA. After an initial RAG answer, an NLI-based evidence score triggers up to three rounds of PICO-targeted PubMed retrieval. A rule-based controller then decides whether to preserve the answer, retrieve again, or generate a candidate and commit or reject it. On 150 held-out BioASQ yes/no questions, the gated controller improves accuracy from 42.67% to 51.33%, an unconditional-replacement variant reaches 54.00%, and the difference from the gated controller is not significant (p=0.125). All conditions remain below the 73.33% always-Yes majority baseline. The authors frame the gating question as open.

Critical strengths

S1 The paper isolates design choices with paired ablations on frozen, shared initial retrieval, uses exact McNemar tests, and reports class-imbalance-robust metrics alongside a majority-class reference.

Critical weaknesses

W1 The performance regime is uninformative, since all tested conditions remain well below (54% < 73%) the naive reference of always predicting "Yes". While balanced accuracy and MCC exceed the trivial predictor, a balanced accuracy of 0.58 on a binary task remains weak, and raw accuracy below the majority class is hard to justify for a QA system. Even if the authors present better results during the rebuttal, these remain difficult to verify without full explanation of experiments or seeing the source code.

W2 A single benchmark, a single generator and 150 questions are insufficient to support general claims about recursive retrieval. In addition, the paper's central comparison (gated vs. unconditional replacement) rests on four discordant pairs (p=0.125). The results that must be provided during the rebuttal to resolve this issue are likely too extensive to be properly validated during the rebuttal phase.

Based on these two substantial weaknesses, I currently see very few merits in the paper and see it as unlikely that the rebuttal will change my rating of 1 (Clear rejection).

Questions for authors

No questions that arise within the weaknesses mentioned above are repeated.

Q1 Why did you retrieve from the PubMedQA abstracts for BioASQ questions instead of PubMed directly? The gold document is in the top-5 for only 7.3% of questions. Does the gain come from recursion or simply from getting access to PubMed later in the loop? How does a single-pass baseline that retrieves from PubMed perform, and how does the generator do without any retrieval?

Q2 Unconditional replacement makes zero harmful changes across the entire benchmark (100% precision). If blindly overwriting every answer never hurts accuracy, is the test set adequate (lack of distractors, too small) to test this behavior?

Q3 The earlier final-test run in Appendix D went from 36.0% to 70.0% with no Maybe answers, while the held-out run goes from 42.67% to 51.33% with 31 Maybe answers and the binary-commit path never triggered. What changed between the two runs? It’s hard to tell which one reflects the system you actually intend to propose.

Reviewer_ebvrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies recursive correction for scientific yes/no question answering under a RAG setup. The proposed SelfHeal-RAG combines hybrid retrieval over PubMedQA abstracts, NLI-based evidence scoring, recursive PubMed retrieval, and a rule-based controller that decides whether to preserve an answer, retrieve more evidence, generate a candidate correction, or commit/reject that candidate. The main evaluation uses 150 frozen BioASQ yes/no questions and compares a one-pass baseline, retrieval-off control, ungated recursion, evidence-gated recursion, and unconditional replacement. The paper reports that recursive correction improves accuracy from 42.67% to 51.33% relative to the baseline, while unconditional replacement reaches 54.00%.

Critical strengths

S1. I appreciate the authors' honesty in highlighting their central negative result rather than obscuring it. The manuscript clearly acknowledges that selective evidence gating does not always outperform unconditional replacement. Furthermore, the statistical analysis is notably thorough.

S2. The conceptual formulation separating the decision to seek correction from the decision to commit a generated candidate addresses a genuine conceptual gap in corrective RAG. Most existing frameworks conflate generation with acceptance. Framing the loop as an explicit state-machine controller with a discrete action space provides a clear structural paradigm for analyzing where iterative refinement pipelines succeed or fail.

Critical weaknesses

W1. My biggest concern is that the central experimental comparison is confounded. The authors claim this comparison isolates candidate commitment by holding pre-generation opportunity and retrieval identical. However, Table 5 reveals that v5_gated executes 241 PubMed API calls, whereas unconditional_replacement executes only 7 calls. This extreme disparity demonstrates that the two conditions did not follow matched retrieval or generation trajectories. Consequently, the empirical finding cannot be reliably attributed to the adjudication gate itself.

W2. The baseline retrieves strictly from a local PubMedQA corpus (PQA-A/U), in which the authors note gold BioASQ PMIDs appear only 7.33% of the time. In contrast, the recursive stages query the live PubMed database via E-utilities. Hence, the observed performance gain (42.67% →\rightarrow 51.33%) is heavily confounded by the expansion of the underlying corpus rather than the recursive controller mechanism. Furthermore, absolute accuracy remains below the trivial always-Yes majority baseline, and comparisons against competitive, standard iterative RAG baselines (e.g., Adaptive-RAG, i-MedRAG) are absent.

W3. The paper's representation quality falls below the standard. The main text lacks an architectural schematic or pipeline diagram; the only visual overview (Figure 4) is deferred to Appendix H due to space constraints. Describing the SelfHeal-RAG purely through dense textual exposition makes comprehending and verifying the operational mechanics of SelfHeal-RAG unnecessarily difficult. In addition, tables in Appendix A and B do not have captions.

Questions for authors
  1. In the supplementary repository materials, I observed the presence of research_audit.md and research_audit.json. These files suggest that the research project was executed via an automated, agentic research loop. Notably, several critical vulnerabilities identified in my review appear to have been flagged in these audit logs, yet remained unrectified in the submission. Could the authors clarify the exact division of labor between human researchers and autonomous AI agents in ideating, executing, and drafting this submission?
  2. How does the evidence score SE(y,E)S_E(y, E) behave when N=0N=0 (e.g., when generated responses contain no valid subject-relation-object triples)? Furthermore, because the NLI scorer is a cross-encoder trained on general-domain SNLI/MultiNLI rather than scientific biomedical corpora, did you calibrate whether SE≥0.85S_E \ge 0.85 correlates reliably with human-adjudicated biomedical support, or does the max-pooling operator over sentences introduce an optimistic bias as the evidence pool expands?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.