- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors design SelfHeal-RAG for biomedical question answering, aiming to improve answer correction through recursive retrieval and an evidence-gated controller. The framework assesses whether an initial answer is supported by retrieved evidence. The results show improvements over the single-pass baseline. While the research question is relevant, the doubtful motivation, unclear method description and limited evaluation make the paper's contribution difficult to assess.
Critical strengths
The authors demonstrate a high level of transparency by providing almost all research materials, even including SHA-256 hashes of the Python source files.
Critical weaknesses
The motivation is unclear to me: iterative RAG methods already exist, but the paper neither clearly explains what is missing from them nor compares against them experimentally. The discussion of hallucination evaluation and the emphasis on code hashes disconnect from the main argument. The authors also do not explain why the Jaccard threshold is 0.90 or why in Eq. (3).
Some works may be related to the authors' work.
- RAG-Critic: Leveraging Automated Critic-Guided Agentic Workflow for Retrieval Augmented Generation
- Chain-of-Retrieval Augmented Generation
- Self-Critique Guided Iterative Reasoning for Multi-hop Question Answering
- FIRE: Fact-checking with Iterative Retrieval and Verification
Questions for authors
- Experimental comparisons with recent methods would help me better understand the value.
- Could the authors explain why the Jaccard threshold of 0.90 and in Eq. (3)? How sensitive are the results to these choices?
- Could the authors test the method on more diverse datasets? Since both current datasets are biomedical, it is hard to tell whether the findings generalize to other domains.
- Could the authors clarify how the discussion of hallucination evaluation and the inclusion of code hashes relate to the paper's main argument? Their relevance is currently unclear to me.