- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies coverage-gated ontology checks for clinical hallucination detection. Deterministic checks run before an LLM auditor produces its verdict, with experiments across 14 auditors on synthetic and external benchmarks.
Critical strengths
-
Separating check execution from the auditor is a practical design, particularly for models without tool-calling support.
-
The broad model comparison, prompt-matched controls, and knowledge-deletion experiments provide useful evidence about both improvements and failure cases.
Critical weaknesses
-
My main concern is what these findings add to the AI and healthcare community. Most gains come from synthetic claims sharing the checker’s knowledge sources. In the only external comparison with a prompt-matched baseline, MCC drops from 0.486 to 0.397. The main conclusions should reflect this limited evidence of benefit on independent clinical claims.
-
Coverage does not establish sufficient evidence or improved judgments. After knowledge deletion, 64 false claims receive "clean", and accuracy for the three weakest auditors falls from 0.948 to 0.708. Supporting passages can also trigger checks unrelated to the target claim. These results weaken the claim of knowing in advance where checks help. The 7B comparison also does not establish equivalence with larger models.
Questions for authors
Please refer to the weaknesses above.