acceptodds
Withdrawn from ICLR 2027

Clinical Hallucination Detection with Coverage-Gated Ontology Checks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) that write clinical text can state false facts in a fluent, convincing form. A common safeguard is a second LLM, an auditor, that checks each claim by calling external knowledge tools. In practice this is fragile: auditors misuse the tools or misread their replies, and some models, including the medical model we test, cannot call tools at all. We propose coverage-gated ontol- ogy checking, which takes the knowledge lookup out of the auditor’s hands. Code first decides whether a clinical ontology can speak to a claim. If it can, the ontol- ogy checks the claim, and the auditor reads the results and gives a single verdict, without ever calling a tool. We evaluate the approach on fourteen auditors from seven model families, from 3B to 128B parameters. On a clinical hallucination benchmark it improves every auditor, raising the Matthews correlation coefficient by 0.09 to 0.30 over the same auditor without checks. The gains are largest for weaker models, and with checks a 7B auditor matches 70B and 120B auditors without them. The approach also works for models that cannot call tools, and it outperforms tool calling for most models that can. Because coverage is decided before the auditor runs, it shows in advance where the ontology can help.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
hallucination detection, tool-augmented language models, knowledge grounding, ontologies, clinical NLP, selective prediction, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_C4kuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies coverage-gated ontology checks for clinical hallucination detection. Deterministic checks run before an LLM auditor produces its verdict, with experiments across 14 auditors on synthetic and external benchmarks.

Critical strengths
  1. Separating check execution from the auditor is a practical design, particularly for models without tool-calling support.

  2. The broad model comparison, prompt-matched controls, and knowledge-deletion experiments provide useful evidence about both improvements and failure cases.

Critical weaknesses
  1. My main concern is what these findings add to the AI and healthcare community. Most gains come from synthetic claims sharing the checker’s knowledge sources. In the only external comparison with a prompt-matched baseline, MCC drops from 0.486 to 0.397. The main conclusions should reflect this limited evidence of benefit on independent clinical claims.

  2. Coverage does not establish sufficient evidence or improved judgments. After knowledge deletion, 64 false claims receive "clean", and accuracy for the three weakest auditors falls from 0.948 to 0.708. Supporting passages can also trigger checks unrelated to the target claim. These results weaken the claim of knowing in advance where checks help. The 7B comparison also does not establish equivalence with larger models.

Questions for authors

Please refer to the weaknesses above.

Reviewer_3v9HOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes a strategy based on coverage-gated ontology checks to safeguard hallucination in LLM auditing. Evaluating the strategy in a clinical hallucination benchmark, the study observed that the proposed strategy improves the performance of auditors, with largest gains when the model is the weakest.

Critical strengths

The main contribution of this paper is using a topology based strategy to improve the safeguard auditing of LLMs. The observations are based on multiple auditor and model settings on a clinical hallucination benchmark.

Critical weaknesses

The innovations of the method are not clear. The technical contributions are mainly the empirical and observatory results on one clinical benchmark dataset. It is unclear if the observations are applicable to other datasets, domains and tasks. There is little theoretical or mechanical investigation of the observed results.

Questions for authors

From your observations, what is the cause of hallucination in the clinical benchmark data and what's the best strategy to mitigate such hallucinations?

Reviewer_kHirOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether deterministic, knowledge-based checks can improve an LLM’s detection of clinical hallucinations while avoiding reliance on model-directed tool calling. The proposed pipeline extracts clinical assertions, checks for coverage by the ontology, runs applicable checks of the assertions, and then passes its findings to an auditor model. The authors look at checking drug–condition contraindications, laboratory value plausibility, and clinical relationships. The ontology used is OWL and the HermiT reasoner is used to process it.

Critical strengths

Controls are adequate and show the technique works.

Critical weaknesses

The primary hallucinated claims were generated by perturbing facts from the sources consulted by the checker. This close in-domain test heavily favors the available vocabulary, relation inventory, and factual content of the source material and offers little insight into how the tool would work in a real environment.

The methodological advance over prior verification systems, particularly knowledge-graph based ones, isn't adequately established.

Questions for authors

Can the method demonstrate a positive effect on independently sourced claims?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.