acceptodds
Under review as a conference paper at ICLR 2027

Fair Protocol Reveals Collapse of Medical Hallucination Detectors-and Two Detectors That Survive It

Abstract

State-of-the-art hallucination detectors for medical VQA fail to reliably measure whether answers are grounded in the image. We show that reported progress is largely an artifact of benchmark leakage: detectors exploit surface statistics like answer length, formatting, and sampling dispersion that correlate with hallucination only within the benchmarks used for tuning. We introduce a fair evaluation protocol that strips away these shortcuts via length- and type-matched negatives, image-level splits, and language-leakage controls. Under this protocol, all prior methods collapse to near-chance performance. VASE, SE, RadFlag, and a frozen LMM all score an AUROC of 0.47–0.57 across three benchmarks. A plain trainable MLP reaches 0.70–0.97, so model capacity is not the bottleneck; an input ablation shows that the residual signal comes from question-answer statistics, not image grounding. We then ask what survives. QCE-Net, an interpretable detector that pools image patches with both question and answer as attention queries, matches or exceeds the MLP baseline, recovering the grounding signal that frozen encoders can provide. LLaVA-LoRA, which fine-tunes the medical LMM with LoRA to emit per-sample grounding judgments, goes further: it reaches 0.885 AUROC on VQA-RAD, the first detector under a leakage-free protocol to clear the clinical bar of 0.8 that no frozen-CLIP method reaches.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Hallucination Detection, Large Multimodal Models (LMMs), Evaluation Protocol, Vision-Language Grounding, LoRA Fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fBUoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the medical visual hallucination detection task and finds that reported progress in medical visual hallucination detection is largely caused by benchmark shortcuts (e.g., answer length, format, and image-level leakage). It proposes a "fair protocol" with length- and type-matched negative answers, image-level splits, and language-only controls for evaluation. The paper reports that existing detectors fall to approximately chance AUROC under this protocol. The authors proposes QCE-Net, a trainable head over frozen CLIP features, and LLaVA-LoRA, a low-rank adaptation of LLaVA-Med for grounded versus ungrounded answer classification. The main claimed outcome is the collapse of prior detectors and recovery of signal by QCE-Net and LLaVA-LoRA, outperforming the baselines.

Critical strengths
  • The paper aims to address an important issue: hallucination detectors may exploit answer-level shortcuts rather than image evidence.
  • QCE-Net is simple and interpretable. and the experiments show that the proposed model outperforms baselines.
Critical weaknesses
  1. The innovation is limited. The proposed fairness-oriented evaluation protocol and QCE-Net architecture both overlap substantially with existing work on controlled or answerability-aware VQA evaluation and question-guided or attention-based visual reasoning [1-3].

  2. Positive samples are dataset ground-truth answers, while negatives are answers borrowed from other images. This measures artificial candidate-answer classification rather than natural LMM hallucinations, and borrowed answers may be synonymous, multiply valid, or clinically correct; the CLIP-based “possibly valid” check is circular and lacks clinician adjudication.

  3. It would be better if the authors conducted cross-validation and reported standard deviation to demonstrate that the findings are stable.

[1]. Vardi, Ben, Oron Nir, and Ariel Shamir. "Clip-up: Clip-based unanswerable problem detection for visual question answering." 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2026. [2] Zhang, Yudong, et al. "Dhcp: Detecting hallucinations by cross-modal attention pattern in large vision-language models." Proceedings of the 33rd ACM International Conference on Multimedia. 2025. [3] He, Xingwei, et al. "TUBench: Benchmark Large Vision-Language Models on Trustworthiness With Unanswerable Questions." IEEE Transactions on Dependable and Secure Computing (2026).

Questions for authors
  1. Can the model accurately detect hallucinations if negative samples also have similar anomalies? for example, to cluster the patients first and then conduct the hallucination detection.
Reviewer_zDdPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies an important problem: detecting hallucinations in medical VQA, which the authors define as answers not supported by image evidence. It argues that reported progress mostly reflects benchmark leakage, and proposes a “fair” protocol in which negatives are length- and type-matched answers swapped in from other images (opposite polarity for yes/no questions), with image-level splits.

Under this protocol, existing detectors (VASE, SE, RadFlag, a zero-shot LLaVA-Med judge) score near chance (0.49–0.57 AUROC). A trainable MLP on frozen CLIP features reaches 0.67–0.97, and its input ablation suggests the signal comes mainly from the question–answer pairing.

The authors propose two detectors:

QCE-Net pools CLIP patch features using the question and the answer as attention queries. It beats the MLP on VQA-RAD (0.702 vs 0.670) and ties elsewhere. LLaVA-LoRA fine-tunes LLaVA-Med as a grounded/ungrounded classifier and reaches 0.885–0.969 AUROC.

Results cover VQA-RAD, Path-VQA and OmniMed-VQA-mini, with SLAKE in a supplementary comparison.

Critical strengths

Two stengths:

  1. Careful, full-scale evaluation practice. The paper reimplements baselines at full scale under two backbones and reports on complete test sets with paired bootstrap tests. It also shows that results on small subsets can be misleading (VASE: 0.669 on 100 pairs vs 0.494 on 928). Image-level splits and length/type-matched negatives are sensible hygiene. This matters because it is a useful methodological caution for the medical VQA community, independent of whether the leakage interpretation holds.

  2. An informative empirical observation about the benchmarks. The input ablations show that question–answer pairs alone achieve high AUROC on VQA-RAD and Path-VQA, while the image contributes little for frozen-CLIP detectors. This matters because it is potentially the paper’s most valuable finding: it questions what these benchmarks measure. However, its interpretation, leakage versus items answerable without the image, is not resolved, which limits how much weight it can carry.

Critical weaknesses
  1. The task is redefined so that it no longer matches the hallucination-detection problem it claims to address.

The protocol turns hallucination detection into binary verification of curated triples. Positives are ground-truth answers. Negatives are answers swapped in from other images, or the flipped answer for yes/no questions. For yes/no items (54% of VQA-RAD) this is simply VQA correctness, and for open-ended items it is answer–image matching.

Constructing negatives by swapping images or answers is also not new for medical VQA hallucination (e.g., the image-swap scenario in Wu et al., 2024; MediConfusion, ICLR 2025).

The prior detectors were designed for a different setting: judging a model’s own generations through its uncertainty. VASE, SE and RadFlag compute dispersion over samples conditioned on (image, question). That input is identical for both members of a matched pair, so the scores are tied and AUROC is close to 0.5 by construction (all VASE results lie within 0.012 of 0.5).

To me, this is a very important point. The headline claim, that published detectors collapse because of benchmark leakage, is not supported: the near-chance results reflect a mismatch between method and task. Moreover, neither proposed detector is evaluated on real LMM-generated hallucinations. Also, the published detection mechanisms were not designed for a binary classification task like the one on the curated dataset.

  1. The text-reliance findings are attributed to leakage, while more plausible explanations are not examined.

The MLP and QCE-Net use frozen general-domain CLIP at 224px. This is a weak visual representation for medical images, and CLIP patch tokens are not trained for text alignment. Their reliance on text is therefore expected.

Many VQA items can also be answered from the question and answer alone, through domain knowledge or questioner bias. That is not leakage, but it means those items do not test grounding. The paper’s own text-only MLP (0.679 on VQA-RAD, 0.843 on Path-VQA) shows the protocol is not “leakage-free”. The weak lexical control in Table 6 (0.44–0.50) does not establish that it is.

QCE-Net’s ablations show the same text dominance: removing the image costs 0.004, and the evidence channels add 0.032 on one dataset only. Its attention weights are close to uniform (about 1/256 ± 10%, Fig. 4).

LLaVA-LoRA’s gain plausibly comes from a capable medical MLLM performing VQA-style verification rather than from a hallucination-detection contribution. Any strong MLLM may do comparably. No image-ablated or text-only LoRA run is reported, and no VQA-style zero-shot baseline.

Neither proposed detector is shown to be grounded. The central methodological conclusions therefore do not follow from the evidence.

Questions for authors

Questions:

  1. Applicability of VASE, SE and RadFlag. These scores appear to depend only on (image, question), which is identical across a matched pair. How does the candidate answer enter the score? Is the near-chance AUROC more than an artefact of tied scores?

  2. Grounding of LLaVA-LoRA. Please report LLaVA-LoRA with the image blanked or swapped at test time, plus a text-only LoRA on the same backbone. Also include a VQA-style zero-shot baseline, in which the MLLM answers the question and its answer is compared with the candidate. Without these, the gain cannot be attributed to grounding or to protocol training.

  3. The “leakage-free” claim. A text-only [a, q] MLP reaches 0.679 on VQA-RAD and 0.843 on Path-VQA. How is this consistent with the protocol being leakage-free? Please report results on an image-necessary subset or on counterfactual image pairs.

Clarification

How were the CLIP patch tokens projected into the joint text–image space? How do you justify ⟨pᵢ, q⟩ as text–patch alignment, given that the attention weights in Fig. 4 are close to uniform?

Suggestion (future work)

Evaluate on real hallucinations generated by medical LMMs.

Reviewer_cHZ7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses statistical shortcuts in medical hallucination detection by proposing a fairer evaluation protocol with length- and type-matched negatives, image-level splits, and language-leakage checks. It introduces two detectors, QCE-Net (CLIP-based) and LLaVA-LoRA (fine-tuned LLaVA-Med), evaluated on VQA-RAD, Path-VQA, and OmniMed-VQA-mini, with SLAKE as an additional check. Several prior detectors collapse under the stricter protocol, while the proposed methods show improvements over baselines, with LLaVA-LoRA reaching 0.885 AUROC on VQA-RAD.

Critical strengths
  1. The paper proposes useful controls for evaluating medical hallucination detectors. Length- and type-matched negatives, image-level splits, and lexical-only checks help test whether detectors exploit shortcuts. The ablation in Table 3 also shows that QA-only inputs retain most of the MLP’s performance, suggesting that high AUROC alone does not demonstrate image grounding.

  2. LLaVA-LoRA shows clear improvements from task-specific fine-tuning. It reaches 0.885 AUROC on VQA-RAD and 0.894 on Path-VQA, outperforming both the frozen LLaVA-Med judge and the trainable MLP. Comparing the frozen and fine-tuned versions of the same backbone helps show the benefit of training on this task.

Critical weaknesses
  1. Several experiment design limitations that make the results do not clearly show that the proposed detectors use image grounding:
  • The no-image ablation is only reported for the MLP, not QCE-Net or LLaVA-LoRA.
  • The lexical-only detector also reaches 0.694 AUROC on OmniMed, while QCE-Net’s evidence channels add almost no improvement on Path-VQA.
  • The proposed detectors may still rely mainly on QA statistics or how negative answers are constructed. No-image and image-shuffling experiments for both detectors would help clarify this.
  1. The scoring procedure for the collapsed baselines is unclear. Positive and negative pairs share the same image and question, with only the answer changed. However, Appendix A.15 does not clearly explain how VASE, SE, and RadFlag use this candidate answer. If their scores depend only on the image and question, both pairs would receive the same score, giving chance AUROC by construction. The authors should clarify this before attributing the collapse to benchmark leakage. Comparing 100 samples with the full test set also does not isolate the effect of removing leakage.
Questions for authors
  1. How do VASE, SE, and RadFlag use the candidate answer when computing their scores? Since matched pairs share the same image and question, please clarify whether changing only the answer changes the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.