acceptodds
Under review as a conference paper at ICLR 2027

When Aggregate AUROC Ranks the Shortcut: Trivial-Axis Headroom in Vision-Language Hallucination Detection

Abstract

Hallucination detectors for vision-language generators draw on four signal families (cross-modal alignment, self-consistency, an external classifier, token confidence) and are compared by aggregate AUROC. Proposing no detector, we introduce two diagnostics of what that comparison measures: headroom over a declared trivial axis—a label-free shortcut that never sees the image—and an exact decomposition of AUROC into the pairs that axis orders by itself and the rest. On chest X-ray report generation, our label rule marks asserted-but-unsupported claims hallucinated and denied-but-uncontradicted ones grounded, so reading silence as absence couples the label to claim polarity. A trivial axis reading only polarity reaches 0.84–0.87 AUROC on IU X-ray and MIMIC-CXR, leaving the best family a headroom of only 0.013–0.026 (one interval covering zero), whereas on COCO the best family clears a caption mention-count axis by 0.171–0.177. Because each axis lower-bounds what label-free scores can reach, these headrooms are upper bounds. Implementation choices move each family enough to reverse their ordering, yet at their best the three externally grounded ones converge on IU X-ray. That convergence is not agreement: on chest X-rays, polarity alone orders 70–74% of hallucinated–grounded pairs; cross-modal alignment gets all of them right yet is weakest or tied-weakest among claims denying a finding; and on MIMIC-CXR a classifier trained on neither corpus ranks last in aggregate (0.474, below chance) but first among those claims, and ordering claims by polarity and then by the classifier beats every family's best aggregate. Surface scores also depart from chance on three benchmarks labelled by others (label-free on POPE and M-HalDetect, label-informed on AMBER), even sign-flipped on POPE splits drawing negatives from frequent or co-occurring objects. Where labels are partly decidable from surface form, we recommend reporting headroom over a declared trivial axis, and stratified comparisons, alongside any ranking.

Primary area:
datasets and benchmarks
Keywords:
hallucination detection, vision-language models, benchmark validity, shortcut learning, evaluation methodology, radiology report generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fNWTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether aggregate AUROC reliably reflects hallucination detection performance in vision-language models, or whether simple shortcuts in the benchmark labels dominate it. Rather than proposing a new detector, it introduces three diagnostics: implementation intervals, headroom over a trivial axis, and an exact decomposition of AUROC. It applies them to four signal families (cross-modal alignment, self-consistency, external classifiers, and token confidence) on chest X-ray and general-domain datasets.

Critical strengths

An important question, approached critically. The paper does not add another detector. Instead, it examines whether current evaluation practice supports the conclusions drawn from it, which I think is a valuable contribution.

A clear central finding with a clear explanation. On the chest X-ray datasets, claim polarity alone reaches an AUROC of 0.84–0.87, even though it never sees the image. That leaves only a few hundredths of AUROC for the best detector configurations to improve on. The pair decomposition explains why: polarity alone correctly orders most hallucinated–grounded pairs.

Useful implementation-sensitivity analysis Reasonable configuration choices substantially change both the performance and the ranking of detector families. The stratified analysis is especially informative. It shows that detectors with similar aggregate AUROC can behave quite differently once the shortcut is controlled for.

Breadth of evidence. Three features strengthen the study: (1) multiple generators and datasets; (2) classifier checkpoints trained on neither corpus; (3)additional analyses on existing benchmarks (POPE, AMBER, M-HalDetect).

Critical weaknesses

1. Validity of the medical labels underlying the main result. This is my main concern.

  • The medical labels are obtained by comparing RadGraph parses of generated claims against the reference reports, under a rule that treats unmentioned findings as absent. This rule creates a strong association between claim polarity and the hallucination label.
  • Identifying this association is part of the paper's contribution. However, it raises a question: to what extent does the observed shortcut reflect a general limitation of hallucination detection evaluation, rather than the specific labelling procedure adopted here?
  • The authors' own sensitivity analysis makes this concern more pressing. In Appendix J.9, replacing the reference parses with reader consensus on a MIMIC-CXR sample changes a noticeable share of the claim labels and substantially lowers the AUROC of the polarity-only baseline. Since this baseline is the quantity at the centre of the paper's argument, its sensitivity to the reference interpretation is important.
  • The labels have not been validated against radiologist judgements, and the paper itself documents incorrect labels caused by missing laterality and missing pathology mappings (Appendix H).
  • The size of the reported headroom (the gap between the best detector configuration and the trivial polarity baseline) depends directly on label validity. The results convincingly demonstrate a shortcut in the evaluated labelling setup, but more evidence is needed before the quantitative conclusions can be extended to clinically meaningful hallucination detection.
  • Suggestion: an independently validated subset of claims, ideally assessed by radiologists, with the headroom analysis repeated on the corrected labels, would considerably strengthen the paper.

2. Selection of detector configurations. The admissible configurations are fixed without consulting the labels, but the best one is then chosen by evaluation AUROC.

  • This introduces an optimistic selection bias.
  • Families have configuration spaces of different sizes, so the bias may also differ between families and affect comparisons between them.
  • Suggestion: selecting configurations on a separate validation split would address this.

3. Interpretation of headroom. A small gap between a detector's AUROC and a trivial baseline does not necessarily mean that the detector contributes little visual information. The authors themselves show that two detectors can reach similar aggregate performance while relying on different evidence and making different errors. The conclusion that small headroom rules out ranking by aggregate AUROC should therefore be stated with more qualification.

4. Few positive examples within the stratum. The absent-polarity analysis rests on relatively few hallucinated claims, especially on IU X-ray. Confidence intervals are reported, but broader evidence would help establish whether the observed ranking reversals are stable.

5. Comparability across configurations. Some classifier checkpoints can only score a subset of the claims, so parts of the implementation intervals compare performance on different evaluation populations. Suggestion: Reporting comparisons restricted to a common set of claims would make this clearer.

Questions for authors

Questions and suggestions for the authors

  1. Could you validate a subset of the specialist labels independently, ideally with radiologists, and repeat the headroom analysis on the corrected labels?
  2. How do the headroom estimates change when the best configuration for each family is selected on a separate validation split rather than on the evaluation data?
  3. How do the implementation intervals change when every configuration is evaluated on a common set of claims?
  4. Could you qualify the claim that small headroom rules out ranking by aggregate AUROC, in line with your own stratified results?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer fNWT (1/2)

Thank you for a careful review whose main suggestion — a radiologist-validated subset with the headroom analysis repeated on corrected labels — we carried out. We answer your five weaknesses, four questions, and the ethics question in turn.

W1 / Q1 — validity of the medical labels (your main concern). Two board-certified radiologists (~25 and ~20 years of chest radiology) independently rated a stratified sample of 160 IU / LLaVA-Rad claims under an analysis plan fixed before any label was collected (images first, then the reference report; blinded to rule labels, strata and scores; 5 quality-control sentences each, 10/10 passed; final-judgement kappa 0.94). Results, weighted back to the population:

  • The rule is right for most claims (weighted accuracy 0.957) but its errors concentrate in the asserted claims: 33% of sampled asserted-hallucinated claims are judged correct by both radiologists, and 44% of the small asserted-grounded cell is judged hallucinated; precision/recall of the rule for hallucination are 0.699/0.634.
  • Repeating the headroom analysis on the radiologists' consensus labels of the same claims: the trivial axis falls 0.868 → 0.738, the best family clears it by +0.129 [+0.007, +0.273] (vs +0.014 under the rule on the same claims), and the family ranking inverts — cross-modal alignment (best under the rule) falls to 0.738 while the discriminative classifier leads at 0.867, exactly the behaviour the stratified decomposition predicted (Fig. 2b).
  • Two full-population checks agree: excluding only silent assertions from the rule (no LLM, all three cells) gives headroom +0.129–0.139, and the re-run parse audit (below) moves the sample headroom to +0.035. The paper's headline numbers are therefore now stated as conditional on the silence-as-absence rule, and a new subsection reports this validation.

W2 / Q2 — configurations selected on the evaluation labels. Two selection-independent checks: (a) selecting each family's best configuration on one IU generator and evaluating on the other leaves every headroom unchanged (+0.025 and +0.013; the two strongest families select identical configurations on both); (b) the paper's transfer table: IU-selected configurations land within 0.004 AUROC of MIMIC-selected ones. The selection bias direction also only inflates the families' best, i.e. inflates headroom — conservative for the small-headroom finding.

W3 / Q4 — interpretation of headroom. We agree and have revised the wording: small headroom means aggregate AUROC alone cannot resolve a ranking under that rule; it does not mean the detectors carry no visual information (the stratified analysis, and now the radiologist labels, show they do — and differ). The conclusion no longer says "rules out".

W4 — few positives in the absent stratum. Across the three chest X-ray cells (215 hallucinated denials in total), the mean of the per-cell discriminative-minus-cross-modal differences within the stratum is +0.222 [+0.184, +0.265], positive in every cell (+0.241, +0.249, +0.177) with every interval excluding zero. The radiologist-label reversal above shows the same ordering at the aggregate level.

W5 / Q3 — comparability across configurations. Restricting the discriminative family to the checkpoint that scores every claim ('all'; n = 2,222 / 2,389 / 2,884, identical populations), its intervals are [0.833, 0.871], [0.807, 0.863], [0.596, 0.621]. The qualitative conclusions stand: 13 of 15 chest X-ray family pairs still overlap (the two that separate are classifier vs token confidence on the IU cells), and no family is implementation-inert. The revised appendix reports both versions.

AuthorsOpenReview ↗

Reply to Reviewer fNWT (2/2): ethics

Ethics — MIMIC-CXR data use.

Thank you for asking this directly; it deserves a direct answer. How the model was accessed: the two readers in Appendix J.9 and the Figure 9 candidate check were instances of Claude reached through a third-party API relay service that forwards requests to Anthropic's API — not an enterprise deployment covered by a zero-data-retention agreement. Whether the conditions were met: use of online LLM services on credentialed data has become widespread in the field — PhysioNet issued guidance on it in April 2023 and tightened it in September 2025 — but prevalence is not compliance. We subsequently contacted the relay operator, who confirmed to us that the relay platform and its upstream Claude service do not retain submitted inputs or outputs, do not use submitted content for model training, and do not subject it to human review. These statements correspond to the three conditions in PhysioNet's 2025 guidance.

Exact scope. The audit itself sent the findings sections of 404 de-identified reports with their study identifiers, in batched runs; no images were part of that step. Because your question deserves a complete answer, we also audited every local session log post hoc: between July and September 2026 the same route additionally received findings-text fragments in roughly 25 development-time prompts, roughly 100 further study identifiers (mostly directory listings echoed while debugging), and 8 in-session openings of draft figures containing report excerpts. The full itemisation is in the revised ethics statement. The reports are de-identified (HIPAA Safe Harbor; the identifiers are dataset-internal pseudonyms), and no MIMIC-CXR record-level file was included in any released material — the supplementary material contains aggregates only.

Remediation, following your comment. (1) The parse audit and the Figure 9 check were re-run entirely offline with two different open-weight models (Qwen2.5-72B, Llama-3.3-70B) on local hardware under the archived reader instruction; the revised Appendix J.9 and Figure 9 rest only on these labels (reader–reader kappa 0.61, 23/485 labels change, polarity 0.855 → 0.759, headroom +0.035, Figure 9 pool 81/244), and no Claude-produced MIMIC-CXR label is used anywhere in the revision or will be shared. (2) We have asked both the relay operator and Anthropic to delete the affected requests and not to use their content for training, supplying the identifiers of every affected request (5,233 across four sessions, July–September 2026). (3) Our pipeline now processes credentialed data with local open-weight models only. The ethics and AI-use statements disclose the incident and this policy. The IU X-ray check (Appendix J.2) used public, de-identified Open-i data and is unaffected.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer,

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your thoughtful comments and understand that you have a busy schedule.

As the discussion deadline is approaching, we would like to gently follow up on our rebuttal and the additional analyses we have provided. We would be very grateful if you could take a moment to review our responses and let us know whether any concerns remain that we could further clarify.

Thank you again for your valuable time and consideration.

Best regards, The Authors

Reviewer_JkfpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Hallucination detectors for vision-language models are usually ranked by aggregate AUROC. This paper asks what that number measures. It adds two diagnostics: (1) "headroom", meaning how much a detector family gains over a simple score that never looks at the image (a "trivial axis"), and (2) an exact split of AUROC by pair type (assert/deny × hallucinated/grounded). On chest X-ray report claims, labels are tied to polarity because the rule treats report silence as absence. A polarity-only score reaches 0.84–0.87 AUROC, so the best family has only 0.013–0.026 headroom (Table 1). On COCO the headroom is 0.171–0.177. The paper also shows that each family's AUROC moves a lot with implementation choices (intervals overlap for 24 of 27 family pairs).

Critical strengths
  • The pair decomposition is simple and exact. I checked that the weights and win rates in Table 6 reproduce the reported AUROCs. The polarity AUROC also follows from the counts in Section 4.4 (117/143 and 40/2,079).
  • Reporting is unusually open. All configurations are listed (Appendix K), limitations are stated, and silent defaults are found (Qwen2-VL's top_k=1 makes self-consistency constant; CheXagent defaults to beam search).
  • The bootstrap is clustered by study or image, and paired for differences.
  • The aggregate-versus-stratum reversal (Figure 2b, Table 2) is a useful example of how an aggregate number can hide behaviour.
  • The message is practical and the recommendations are easy to follow.
Critical weaknesses
  • Headroom mixes polarity and detection. The trivial axis and cross-modal alignment both use the label rule's own parse, so alignment scores a perfect 1.000 on polarity by design. A family that is good within strata but weak on polarity gets a large negative headroom (the classifier on MIMIC-CXR: −0.22). Only AUROC is reported, with no PR-AUC, partial AUC or fixed false-positive-rate view, even though only 7–18% of claims are hallucinated.

  • Implementation intervals depend on what is admitted. The classifier's low end on IU (0.514, 0.411) comes from checkpoints trained on a different corpus. The "reference" settings are reconstructed, not quoted, so the 97% and 91% compression depends on that choice. The best configurations are picked on the evaluation labels, and the authors admit some configuration sets were fixed after earlier results were seen. This conflicts with the claim that every set was fixed before its AUROC was computed.

  • The stratified evidence is thin. The absent stratum has only 40, 65 and 110 hallucinated claims, and the classifier's lead over self-consistency is not significant on any IU cell. Families are compared at settings chosen for aggregate AUROC. The MIMIC-CXR reversal depends on one score construction that the main text does not explain.

  • Narrow scope and dated models. There are five cells and one specialist domain, and the specialist-versus-COCO comparison is confounded (claim unit, label source, prevalence, sample count). The models are older (LLaVA-1.5, Qwen2-VL, LLaVA-Rad, CheXagent, OWL-ViT). Please add current models, the specialist detectors cited in Section 2 (ReXTrust, RadFlag), and an LLM-judge or trained verifier. The checks on other benchmarks show that shortcuts exist, but not how much headroom there is.

  • Modest novelty and presentation. Splitting AUC by pair type and using image-free baselines are established ideas (CRoPS, M3ID). The main text is dense and leans heavily on supplementary material.

Questions for authors
  1. With image-verified labels (even on a small sample), does the polarity axis and the headroom survive?
  2. What are the results if cross-modal alignment does not receive the parsed polarity?
  3. Do the findings hold with configurations chosen on a held-out split, and with PR-AUC, partial AUC or a fixed false-positive rate?
  4. How is a finding probability turned into a claim score for denials versus assertions?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer Jkfp (1/2)

Thank you for the most exacting of our reviews — including re-deriving Table 6 and the polarity counts. The checks you asked for are done; several confirmed your suspicion in a way that sharpens the paper's point rather than weakening it.

W1 — "headroom mixes polarity and detection"; only AUROC reported.

  • Any claim-level detector must read the claim's stated polarity: "no effusion" and "effusion" are different claims, and a score that ignores polarity cannot distinguish a claim from its negation. The decomposition exists to isolate that term exactly rather than leave it mixed in.
  • Your suspicion that cross-modal alignment's rank is polarity-driven is now confirmed against expert labels: two board-certified radiologists re-labelled a stratified IU sample (global response; kappa 0.94, analysis plan fixed before labelling), and on their consensus labels cross-modal falls 0.879 → 0.738 while the discriminative classifier leads at 0.867; headroom over the axis opens from +0.014 to +0.129 [+0.007, +0.273]. The native-rule aggregate rewarded polarity encoding — which is what the paper claims such a comparison measures.
  • Operating-point metrics (prevalence 7–18%), now reported: on IU / LLaVA-Rad the AUROC-best family is beaten by the trivial axis in average precision (0.481 vs 0.628), partial AUC at FPR ≤ 0.1 (−0.101) and TPR at 5% FPR (−0.188). Choosing the best family per metric, chest X-ray headroom is at most +0.087 (AP +0.039 / +0.087 / +0.077; pAUC +0.002 / +0.008 / +0.026; TPR@5%FPR +0.003 / +0.007 / +0.082), against +0.36–0.50 on COCO. At clinical operating points the picture is worse for the detectors, not better.
  • The classifier's −0.22 headroom on MIMIC-CXR is the phenomenon, not an artifact: under expert labels that family is the best one; the native-rule aggregate punishes it.

W2 — what the intervals admit; reconstructed references; the "fixed before" contradiction. You are right about the contradiction, and we have corrected §4.1 to the precise statement: each configuration set was fixed before its own cell's sweep was scored, but after sweeps on other cells had been seen; the one label-adjacent case (COCO's K=5) was already disclosed in the appendix. The direction of any such bias can only inflate a family's best value, i.e. inflate headroom. On checkpoints: provenance is a real configuration coordinate practitioners face (the appendix traces the IU low end to training-corpus label definitions), but we now also report the discriminative interval restricted to the full-coverage 'all' checkpoint — [0.833, 0.871], [0.807, 0.863], [0.596, 0.621] on identical claim populations (n = 2,222 / 2,389 / 2,884) — and 13 of 15 chest X-ray family pairs still overlap. Compression is presentational: the paper's claims rest on intervals, headroom and the decomposition, none of which involves the reconstructed reference configurations; the revision labels them "reconstructed common-use".

W3 — stratified evidence is thin; the MIMIC reversal rests on an unexplained construction. Across the three cells (215 hallucinated denials in total), the mean of the per-cell discriminative-minus-cross-modal differences within the absent stratum is +0.222 [+0.184, +0.265] under a joint study-level bootstrap, positive in every cell (+0.241, +0.249, +0.177), every interval excluding zero. The radiologist labels reproduce the same ordering at the aggregate level. The score construction is now stated in §4.1 (see Q4). The clean-checkpoint caveat on IU remains in the text.

AuthorsOpenReview ↗

Reply to Reviewer Jkfp (2/2)

W4 — scope and dated models; add ReXTrust, RadFlag, an LLM judge. RadFlag is an instance of the self-consistency family we audit (temperature resampling plus an entailment judge); ReXTrust is a supervised probe on hidden states — a trained detector whose configuration space includes its training labels, outside the four untrained signal families audited here. The diagnostics themselves are detector-agnostic: any scored detector evaluated on these corpora faces the same axis, and its headroom is computable the same way. We agree the survey could cover more and newer detectors and now state this scope limit explicitly; the generators are the standard public specialist checkpoints, and our object is the evaluation protocol rather than the generator frontier. On the third-party benchmarks we agree and already say so: those measure exploitability, not headroom, which needs per-benchmark generations and all four families.

W5 — novelty. CRoPS and M3ID are decoding-time mitigation methods: they contrast the output distribution with vision-ablated distributions to change generation; neither audits a detector comparison nor decomposes an evaluation metric over label-rule-induced pair types. The nearest ancestors are hypothesis-only baselines in NLI (cited in §2) and covariate-adjusted ROC analysis in biostatistics (Janes & Pepe; now cited). What the discussion period adds is, to our knowledge, new to this literature: a demonstration on clinically validated labels that the aggregate ranking under the benchmark's native rule selects the wrong detector, exactly as the trivial-axis diagnostics predicted.

Q1. Partly, and informatively: under image-verified labels the axis survives at a reduced level (0.738 — about two thirds of the sampled rule-hallucinated assertions are judged hallucinated by the radiologists as well), the headroom opens to +0.129 [+0.007, +0.273], and the family ranking inverts. The axis is real but rule-inflated.

Q2. A polarity-blind cross-modal score cannot distinguish "effusion" from "no effusion", so it is not an admissible claim-level detector. The meaningful polarity-free quantity is performance where polarity is constant: the within-stratum AUROCs we report (cross-modal 0.55–0.61 vs classifier 0.79–0.84 in the absent stratum).

Q3. Selection on a held-out generator leaves every headroom unchanged (+0.025 and +0.013; the two strongest families select identical configurations on both IU generators), and the operating-point metrics are in W1 above; both are in the revised appendix.

Q4. For each finding a claim names, with checkpoint probability p(present): the contribution is 1 − p if the claim asserts the finding and p if it denies it; contributions aggregate over the claim's findings by the configuration's max / mean / min. The cross-modal contrastive posterior uses the same orientation. Now stated in §4.1.

We believe these additions address the soundness concerns directly, several by confirming them against expert labels, and we would gladly run further checks within the discussion window.

Reviewer_JkfpOpenReview ↗

Official Comment by Reviewer Jkfp

Thank you for the detailed responses. Most of my concerns have been adequately addressed, so I am increasing my score to 3.

AuthorsOpenReview ↗

Thank You for Your Feedback!

Thank you very much for your thoughtful feedback and for taking the time to reconsider our responses. We are pleased that our clarifications and additional analyses have addressed most of your concerns. We sincerely appreciate your constructive comments and the time you have devoted to reviewing our work.

Reviewer_jha9OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether aggregate AUROC is a reliable way to compare hallucination detectors for vision-language models. Instead of proposing another detector, the authors investigate how much of the reported performance can be explained by simple shortcuts that do not require looking at the image.

Critical strengths
  1. I think the paper raises an important issue that is often overlooked when evaluating hallucination detectors. A high AUROC does not necessarily mean that a method is actually detecting visual hallucinations. The finding that claim polarity alone reaches around 0.84–0.87 AUROC on chest X-ray benchmarks, while the best detector adds only about 0.01–0.03, is quite striking. I particularly like the pairwise AUROC analysis, which helps explain why methods with similar aggregate scores may actually be detecting very different things.

  2. The experimental analysis is thorough. The authors compare different implementations within each detector family, evaluate multiple generators and datasets, and examine how the results change across polarity groups. I also appreciate the additional experiments with independent classifier checkpoints and the attempts to validate the labeling procedure. These analyses make the conclusions more convincing than simply reporting a few benchmark scores.

Critical weaknesses
  1. My main concern is that the strongest finding depends heavily on how hallucinations are labeled in the medical datasets. The paper treats a finding that is not mentioned in the reference report as absent, which naturally creates a strong relationship between claim polarity and the hallucination label. I understand that identifying this shortcut is part of the paper's contribution, but it also raises questions about how representative these labels are of actual clinical hallucinations. The authors acknowledge issues with laterality and incomplete pathology mapping, and Appendix J.9 shows that replacing the automatic labels with the independent readers' consensus changes 8% of the evaluated claims. More importantly, the polarity-only AUROC drops from 0.855 to 0.718 on that sample. This is a substantial difference and suggests that the estimated headroom can depend quite strongly on the chosen labeling rule. Since this is central to the paper, I would like to see a clearer discussion of how much the main conclusions would change under alternative, clinically reasonable label definitions.

  2. I also have some concerns about how the implementation intervals are compared. The paper defines each interval on an evaluation dataset, but different medical classifier checkpoints do not always cover the same set of claims. For example, Appendix K reports 2,222 claims for the "all" checkpoint on IU/LLaVA-Rad, but only 1,546 for NIH. This means that part of the observed performance range may come from evaluating different subsets rather than only changing the implementation. In addition, the best and worst configurations are selected using their performance on the evaluation data, which makes the resulting intervals somewhat optimistic. The authors acknowledge this selection issue and provide some matched-subset analyses in the appendix, which is helpful. However, I think the central claim about implementation-dependent rankings would be stronger if the full comparison were also reported on a common set of claims.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Reply to Reviewer jha9

Thank you — your two concerns are exactly the ones we spent the discussion period on, and both are now answered with numbers.

W1 — how much do the conclusions change under alternative, clinically reasonable label definitions? Three ways, forming one gradient:

  • Alternative rules, all claims, no LLM. Re-labelling every chest X-ray cell: excluding only asserted claims about findings the reference never mentions (the rule's silence-as-absence side) moves headroom from +0.013–0.026 to +0.129 / +0.136 / +0.139 (every 95% CI excluding zero — the same order as COCO's +0.17); broadening only the reference's Lung Opacity definition (the J.9 readers' definition, which you quote) leaves it small, +0.028–0.051. The lever is the silent-assertion side, which carries 48–69% of the rule's hallucinations.
  • The J.9 audit you cite, re-run with two different open-weight reader families (replacing the original same-model readers; also for data-use reasons — see our reply to Reviewer fNWT): 4.7% of sample labels change, polarity falls 0.855 → 0.759, headroom on the sample +0.035.
  • Radiologist validation. Two board-certified radiologists (~25 and ~20 years), analysis plan fixed before labelling, kappa 0.94: the rule's weighted accuracy is 0.957, but precision/recall for hallucination are 0.699/0.634, with errors concentrated in asserted claims (33% of the asserted-hallucinated sample overturned). On their consensus labels the axis falls 0.868 → 0.738, headroom opens to +0.129 [+0.007, +0.273], and the family that aggregate AUROC ranked first under the native rule (cross-modal alignment) drops to last while the discriminative classifier leads — the reversal the stratified decomposition (Fig. 2b) predicted.

The revision conditions every headline number on the declared rule and adds this as a subsection: the small headroom is a property of silence-as-absence labelling; the diagnostics are what detects this, with or without expert labels.

W2 — intervals compare different claim subsets, and best/worst are selected on the evaluation data. Restricting the discriminative family to the checkpoint that scores every claim ('all'), its intervals on identical populations (n = 2,222 / 2,389 / 2,884) are [0.833, 0.871], [0.807, 0.863], [0.596, 0.621]; 13 of 15 chest X-ray family pairs still overlap, so the configuration-dependence conclusion stands, and the matched-subset analyses you note remain in the appendix. On selection: choosing each family's best configuration on one IU generator and evaluating on the other leaves every headroom unchanged (+0.025 and +0.013), and the paper's transfer table shows IU-selected configurations landing within 0.004 of MIMIC-selected ones.

Both additions are in the revised PDF (changes in blue).

AuthorsOpenReview ↗

A Gentle Reminder Regarding Our Rebuttal

Dear Reviewer,

We sincerely thank you for the time and effort you have dedicated to evaluating our manuscript. We greatly appreciate your constructive feedback and fully understand the demands of your busy schedule.

With the discussion period drawing to a close, we wanted to kindly bring our rebuttal and supplementary analyses to your attention. We would be grateful if you could review our responses when your schedule permits and let us know if any questions or concerns remain. We would be happy to provide further clarification.

Thank you again for your valuable feedback and consideration.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response: the label-rule concern, tested three ways (radiologist validation added)

We thank all three reviewers. The reviews converge on one question: is the small headroom a property of hallucination detection on these benchmarks, or of our label rule, which reads a reference report's silence as absence? During the discussion period we tested this three ways. No previously reported number changed; every addition either re-analyses stored per-claim scores under alternative labels or brings new, independent labels.

  1. Alternative label rules, no LLM, all 7,495 chest X-ray claims. We re-derived every reference parse (a verification gate reproduces all 4,611 stored IU labels exactly) and re-labelled every cell under clinically reasonable variants: R1 treats silence as evidence of nothing (claims about unmentioned findings are excluded); R1a excludes only asserted unmentioned findings; R2 only broadens the reference's Lung Opacity definition.
  2. Independent re-audit of the MIMIC-CXR reference parses. The Appendix J.9 audit was re-run from scratch with two different open-weight model families as readers (Qwen2.5-72B, Llama-3.3-70B; run fully offline; it also resolves the data-use question, see our reply to Reviewer fNWT). This removes the same-model caveat the paper had flagged (the original two readers were instances of one model).
  3. Radiologist validation. Two board-certified radiologists (~25 and ~20 years of chest radiology) independently rated a stratified sample of 160 IU X-ray / LLaVA-Rad claims (all 26 asserted-grounded, all 40 denied-hallucinated, 50/117 asserted-hallucinated, 44/2,039 denied-grounded; inverse-fraction weights) in an offline interface: images first, then the reference report; blinded to rule labels, strata and detector scores; 5 planted quality-control sentences each (10/10 judged correctly); ~85 minutes per reader; final-judgement kappa 0.94. The analysis plan was fixed before any label was collected and is included in the revised appendix.

Headroom of the best family over the trivial axis, by labelling:

LabelsIU / LLaVA-RadIU / CheXagentMIMIC / LLaVA-Rad
Native rule (paper)+0.013+0.025+0.026
Parse re-audited by independent readers (sample)——+0.035
R1a: no silence-as-absence for assertions+0.136 [+0.072, +0.199]+0.129 [+0.053, +0.211]+0.139 [+0.076, +0.201]
Radiologist labels (sample, weighted)+0.129 [+0.007, +0.273]——
COCO, for scale (paper)+0.171 (LLaVA-1.5)+0.177 (Qwen2-VL)—

The answer, in one paragraph. The reviewers' concern is correct, and quantifying it demonstrates the paper's point. The polarity axis is real but rule-inflated: the radiologists judge about two thirds of the sampled rule-hallucinated assertions incorrect as well (so the axis stays at 0.738 under their labels, well above chance), yet the rule's errors concentrate exactly where the shortcut lives (33% of sampled asserted-hallucinated and 44% of asserted-grounded claims are overturned; weighted accuracy 0.957, but precision/recall for hallucination only 0.699/0.634). Under the native rule, aggregate AUROC ranks cross-modal alignment first; under the radiologists' labels of the same claims, cross-modal falls to 0.738 while the discriminative classifier leads at 0.867 — the reversal our stratified decomposition predicted without access to expert labels (Fig. 2b). On these benchmarks' native labels, aggregate AUROC ranked the shortcut; the diagnostics the paper proposes detected it, and the corrected labels land where the diagnostics pointed.

Revisions (revised PDF and supplementary material uploaded alongside these replies; changes marked in blue, main text ends on page 10):

  • Every headline headroom statement is now conditional on the declared label rule; the conclusion's "rules out ranking them by aggregate AUROC" is replaced by: small headroom means aggregate AUROC alone cannot resolve a ranking under that rule — and under corrected labels the ranking it induced here reverses.
  • New subsection "Is the axis an artifact of the rule?" (rule variants + radiologist study), using the discussion period's extra page.
  • Appendix J.9 rewritten on the two local open-weight readers; Figure 9 re-selected from the re-audited candidate pool; ethics and AI-use statements updated (reply to fNWT).
  • Added: operating-point metrics (PR-AUC, partial AUC, TPR at fixed FPR), implementation intervals restricted to the full-coverage checkpoint, a pooled absent-stratum test, configuration selection on a held-out generator, and the exact per-claim scoring construction (§4.1).
  • §4.1's "every configuration is fixed before its AUROC is computed" corrected to the precise per-cell statement (Appendix "Order of decisions").

Point-by-point replies follow under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.