acceptodds
Withdrawn from ICLR 2027

LOCI: Mitigating Hallucinations in Radiology Reports with Phrase-Triggered Counterfactual Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hallucinations remain a critical reliability issue for medical large vision-language models, particularly when generated text asserts clinical findings unsupported by the image. In chest X-ray report generation, mitigating such errors is challenging because clinically meaningful findings are often expressed as multi-token phrases whose interpretation depends on assertion scope, such as positive, negated, or uncertain. Existing training-free decoding methods typically operate at the token level or rely on global visual perturbations, making it difficult to selectively intervene on a specific clinical assertion. We introduce LOCI, a training-free, scope-aware decoding method that activates only when a candidate completes a clinical finding phrase in positive scope. For the matched finding, LOCI constructs a text-derived direction at the intervention layer, subtracts its projection from image-token hidden states in a temporary counterfactual branch, and reranks only plausible matched candidates according to the induced relative change in log probability. The edited branch is discarded after the current decoding step. Across three frozen 7B LVLMs and four chest X-ray datasets, LOCI obtains the best RadGraph F1 point estimate in all 12 backbone-dataset settings and the best CheXbert 5-category F1 result in 11 of 12 settings, although statistically resolved gains are concentrated in a subset of comparisons. On PadChest-GR, RadFact evaluation further shows higher logical F1 and fewer false positive findings across all three backbones. Mechanistic controls further show that the counterfactual signal is strongest for matched same-layer directions under the true-image condition. These results support phrase-triggered counterfactual sensitivity as a targeted inference-time signal for mitigating positive-finding hallucinations in frozen LVLMs. Code is available at https://anonymous.4open.science/r/LOCI-837B/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
radiology report generation, medical vision-language models, hallucination mitigation, training-free decoding, counterfactual decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3UUJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LOCI, a training-free decoding method for chest X-ray report generation with medical VLMs. At each decoding step it looks at the top-K candidate tokens and checks whether any of them would complete a clinical finding phrase (from a hand-written lexicon of 26 findings) as a positive statement rather than a negated or uncertain one. If so, LOCI runs a check on that candidate: it takes the model's own text representation of the finding (a direction vector obtained by feeding just the phrase, without an image, and reading the hidden state at one chosen layer), removes that direction from the image-token hidden states at the same layer, and recomputes the next-token distribution. If the candidate's probability drops a lot, the model was relying on image evidence for that finding, and the candidate is scored up; if it barely changes or goes up, the candidate is being driven by language prior, and it is scored down (Eqs. 1–2, with weight α). Only the matched candidates are re-scored; everything else, and every step where nothing triggers, is plain greedy decoding. The edited representation is used only for this check and then thrown away.

Critical strengths
  1. The motivation is clear and sensible. Clinical statements are often sentence-level rather than token-level, and LOCI explicitly targets this scenario with proper triggering mechanism.

  2. Compehensive controlled experiments. Beyond the main tables, the authors actually test whether the signal is concept-specific and comes from the image: energy-matched random and wrong-concept directions, editing text tokens versus image tokens, the same candidate events scored with the true, a shuffled and a blank image, and a λ=0 run to confirm the extra forward pass is not doing anything by itself. They also report the uncomfortable numbers rather than hiding them: half the comparisons are not significant, the margin adds only 0.07 AUC, and the method introduces 0.36 new false positives per report. I trust these results more than I usually trust a decoding paper, which matters here because the effects are small.

Critical weaknesses
  1. Effect sizes are small and unevenly supported, and the regime studied is far from clinical use. Outside Qwen2.5-VL on MIMIC-CXR and PadChest-GR, improvements are typically 0.3–2 points with intervals that include zero (Table 23); against the strongest competitor only 5/12 RadGraph and 4/12 5c-F1 comparisons are resolved, before any correction for 36 tests. All three backbones are weak zero-shot report generators (MIMIC RadGraph F1 of 8–20, 5c-F1 of 17–30), well below task-adapted systems, and the paper offers no evidence that LOCI helps a model whose positive precision is already high. The abstract's "best in all 12 settings" framing therefore overstates what the data support, and the practical significance of the method remains unestablished. This is the main reason I do not recommend a clear accept.

  2. The clinical claim rests entirely on automatic proxies, and the mechanism covers only part of the error surface. No radiologist reviewed any changed report. RadFact depends on GPT-4 sentence splitting and Llama-3 entailment, CheXbert/RadGraph are applied per image against study-level references (multi-view studies are 2/3 of MIMIC pairs), and PadChest references are grounded sentence lists. Meanwhile the hand-built lexicon exposes only 55–72% of unsupported positives to a trigger (Table 11), LOCI introduces 0.36 new unsupported positives per report while correcting 0.65 (Table 9), and the margin's incremental predictive value is +0.07 AUC. Together these leave open how much of the measured gain would survive expert adjudication and whether the approach generalizes beyond these 26 English chest X-ray findings.

Questions for authors
  • Does LOCI improve a strong task-adapted generator (e.g., a MIMIC-fine-tuned LLaVA/Qwen, or a MAIRA-2/CheXagent-class model)? One such backbone would show whether the method matters where positive precision is already reasonable.
  • Figure 6 shows a sharp layer peak for Qwen2.5-VL; how sensitive are the IU-Xray, CheXpert Plus and PadChest-GR results to the MIMIC-selected layer and (α, λ)? Please report the test-set metrics at the two neighboring layers.
  • Why does Qwen2.5-VL's PadChest-GR RadGraph F1 double (3.31→6.56) while the other backbones barely move? Is the change driven by report length, sentence structure, or specific findings?
  • The blank-image contrast with LOCI's trigger reaches 20.08 5c-F1 vs 22.88 for LOCI (Table 2). How much of LOCI's benefit should be attributed to the trigger schedule versus the concept-matched counterfactual? A VCD-style contrast run only at LOCI trigger steps would separate these.
Reviewer_NWxVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces LOCI, a training-free decoding method designed to reduce hallucinations of positive clinical findings in chest X-ray reports. Unlike existing approaches that primarily intervene at the token level or through global visual perturbations, LOCI is triggered when candidate tokens complete a phrase describing a clinical finding. It identifies the scope of assertions based on the existing context and intervenes only in cases involving affirmative clinical findings. Leveraging the hidden states of clinical concepts within large vision-language models, LOCI constructs a temporary counterfactual branch: it subtracts the projection of the image tokens' hidden states onto the direction of the corresponding clinical concept's hidden state. Finally, it performs local re-ranking of eligible candidate tokens based on the log-probability difference between the original and counterfactual predictions. The authors evaluated this method using three frozen 7B-parameter vision-language models across four chest X-ray datasets, with LOCI achieving superior performance in the vast majority of experiments. Furthermore, the authors investigated the method's effectiveness and underlying mechanisms through RadFact evaluations, comparisons across different concept directions, image perturbation experiments, and component ablation analyses.

Critical strengths
  1. LOCI focuses on clinically significant multi-word phrases, which serve as key semantic units in medical report generation. By leveraging counterfactual branching to intervene on affirmative clinical findings while accounting for the sensitivity of image representations, LOCI integrates representation-level interventions with decoding-time re-ranking in a unified framework. This approach provides a systematic correction for "positive hallucinations" (the generation of clinical findings that are not actually present) in Large Vision-Language Models (LVLMs).
  2. The paper features an extensive experimental scope and a comprehensive set of ablation studies. Beyond conducting experiments across three frozen models and four datasets, the authors also provide thorough mechanistic control studies.
Critical weaknesses
  1. Due to the lack of direct verification of visual authenticity, existing metrics, which primarily measure consistency with reference text, are insufficient to prove that visual hallucinations have genuinely decreased.
  2. Compared to the strongest baseline, a statistically significant advantage is observed only in certain settings; furthermore, the inference time is more than double that of the baseline, representing an inadequate trade-off between performance and cost.
Questions for authors
  1. I noticed that the paper mentions that, for Qwen2.5-VL, LOCI reduces the number of false positives by an average of 0.65 per report, while simultaneously introducing 0.36 new false positives. Have the authors considered whether the diseases associated with these newly introduced false positives pose a significantly greater risk to the patient-that is, could these new false-positive findings lead to more severe consequences in terms of misdiagnosis? Furthermore, are the false-positive conditions eliminated by LOCI consistently concentrated within one or a few specific disease categories? Have the authors conducted any statistical analysis regarding these aspects?
  2. Regarding the description in Section 3.1 of the paper: "Thus coordinated negations such as no focal consolidation, edema, or pneumothorax are skipped, whereas persistence statements such as no change in pleural effusion remain positive", is there an error in the latter part of the statement? Should "no change" be classified as "negative"?
  3. Might LOCI transform one type of false positive finding into another, or into a false negative, or an inappropriate expression of uncertainty? Could you provide some robust analysis discussing these instances where errors shift from one form to another?
Reviewer_gWmHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LOCI is a training free decoding method for reducing hallucinated positive findings in chest X-ray reports from frozen LVLMs. At each step it checks whether any top K candidate token would complete a phrase from a ten finding lexicon in positive scope, using a rule based negation and uncertainty gate on the current clause. For each triggered finding it runs one extra forward pass where the projection of image token hidden states onto a text derived finding direction at a chosen layer is subtracted, computes the change in log probability for the matched candidates, and reranks only those candidates by base log probability plus α times the drop. Candidates that lose probability when the image side concept is removed are promoted, those that do not are demoted. Evaluated on LLaVA-Med, Hulu-Med and Qwen2.5-VL across MIMIC-CXR, IU-Xray, CheXpert Plus and PadChest-GR against VCD, Med-VCD, OPERA, ProjectAway and VTI, LOCI has the best RadGraph F1 point estimate in all twelve settings, with paired bootstrap intervals excluding zero in about half, plus RadFact gains on PadChest-GR and a set of mechanistic controls.

The paper is unusually honest about its own statistics, which I appreciate, but what it is honest about is that the gains are small, often not significant, and obtained on frozen general purpose models that are very far from usable report generators. The mechanism is a modest change on top of ProjectAway and the strongest control gets most of the way there.

Critical strengths
  1. The statistical reporting is better than most papers in this space. Paired study clustered bootstrap intervals are given for every comparison, including against the strongest competing method in Table 24, and the abstract itself says gains are resolved only in a subset. That is rare.
  2. The control set is thoughtful. Random direction, wrong concept direction, blank and shuffled image conditions, energy matched perturbations, and the margin AUC analysis in Table 18 all address the obvious objection that any perturbation would do.
  3. Table 2 is a good design. Separating a conservative prompt, a lexicon penalty, blank image contrast, direct erasure and the full method along five binary design axes makes it clear which pieces matter.
Critical weaknesses
  1. The absolute numbers are so low that the setting is hard to care about. RadGraph F1 of 8 to 20 on MIMIC-CXR and 1 to 7 on PadChest-GR, CheXbert 14 class F1 of 5 to 21. Fine tuned report generators sit around 25 to 30 RadGraph F1 on MIMIC. Nobody deploys a frozen Qwen2.5-VL for radiology reports. The paper cites R2GenGPT and similar but does not test on a single model that was trained for this task, so the question of whether LOCI helps a competent generator is open.
  2. Half of the comparisons are not significant. Against standard decoding, 6 of 12 RadGraph and 6 of 12 5c-F1 intervals exclude zero. Against the best competing decoder, 5 of 12 and 4 of 12. Three of the four CheXpert Plus comparisons and most PadChest-GR comparisons are unresolved. The all twelve first place claim in the abstract is about point estimates on a 202 case test set in one of those datasets.
  3. The lexicon is the evaluation. The ten canonical findings in Appendix C.1 are CheXbert observations, and the 5c-F1 subset is Atelectasis, Cardiomegaly, Consolidation, Edema, and Pleural Effusion, all of which are in the lexicon. The method is triggered on exactly the phrases the metric scores, so a 5c-F1 gain is partly by construction, and 14c-F1 gains are much smaller.
  4. The simplest control is close. In Table 2 blank image contrastive, which keeps the trigger and the scope gate but uses no concept direction, gets 20.08 5c-F1 against 22.88 for full LOCI, and reaches LOCI level logical precision on RadFact. Scope aware direct erasure, which is ProjectAway with the trigger, gets 10.88 RadGraph against 11.07. Most of the value appears to be in deciding when to intervene, not in the counterfactual direction, and the trigger plus gate is a hand written rule system.
  5. Novelty over ProjectAway is thin. ProjectAway already projects text embeddings out of image representations to reduce hallucination. LOCI adds a phrase trigger, a scope gate, and uses the edited branch as a contrast instead of decoding from it. The paper says as much in related work.
  6. Several baselines fall well below standard decoding, for example OPERA on LLaVA-Med MIMIC at 4.48 against 7.98 and VCD below baseline on most Hulu-Med settings. The paper says all methods share prompts and filters, but does not say whether baseline hyperparameters were tuned on validation the way LOCI's were.
Questions for authors
  1. Does LOCI improve a model that was fine tuned for report generation, for example CheXagent, MAIRA-2 or R2GenGPT, where the base RadGraph F1 is above 25? This would most change my assessment.
  2. Were the hyperparameters of VCD, OPERA, ProjectAway and VTI selected on the same MIMIC-CXR validation split as LOCI? If not, please redo with matched tuning, since several baselines currently sit below standard decoding.
  3. What is the 14c-F1 and RadGraph F1 change restricted to CheXbert observations not in the lexicon? This would show whether any gain exists outside the targeted phrases.
  4. Blank image contrastive in Table 2 keeps the trigger and gate and gets most of the way to LOCI. What is the paired CI for LOCI minus blank image contrastive on 5c-F1 and RadFact precision?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.