LOCI: Mitigating Hallucinations in Radiology Reports with Phrase-Triggered Counterfactual Decoding

Hallucinations remain a critical reliability issue for medical large vision-language models, particularly when generated text asserts clinical findings unsupported by the image. In chest X-ray report generation, mitigating such errors is challenging because clinically meaningful findings are often expressed as multi-token phrases whose interpretation depends on assertion scope, such as positive, negated, or uncertain. Existing training-free decoding methods typically operate at the token level or rely on global visual perturbations, making it difficult to selectively intervene on a specific clinical assertion. We introduce LOCI, a training-free, scope-aware decoding method that activates only when a candidate completes a clinical finding phrase in positive scope. For the matched finding, LOCI constructs a text-derived direction at the intervention layer, subtracts its projection from image-token hidden states in a temporary counterfactual branch, and reranks only plausible matched candidates according to the induced relative change in log probability. The edited branch is discarded after the current decoding step. Across three frozen 7B LVLMs and four chest X-ray datasets, LOCI obtains the best RadGraph F1 point estimate in all 12 backbone-dataset settings and the best CheXbert 5-category F1 result in 11 of 12 settings, although statistically resolved gains are concentrated in a subset of comparisons. On PadChest-GR, RadFact evaluation further shows higher logical F1 and fewer false positive findings across all three backbones. Mechanistic controls further show that the counterfactual signal is strongest for matched same-layer directions under the true-image condition. These results support phrase-triggered counterfactual sensitivity as a targeted inference-time signal for mitigating positive-finding hallucinations in frozen LVLMs. Code is available at https://anonymous.4open.science/r/LOCI-837B/.