- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents LOCI, a training-free decoding method for chest X-ray report generation with medical VLMs. At each decoding step it looks at the top-K candidate tokens and checks whether any of them would complete a clinical finding phrase (from a hand-written lexicon of 26 findings) as a positive statement rather than a negated or uncertain one. If so, LOCI runs a check on that candidate: it takes the model's own text representation of the finding (a direction vector obtained by feeding just the phrase, without an image, and reading the hidden state at one chosen layer), removes that direction from the image-token hidden states at the same layer, and recomputes the next-token distribution. If the candidate's probability drops a lot, the model was relying on image evidence for that finding, and the candidate is scored up; if it barely changes or goes up, the candidate is being driven by language prior, and it is scored down (Eqs. 1–2, with weight α). Only the matched candidates are re-scored; everything else, and every step where nothing triggers, is plain greedy decoding. The edited representation is used only for this check and then thrown away.
Critical strengths
-
The motivation is clear and sensible. Clinical statements are often sentence-level rather than token-level, and LOCI explicitly targets this scenario with proper triggering mechanism.
-
Compehensive controlled experiments. Beyond the main tables, the authors actually test whether the signal is concept-specific and comes from the image: energy-matched random and wrong-concept directions, editing text tokens versus image tokens, the same candidate events scored with the true, a shuffled and a blank image, and a λ=0 run to confirm the extra forward pass is not doing anything by itself. They also report the uncomfortable numbers rather than hiding them: half the comparisons are not significant, the margin adds only 0.07 AUC, and the method introduces 0.36 new false positives per report. I trust these results more than I usually trust a decoding paper, which matters here because the effects are small.
Critical weaknesses
-
Effect sizes are small and unevenly supported, and the regime studied is far from clinical use. Outside Qwen2.5-VL on MIMIC-CXR and PadChest-GR, improvements are typically 0.3–2 points with intervals that include zero (Table 23); against the strongest competitor only 5/12 RadGraph and 4/12 5c-F1 comparisons are resolved, before any correction for 36 tests. All three backbones are weak zero-shot report generators (MIMIC RadGraph F1 of 8–20, 5c-F1 of 17–30), well below task-adapted systems, and the paper offers no evidence that LOCI helps a model whose positive precision is already high. The abstract's "best in all 12 settings" framing therefore overstates what the data support, and the practical significance of the method remains unestablished. This is the main reason I do not recommend a clear accept.
-
The clinical claim rests entirely on automatic proxies, and the mechanism covers only part of the error surface. No radiologist reviewed any changed report. RadFact depends on GPT-4 sentence splitting and Llama-3 entailment, CheXbert/RadGraph are applied per image against study-level references (multi-view studies are 2/3 of MIMIC pairs), and PadChest references are grounded sentence lists. Meanwhile the hand-built lexicon exposes only 55–72% of unsupported positives to a trigger (Table 11), LOCI introduces 0.36 new unsupported positives per report while correcting 0.65 (Table 9), and the margin's incremental predictive value is +0.07 AUC. Together these leave open how much of the measured gain would survive expert adjudication and whether the approach generalizes beyond these 26 English chest X-ray findings.
Questions for authors
- Does LOCI improve a strong task-adapted generator (e.g., a MIMIC-fine-tuned LLaVA/Qwen, or a MAIRA-2/CheXagent-class model)? One such backbone would show whether the method matters where positive precision is already reasonable.
- Figure 6 shows a sharp layer peak for Qwen2.5-VL; how sensitive are the IU-Xray, CheXpert Plus and PadChest-GR results to the MIMIC-selected layer and (α, λ)? Please report the test-set metrics at the two neighboring layers.
- Why does Qwen2.5-VL's PadChest-GR RadGraph F1 double (3.31→6.56) while the other backbones barely move? Is the change driven by report length, sentence structure, or specific findings?
- The blank-image contrast with LOCI's trigger reaches 20.08 5c-F1 vs 22.88 for LOCI (Table 2). How much of LOCI's benefit should be attributed to the trigger schedule versus the concept-matched counterfactual? A VCD-style contrast run only at LOCI trigger steps would separate these.