- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates whether medical vision-language models genuinely use radiographic images when answering questions, or whether their answers are largely anchored by accompanying radiology reports. Using MedGemma-27B and 41,587 paired trials from MIMIC-CXR, the authors compare model responses when the original report is available versus when it is removed, while substituting a different image under controlled conditions. For 13 finding-specific questions, the rate at which generated answers change after an image swap increases from 4.43% with the report to 19.93% without it, a difference of 15.5 percentage points. The authors report that the direction of this effect persists across alternative answer-scoring protocols, several model families, and additional dataset controls. The paper frames this as a methodological audit of image sensitivity and report anchoring, rather than a measure of diagnostic accuracy. Its central finding is that report availability can substantially alter how much a model’s answer responds to the image.
Critical strengths
-
The paper identifies a meaningful confound in medical VQA, a report may supply the answer independently of the image. It holds the question and report fixed while swapping the image, isolating a useful measure of image sensitivity.
-
The primary analysis includes 41,587 trials across 3,199 cases from 293 patients, with patient-clustered confidence intervals. The report effect persists under alternative answer readouts, across several model families, and in an external chest X-ray dataset. The report-token attention intervention attempts to identify when direct access to report tokens influences image sensitivity, going beyond purely observational results.
Critical weaknesses
-
An answer changing after an image swap does not establish that the model correctly interprets the relevant finding. Conversely, a stable answer could be correct for both images. The paper acknowledges this, but it limits the clinical implications of the central result.
-
Report-derived labels limit validity. The study uses report-derived findings rather than independently adjudicated image labels. It therefore cannot determine whether the model's answer is visually correct or whether report anchoring produces a clinically consequential error.
-
Image substitutions introduce confounds. Although most substitutes come from the same patient, the images can differ in acquisition time and view position. The authors report robustness checks, but the main effect may partly reflect changes in view or other image characteristics rather than the queried finding alone.
-
The primary analysis involves a methodological choice made after external review. The authors exclude the composite question from the primary analysis partly because its effect was much larger than those of the finding-specific questions. The rationale is disclosed, but this makes it important to assess how conclusions depend on the question-set definition and analysis hierarchy.
-
The causal mechanism remains only partially resolved. The attention knockout supports a role for direct report-token access, but it does not establish a unique layer where report information is used. The paper itself reports that cross-condition residual patching does not separate from a wrong-donor control.
-
Generalization remains limited. MIMIC-CXR carries the headline analysis; the external cohort is smaller and lacks equivalent labels. Results across model families help, but do not establish broad performance across clinical tasks or modalities.
Questions for authors
- Can the authors validate image sensitivity against independently adjudicated image findings, ideally with radiologist annotations?
- How much of the observed effect is attributable to the queried finding itself versus view-position changes, acquisition differences, and other image-level changes?
- Why should the composite question be excluded from the primary analysis, and do the main conclusions remain unchanged under a prespecified analysis of all 14 questions?
- What evidence supports the causal interpretation of the attention knockout, given that residual patching does not distinguish the proposed mechanism from a wrong-donor control?
- Can the authors identify cases where report anchoring causes a demonstrably incorrect answer, rather than simply a failure to respond to an image swap?
- How well does the effect generalize to other medical imaging modalities and to settings where no radiology report is supplied at inference time?
I have not received any feedback, so I will maintain my original rating.