acceptodds
Withdrawn from ICLR 2027

MODALENS: Measuring Image Sensitivity in Report-Conditioned Medical VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A radiology report can already answer a clinical question, so it is hard to tellwhether a vision-language model also uses the image. MODALENS, a pairedimage-swap audit, measures how report availability changes image sensitivity:MedGemma-27B on 3,199 paired MIMIC-CXR cases from 293 patients, the 13finding-specific questions per case, each image replaced by one from another study,usually of the same patient, with question and report fixed. Under an explicit answerinstruction, the model’s generated answer changes on 4.43% of trials with the reportand 19.93% without it, a paired increase of 15.5 points (patient-clustered 95%CI [14.5, 16.5]), so report availability reduces image-swap sensitivity under thisprotocol, and substitutions also move continuous answer scores where the binaryprediction does not change. The direction replicates in two further general-domainlineages and on a second institution’s chest X-ray collection; two medical modelsread the substituted image at chance with or without the report. A cumulativeattention block on the report tokens restores image sensitivity when it begins at orbelow layer 22 and not from layer 28 upward, a causal depth bound rather than alocus. The labels are derived from reports, which limits conclusions about visualcorrectness. Code, exact prompts and a committed run record behind every numberare provided as anonymized supplementary material and will be released

Primary area:
datasets and benchmarks
Keywords:
vision-language models, multimodal evaluation, medical imaging, shortcut learning, chest X-ray, model auditing, counterfactual evaluation, clinical machine learning, trustworthy machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_5TVTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates whether medical vision-language models genuinely use radiographic images when answering questions, or whether their answers are largely anchored by accompanying radiology reports. Using MedGemma-27B and 41,587 paired trials from MIMIC-CXR, the authors compare model responses when the original report is available versus when it is removed, while substituting a different image under controlled conditions. For 13 finding-specific questions, the rate at which generated answers change after an image swap increases from 4.43% with the report to 19.93% without it, a difference of 15.5 percentage points. The authors report that the direction of this effect persists across alternative answer-scoring protocols, several model families, and additional dataset controls. The paper frames this as a methodological audit of image sensitivity and report anchoring, rather than a measure of diagnostic accuracy. Its central finding is that report availability can substantially alter how much a model’s answer responds to the image.

Critical strengths
  1. The paper identifies a meaningful confound in medical VQA, a report may supply the answer independently of the image. It holds the question and report fixed while swapping the image, isolating a useful measure of image sensitivity.

  2. The primary analysis includes 41,587 trials across 3,199 cases from 293 patients, with patient-clustered confidence intervals. The report effect persists under alternative answer readouts, across several model families, and in an external chest X-ray dataset. The report-token attention intervention attempts to identify when direct access to report tokens influences image sensitivity, going beyond purely observational results.

Critical weaknesses
  1. An answer changing after an image swap does not establish that the model correctly interprets the relevant finding. Conversely, a stable answer could be correct for both images. The paper acknowledges this, but it limits the clinical implications of the central result.

  2. Report-derived labels limit validity. The study uses report-derived findings rather than independently adjudicated image labels. It therefore cannot determine whether the model's answer is visually correct or whether report anchoring produces a clinically consequential error.

  3. Image substitutions introduce confounds. Although most substitutes come from the same patient, the images can differ in acquisition time and view position. The authors report robustness checks, but the main effect may partly reflect changes in view or other image characteristics rather than the queried finding alone.

  4. The primary analysis involves a methodological choice made after external review. The authors exclude the composite question from the primary analysis partly because its effect was much larger than those of the finding-specific questions. The rationale is disclosed, but this makes it important to assess how conclusions depend on the question-set definition and analysis hierarchy.

  5. The causal mechanism remains only partially resolved. The attention knockout supports a role for direct report-token access, but it does not establish a unique layer where report information is used. The paper itself reports that cross-condition residual patching does not separate from a wrong-donor control.

  6. Generalization remains limited. MIMIC-CXR carries the headline analysis; the external cohort is smaller and lacks equivalent labels. Results across model families help, but do not establish broad performance across clinical tasks or modalities.

Questions for authors
  1. Can the authors validate image sensitivity against independently adjudicated image findings, ideally with radiologist annotations?
  2. How much of the observed effect is attributable to the queried finding itself versus view-position changes, acquisition differences, and other image-level changes?
  3. Why should the composite question be excluded from the primary analysis, and do the main conclusions remain unchanged under a prespecified analysis of all 14 questions?
  4. What evidence supports the causal interpretation of the attention knockout, given that residual patching does not distinguish the proposed mechanism from a wrong-donor control?
  5. Can the authors identify cases where report anchoring causes a demonstrably incorrect answer, rather than simply a failure to respond to an image swap?
  6. How well does the effect generalize to other medical imaging modalities and to settings where no radiology report is supplied at inference time?
Reviewer_MbCfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces MODALENS, a paired image-swap audit designed to measure how report availability affects a medical VLM’s sensitivity to the input image. Using 3,199 frontal MIMIC-CXR images from 293 patients, the audit compares the model’s answers to a fixed question when given either the original image or a substitute from another study, typically from the same patient. Each comparison is conducted both with and without the report; in the report-present condition, the report remains unchanged when the image is swapped. The primary metric is the flip rate: the proportion of trials in which swapping the image changes the model’s yes/no answer.

The main results use MedGemma-27B to answer questions about 13 CheXpert findings under an explicit Yes/No instruction. The flip rate is 4.43% with the report and 19.93% without it, a paired difference of 15.5 percentage points. Continuous yes/no margins show a similar pattern, with mean absolute changes of 0.69 and 2.31, respectively. These findings suggest that providing the report reduces the model’s sensitivity to image substitutions.

The authors examine the robustness of this pattern across three readouts, prompt and answer-option ordering, question templates, an abstention option, and substitute-image resampling. The same direction is observed for MedGemma-4B, Qwen3.5-9B and -27B, and LLaVA-NeXT-Mistral-7B, although CheXagent shows the opposite pattern and LLaVA-Med shows only a small effect. Evaluation on IU X-Ray provides additional evidence.

Critical strengths

1. The phenomenon is shown to be consistent across settings and models, which makes the paper a reliable empirical analysis.

The central finding is that report availability sharply reduces a VLM’s sensitivity to image substitution. The authors show it is not a product of one particular measurement choice: it holds in direction and significance across a wide range of controlled variations.

Critical weaknesses

1. The experimental setting does not adequately reflect the clinical scenarios used to motivate the audit, limiting the practical significance of the findings.

  • In practice, a VLM is used to interpret the image and produce a report or answer clinical questions when no radiologist reading is yet available. Once a radiology report exists, the clinical question has already been answered by an expert, and there is little reason to ask a model the same finding-level yes/no questions with that report in its input.

  • The paper motivates the setup with report verification and second reading, but it does not show that such workflows exist in deployment, cite systems that use them, or describe how a model would be integrated into them. It is therefore unclear what practical conclusion follows from showing that a model given the answer-bearing report relies on it. The result is closer to an expected property of conditioning on informative text than to a characterization of a real deployment risk.

2. The extensive evaluation remains primarily descriptive, without establishing the clinical consequences or explaining the observed effect.

  • Most experiments measure the same underlying quantity—how much the model’s output changes after an image swap—across different readouts, prompts, models, and datasets.
  • The mechanistic analyses also provide limited explanation: logit-lens, probing, and steering experiments yield negative results, while the knockout analysis only constrains the depth at which the effect may arise. No mitigation is evaluated.
  • Consequently, the paper documents a recurring behavioral pattern but does not establish why it occurs, when it produces clinically meaningful errors, or how it might be addressed, limiting its contribution beyond an empirical observation.
Questions for authors

1. What is the main practical takeaway for the use of VLMs in real clinical settings?

The paper documents that report availability reduces image sensitivity across many settings, but it is less clear what this implies in practice. Concretely:

  • Which deployment scenarios do you consider at risk?
  • What should practitioners or developers do differently because of these findings, for example avoid supplying reports, add a verification step, or evaluate models with this audit before deployment?
  • Under what conditions would you consider the observed reliance on the report acceptable, given that the report is an expert reading?

2. What causes the anchoring to the report?

  • The paper shows which input content drives the effect (answer-bearing text, via the text controls) and where report access matters (layers 0–~22, via the attention knockout). It does not test why the model relies on the report.

[Suggestions for future work]

  • The paper’s 68 tables and 44-page appendix make its core message difficult to identify, and the writing is often unclear and hard to follow. I strongly encourage the authors to streamline the manuscript and emphasize its central claims in a future revision.
Reviewer_obrcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how radiology reports affect image sensitivity in medical vision-language models. The authors propose a paired image-swap evaluation that compares changes in model answers with and without the original report. On MIMIC-CXR, MedGemma-27B changes its answer less often after an image swap when the report is present. The authors also evaluate additional models, prompts, and a second dataset, and use attention interventions to investigate the role of report tokens.

Critical strengths

Practically relevant question. This paper examines whether medical VLMs remain sensitive to images when radiology reports are available. This is an important question for applications where a model is expected to assess an image independently rather than restate the report.

Critical weaknesses
  1. Unclear motivation and problem formulation. This paper links report verification and conflicting evidence to image sensitivity, but does not clearly explain how its image-swap measurement addresses these motivating concerns.
  2. Limited insight into the observed behavior. This paper finds that reports generally reduce sensitivity to image changes, but does not distinguish the potential causes. The attention interventions provide some evidence, yet do not explain how image and report information interact. Most additional experiments test the robustness of the observation rather than deepen its interpretation.
Questions for authors
  1. Many experiments repeatedly validate the same observation, while the conclusions are heavily qualified by limitations. What new insight do these analyses provide beyond showing that reports reduce image-swap sensitivity?
  2. Whole-image swaps can change both the target finding and unrelated image features, making it difficult to determine what drives the observed answer changes.
  3. This paper combines image substitution with report ablation, but does not clearly establish a substantive advance over existing counterfactual evaluations of visual reliance.
  4. The attention interventions provide some evidence about the role of report tokens, but do not explain how image and report information interact. Many additional experiments repeatedly test the same observation, while the conclusions are heavily qualified. The insight gained beyond reduced image-swap sensitivity remains limited.
1 comment

I have not received any feedback, so I will maintain my original rating.

Reviewer_kxhAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript introduces MODALENS, a paired image-swap audit measuring how report availability changes image sensitivity in medical vision-language models. The method hold the question and report fixed, replace the original image with an image from another study of the same patient, and compare answer-flip rates and continuous score changes with and without the report. On 3,199 MIMIC-CXR cases from 293 patients, using 13 finding-specific questions, MedGemma-27B’s generated-answer flip rate increases from 4.43% with the report to 19.93% without it. The manuscript also includes additional models, a second institutional dataset, text controls, and report-token attention interventions.

Critical strengths

The controlled paired design and robustness checks across readouts, models, and datasets provide credible evidence that report availability reduces measured image sensitivity in several evaluated settings.

Critical weaknesses
  1. Replacing the image may change several findings, the imaging view, and other features at once. An answer change does not show that the model noticed the finding being asked about. Likewise, an unchanged answer does not prove that the model ignored relevant visual evidence. Labels taken from reports cannot independently confirm what the displayed image actually shows.
  2. Earlier studies already use image swaps and region-occlusion tests. MODALENS adds a comparison between having and not having a report. However, matching the substituted image’s label does not necessarily mean that the image caused the answer to change. The attention analysis mainly shows roughly when access to report tokens affects image sensitivity. These results do not yet clearly establish what additional model failures the audit can reliably identify or what new explanation it provides for reliance on text.
Questions for authors
  1. Validate the sensitivity measure using independent radiologist annotations, especially for image pairs where the queried finding truly changes. Assess whether the measure distinguishes reasonable reliance on an accurate report from failure to use relevant visual evidence.
  2. Specify what MODALENS reveals that existing image-swap or occlusion tests miss. Provide evidence that comparing answers with and without a report adds useful information beyond changing the sensitivity score.
  3. Repeat the report-certainty and directional analyses under the final setup: an explicit Yes/No instruction, the 13 finding-specific questions, and generated-answer scoring. Report whether the conclusions still hold.
1 comment

Since I have not found the authors' responses to my concerns, I will keep my rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

General response

We thank the reviewers for their time. None of the four reviews raises a major soundness concern or disputes a reported number, and the reviews describe the evidence as credible and consistent across settings and models. The ratings therefore turn on novelty and insight. Each point raised there is answered by a result already in the submission, which we list below.

“Limited insight beyond reduced image-swap sensitivity.” Four results in the paper are not predicted by that statement: (i) a hedged report (“possible”, “cannot exclude”) pins the answer as firmly as a certain one; (ii) when the substitute image’s label turns a finding positive, the model follows it markedly less often with the report than without; (iii) asked directly whether the report matches the image, the model almost never says no, even when the labels disagree; (iv) a chest-X-ray-tuned model (CheXagent) shows the opposite sign and LLaVA-Med a near-zero effect, so the behaviour is a property of the model, not of the task. These separate causes; they do not repeat one observation.

“Whole-image swaps change more than the target finding.” The same substitution is applied in both arms. Whatever else the swap changes is identical with and without the report, and the reported effect is the paired difference between arms, not the raw flip rate.

“No advance over existing image-swap tests.” Those vary the image alone. MODALENS varies report availability on the same paired cases, with patient-clustered intervals, which isolates the report’s contribution and exposes the per-model sign change.

“Unclear motivation.” A report accompanies the image in longitudinal reading (the prior study’s report with the current image), in report verification, and in agent pipelines that retrieve prior records. Most of our substitutions are same-patient, which is the first setting.

Question set. The primary analysis uses the 13 finding-specific questions (4.43% vs 19.93%, +15.5 points [14.5, 16.5]). The analysis on all 14 questions is reported as a robustness check and gives the same direction.

Scope. As the abstract states, labels are report-derived, so the paper claims image sensitivity, not visual correctness. Independent reader labels on the changed-label pairs are the next step. We will also make the presentation easier to follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.