acceptodds
Under review as a conference paper at ICLR 2027

Foreground-Matched Occlusion Controls for Auditing Annotated-Region Sensitivity in Medical VLMs

Abstract

Occlusion audits of medical vision-language models (VLMs) read a larger answer shift from blacking out an annotated region than from a same-sized control box as use of the region, assuming both occlusions are equally strong. A black fill on air changes almost no pixels, so a control there loses to nearly any tissue occlusion. A minimum-overlap control could favour air (replayed on 229 items, its mean tissue coverage was 0.659 against 0.943 matched), and under it every model run appeared region-sensitive. We introduce a foreground-matched control (same size, matched tissue coverage, no overlap with any annotated box) and a proposition showing, under a monotone foreground-dose model, that a comparator covering less tissue biases the win rate upward and matching bounds that term. On SLAKE the matched control gives graded sensitivity: both MedGemma checkpoints lead (strict-win rates 0.639 and 0.612) and survive post-hoc resampling of 175 source images, while four general models are lower (0.525–0.563) and their detection mostly depends on how ties are scored. The scope is as follows: historical and matched runs are unpaired except for MedGemma-4B, whose CIPE win rate is marginally lower when matched, so we do not attribute their difference to the control; 93.0% of the 229 items sit on images the MedGemma family reports training on, so the ranking is exposure-confounded and is not read as specialisation; and our CIPE readout showed no advantage over cheaper ones in a pre-registered head-to-head. The masked boxes annotate the image, not the question target, so this is image-level sensitivity, not target localisation, for one specialist family at two scales on the 229 of 400 SLAKE items admitting a matched control, which favour sparsely annotated images. We also build MedTrust-IN v0.3 (released upon publication): 1,122 panel-defined text-only-hard medical-VQA items from source test splits, with a fail-closed build-time provenance gate. An audit should report and match what its control covers, drop and count unmatched items, and report ties at half credit beside strict wins.

Primary area:
datasets and benchmarks
Keywords:
medical vision-language models, region grounding, occlusion controls, benchmark auditing, confounds, dataset release

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_B8uvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a confounding factor in occlusion-based evaluations of medical vision-language models (VLMs): conventional control regions may contain substantially less foreground tissue than annotated regions, potentially exaggerating apparent region sensitivity. To address this issue, the authors propose a foreground-matched occlusion control that matches region size and foreground coverage while avoiding annotation overlap. Experiments on SLAKE evaluate six VLMs, showing stronger annotated-region sensitivity for two MedGemma checkpoints, although the comparison is confounded by training-data exposure. The paper also identifies several evaluation-pipeline artifacts, examines the limitations of the proposed CIPE metric, and constructs MedTrust-IN v0.3, a 1,122-item benchmark for assessing image dependence.

Critical strengths
  1. A meaningful improvement to occlusion-based evaluation. The paper identifies an important confound in conventional occlusion audits: control regions may contain substantially less foreground tissue than annotated regions.
  2. Useful theoretical and statistical analysis. Proposition 1 formalizes how foreground-coverage differences can bias region-sensitivity estimates under explicit assumptions.
  3. Thorough methodological auditing and transparent reporting. The paper systematically investigates answer-extraction failures, readout normalization errors, metric–stratifier circularity, and training-data exposure.
Critical weaknesses
  1. Limited validity and generalizability of the primary findings. The SLAKE experiments occlude image-level annotations rather than question-specific regions, while 93.0% of evaluated items involve images appearing in MedGemma's disclosed training sources (Appendix S1). These factors limit conclusions about visual grounding and generalization. The ChestX-Det follow-up remains exploratory due to post-outcome protocol amendments.
  2. Incomplete confound control and sample selection bias. Foreground matching does not control anatomical structure, spatial position, or masking artifacts. Only 229 of 400 candidates admit valid controls, favoring sparsely annotated images. Moreover, overlapping annotations in 58 retained items prevent guaranteed union-level coverage matching, potentially compromising control comparability.
  3. Insufficient evidence isolating the benefit of foreground matching. Legacy and matched runs are unpaired for four general VLMs, with only 34 shared eligible QA items. The MedGemma-4B paired comparison shows a marginal difference (0.054, p = 0.040), but also changes placement rules and sampling. Thus, the improvement cannot be confidently attributed to foreground matching alone.
  4. Limited validation of MedTrust-IN. MedTrust-IN v0.3 remains highly label-imbalanced (85.6% "yes"), with all 121 SLAKE items having positive answers. Its panel-defined text-only-hard criterion does not establish visual necessity, and 59 items remain unsolved by all panel models even with images. These limitations weaken its reliability for evaluating visual dependence.
Questions for authors
  1. Question-specific grounding and training exposure: Can the authors provide stronger evidence of region sensitivity on image-disjoint, target-aligned medical VQA samples that are absent from the evaluated models' disclosed training data? Such evidence would materially strengthen the interpretation and generalizability of the main findings.
  2. Control validity and selection effects: How sensitive are the results to alternative foreground definitions, occlusion fills, and anatomical matching criteria? Can the authors quantify the impact of overlapping annotations and characterize whether the 171 excluded items exhibit systematically different model behavior or clinical characteristics?
  3. Causal contribution of foreground matching: Can the authors conduct a fully paired comparison between legacy and foreground-matched controls on identical QA samples, using the same validated readout and a comparable number of control draws? This would help determine whether foreground coverage, rather than other placement differences, drives the reported improvements.
  4. Benchmark validity: What evidence supports the reliability of MedTrust-IN v0.3 as a benchmark of visual dependence beyond its panel-defined text-only-hard criterion? In particular, can the authors provide balanced-label evaluations and independent validation of image-supported answerability to assess the effects of answer priors and screening-model dependence?
Reviewer_ZZcmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the reliability of occlusion-based evaluations of medical vision-language models (VLMs). The authors argue that randomly selected control regions may contain mostly background pixels, leading to an overestimation of annotated-region sensitivity. To address this issue, they propose a foreground-matched occlusion control that matches the size and tissue coverage of annotated regions. Experiments on SLAKE evaluate six VLMs, including two MedGemma checkpoints. The paper also introduces the MedTrust-IN dataset and discusses several evaluation artefacts identified during the experiments.

Critical strengths
  1. A relevant problem with a practical solution. The paper highlights an important confounding factor in occlusion-based evaluation: different occluded regions may produce different effects simply because they contain different amounts of foreground tissue. Matching tissue coverage is a reasonable improvement over unconstrained random controls and complements earlier work on perturbation-based evaluation [1].

  2. Transparency about negative results and limitations. The authors acknowledge several problems in their experimental pipeline, including answer-extraction errors, mismatched evaluation samples, and training-data exposure. Reporting these issues and the failure of CIPE to outperform simpler metrics may provide useful lessons for future evaluation research.

References

[1] Rong et al. (2022). A Consistent and Efficient Evaluation Strategy for Attribution Methods. ICML.

Critical weaknesses

1. The presentation quality is substantially below the standard expected for an ICLR submission.

The paper appears insufficiently polished and, in several places, reads more like an internal experiment report than a finished research paper.

For example, Table 1 directly includes implementation artefact filenames such as image_cluster_sensitivity.json and expclean_*.json. Figure 3 similarly references paired34_stored_mask_descriptive.json, while the text above the figure reads "34 source-key-matched QA on 33 images; stored CIPE masks only." Although these details may be useful for reproducibility, their presentation in the main paper is distracting and makes the scientific message difficult to follow.

The figure quality is particularly concerning. Figure 1 contains overly thick arrows, crowded connections, unconventional symbols, and inconsistent font sizes. Figures 2 and 3 also contain small labels and compressed visual elements that are difficult to interpret. The reader must rely heavily on lengthy captions to understand the results.

More broadly, the main text repeatedly discusses implementation details, corrections, internal experiment records, and withdrawn interpretations. These details are not well organized into a clear scientific narrative. Providing extensive code references and experiment logs does not substitute for a clear explanation or establish experimental soundness. Substantial revision of the writing, figures, and organization is needed.

2. The empirical evidence does not yet establish a sufficiently strong or novel contribution.

The foreground-matched control is a reasonable refinement, but its novelty and significance are limited in the context of existing perturbation and medical visual-grounding research [1–4].

Several limitations weaken the central evaluation. First, only 229 of 400 candidate SLAKE items admit matched controls, creating substantial selection bias. Second, the paper acknowledges that 93% of these items are associated with images appearing in the MedGemma family's disclosed training data, making the specialist-model comparison difficult to interpret. Third, the historical and matched-control experiments are largely unpaired, so the observed differences cannot be confidently attributed to foreground matching.

More fundamentally, matching foreground coverage does not ensure that two occlusions have comparable effects. Anatomical location, spatial structure, and diagnostic relevance remain uncontrolled. Since the annotated boxes are not aligned with individual questions, the reported sensitivity does not establish faithful reliance on the relevant medical evidence. The target-aligned follow-up is exploratory and involves non-blind amendments.

Compared with recent work using more explicit region-level grounding and clinical annotations [2–4], the evidence here is relatively limited. The paper would benefit from a direct, controlled validation showing that foreground matching improves the accuracy of faithfulness assessment, rather than only changing the measured scores.

References

[1] Rong et al. (2022). A Consistent and Efficient Evaluation Strategy for Attribution Methods. ICML.

[2] Nguyen et al. (2025). Localizing Before Answering: A Benchmark for Grounded Medical Visual Question Answering. IJCAI.

[3] Sadanandan and Behzadan (2026). Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMs. MICCAI 2026 Satellite Events.

[4] Jiang et al. (2026). MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence. NeurIPS 2026 Evaluations & Datasets Track.

Questions for authors
  1. Can the authors provide a fully paired comparison between random and foreground-matched controls using identical images, questions, models, and readouts? This would help establish whether the observed differences are genuinely caused by the proposed control.

  2. Can the authors validate the method on an independent, training-disjoint dataset with question-aligned annotations, ideally including a known positive control? How sensitive are the conclusions to the substantial exclusion of candidate samples?

  3. Can the authors substantially revise the figures and overall presentation, replacing implementation-specific labels with clear scientific descriptions and organizing the main results into a more coherent narrative?

Reviewer_oKMJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the evaluation of visual faithfulness in medical vision-language models. It points out that traditional occlusion-based experiments—which control only for the area of ​​the occluded region while ignoring differences in the foreground content—can lead to biased evaluation results. To mitigate this, the authors propose "Foreground-Matched Control," an approach that matches the foreground coverage between target regions and control regions. The method is validated through both mathematical analysis and experiments in the context of medical visual question answering (VQA).

Critical strengths

1.I find the research problem addressed in this paper quite interesting; rather than simply improving the model, the authors identify potential inherent unfairness in traditional occlusion experiments, a perspective that is certainly commendable.

2.The authors not only propose a foreground-matching method but also provide a mathematical derivation to explain why variations in foreground coverage can influence evaluation results, offering a solid theoretical foundation.

Critical weaknesses

1.The text contains numerous records,such as code debugging notes and experimental logs,that are presented from the perspective of a third-party evaluator.

2.This paper functions more as a discussion and documentation of the research process, making it unsuitable for publication as a formal academic paper.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.