- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates a confounding factor in occlusion-based evaluations of medical vision-language models (VLMs): conventional control regions may contain substantially less foreground tissue than annotated regions, potentially exaggerating apparent region sensitivity. To address this issue, the authors propose a foreground-matched occlusion control that matches region size and foreground coverage while avoiding annotation overlap. Experiments on SLAKE evaluate six VLMs, showing stronger annotated-region sensitivity for two MedGemma checkpoints, although the comparison is confounded by training-data exposure. The paper also identifies several evaluation-pipeline artifacts, examines the limitations of the proposed CIPE metric, and constructs MedTrust-IN v0.3, a 1,122-item benchmark for assessing image dependence.
Critical strengths
- A meaningful improvement to occlusion-based evaluation. The paper identifies an important confound in conventional occlusion audits: control regions may contain substantially less foreground tissue than annotated regions.
- Useful theoretical and statistical analysis. Proposition 1 formalizes how foreground-coverage differences can bias region-sensitivity estimates under explicit assumptions.
- Thorough methodological auditing and transparent reporting. The paper systematically investigates answer-extraction failures, readout normalization errors, metric–stratifier circularity, and training-data exposure.
Critical weaknesses
- Limited validity and generalizability of the primary findings. The SLAKE experiments occlude image-level annotations rather than question-specific regions, while 93.0% of evaluated items involve images appearing in MedGemma's disclosed training sources (Appendix S1). These factors limit conclusions about visual grounding and generalization. The ChestX-Det follow-up remains exploratory due to post-outcome protocol amendments.
- Incomplete confound control and sample selection bias. Foreground matching does not control anatomical structure, spatial position, or masking artifacts. Only 229 of 400 candidates admit valid controls, favoring sparsely annotated images. Moreover, overlapping annotations in 58 retained items prevent guaranteed union-level coverage matching, potentially compromising control comparability.
- Insufficient evidence isolating the benefit of foreground matching. Legacy and matched runs are unpaired for four general VLMs, with only 34 shared eligible QA items. The MedGemma-4B paired comparison shows a marginal difference (0.054, p = 0.040), but also changes placement rules and sampling. Thus, the improvement cannot be confidently attributed to foreground matching alone.
- Limited validation of MedTrust-IN. MedTrust-IN v0.3 remains highly label-imbalanced (85.6% "yes"), with all 121 SLAKE items having positive answers. Its panel-defined text-only-hard criterion does not establish visual necessity, and 59 items remain unsolved by all panel models even with images. These limitations weaken its reliability for evaluating visual dependence.
Questions for authors
- Question-specific grounding and training exposure: Can the authors provide stronger evidence of region sensitivity on image-disjoint, target-aligned medical VQA samples that are absent from the evaluated models' disclosed training data? Such evidence would materially strengthen the interpretation and generalizability of the main findings.
- Control validity and selection effects: How sensitive are the results to alternative foreground definitions, occlusion fills, and anatomical matching criteria? Can the authors quantify the impact of overlapping annotations and characterize whether the 171 excluded items exhibit systematically different model behavior or clinical characteristics?
- Causal contribution of foreground matching: Can the authors conduct a fully paired comparison between legacy and foreground-matched controls on identical QA samples, using the same validated readout and a comparable number of control draws? This would help determine whether foreground coverage, rather than other placement differences, drives the reported improvements.
- Benchmark validity: What evidence supports the reliability of MedTrust-IN v0.3 as a benchmark of visual dependence beyond its panel-defined text-only-hard criterion? In particular, can the authors provide balanced-label evaluations and independent validation of image-supported answerability to assess the effects of answer priors and screening-model dependence?