- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces SAFER, a benchmark of 8,294 chest radiograph and report pairs. The authors insert errors through single word substitutions. They evaluate eight vision and language models and several control systems. A classifier trained only on report text achieves an area under the receiver operating characteristic curve of 0.9183 on a held out split. The strongest evaluated vision and language model achieves 0.7001 on the full benchmark. The paper also examines lexical shortcuts, nearly constant model outputs, and weaknesses in its composite score.
Critical strengths
- Its score of 0.9183 is striking. Removing the original and substituted word spans reduces the score to 0.654. These results give a concrete reason to question whether performance on this benchmark measures visual grounding.
- CheXagent obtains the highest composite score among the evaluated models despite a specificity of only 0.032. The reported ranking score for MedGemma changes by just 0.006 when images are shuffled. Both checks prevent an overly favorable reading of the model results.
Critical weaknesses
-
The paper identifies Experiments 6 through 9 as its primary methodological contribution. Appendix A.9, however, ends after Experiment 4. Table 4 provides scores without the exact transformations, retraining procedures, or uncertainty estimates. This matters for the claim that the fixed substitution vocabulary explains approximately 55 percent of the text signal. It also matters because only reports containing eligible words can receive an injected error. A comparison using clean and altered versions of the same eligible reports would test that remaining source of signal. The related work also omits SHOVIR [1], which studies visual shortcuts through controlled radiology image perturbations, and ReXTrust [2], which studies finding level hallucination detection. Their tasks differ from SAFER, but both are relevant to the claimed contribution. arxiv.org
-
The text classifier is evaluated on 2,042 held out cases. MedGemma is evaluated on all 8,294 cases. Appendix A.7 treats a cross source difference of 0.0415 as a bound on the resulting discrepancy in area under the curve. Appendix A.9.3 identifies that 0.0415 value as a difference in balanced accuracy. It cannot bound a difference in another metric across different sample sets. Patient identifiers are also available for MIMIC-CXR [3]. At minimum, they permit a patient separated analysis on that source. The reported gap is large, but the current calculations do not establish its precise size under matched evaluation.
[1] Ruffini, F., Salmé, M., Sicilia, R., Guarrasi, V., and Soda, P. “SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation.” arXiv:2606.30201, 2026.
[2] Hardy, R., Kim, S. E., Ro, D. H., and Rajpurkar, P. “ReXTrust: A Model for Fine-Grained Hallucination Detection in AI-Generated Radiology Reports.” Proceedings of Machine Learning Research, 281:173–182, 2025.
[3] Johnson, A., Pollard, T., Mark, R., Berkowitz, S., and Horng, S. MIMIC-CXR Database, version 1.0.0. PhysioNet, 2019.
Questions for authors
What exact transformations and retraining procedures produced Experiments 6 through 9? Does the residual text signal persist when clean and altered reports are paired from the same eligible originals?
What are the text classifier and MedGemma results on the identical 2,042 cases, with paired uncertainty estimates? How does the text classifier perform with patient separated training and testing on MIMIC-CXR?
2 comments
Official Comment
I appreciate the authors’ thorough response. My concerns are mostly resolved within the limited time available, and I would like to increase my rating.
Thank you for the careful reading. We address each point.
Missing appendix. You are right. Appendices A.9.5–A.9.13 are added with the exact masking rules, both vocabulary lists, the 30-run protocol and the retraining procedures. They distinguish the two ablations (A.9.6):
Uncertainty (Table 4). Over 30 re-split, re-trained runs: swap vocabulary Δ 0.239 [0.223, 0.260] (57.9% ± 2.2% of the above-chance gap); grammar and length/broader-vocabulary intervals include zero.
A.7 and matched evaluation (Table 3; A.7). You are correct that 0.0415 is a balanced-accuracy gap and cannot bound an AUROC difference. It is replaced by the direct paired comparison on the identical 2,042 cases: TF-IDF 0.918 [0.906, 0.930], MedGemma 0.707 [0.687, 0.726], paired Δ 0.212 [0.190, 0.234].
Patient separation (A.6). Our earlier statement about MIMIC-CXR identifiers was wrong. In our subset every study comes from a distinct subject, so the source is patient-disjoint by construction; a MIMIC-CXR-only classifier gives 0.887 subject-grouped vs. 0.885 report-level (paired Δ 0.002 [−0.019, 0.036]).
Paired eligible originals (§6.3). Clean vs. injected versions of the same reports give 0.675 with the swap vocabulary excluded, matching the benchmark's 0.674. This exposed a substring-matching bug in our injector, which accounts for ≈33% of the above-chance gap; <2% remains unexplained. We corrected the injector and release SAFER v1.1, on which the vocabulary-excluded residual falls to 0.552.
Re-evaluation on v1.1 (Exp. 12; §6.5; Table 12). All seven VLMs, baselines and image ablations are re-evaluated. The 1,417 changed studies are run afresh; unchanged-study outputs are reused after 200 anchor studies were re-run (verdict agreement ≥0.995); image ablations are re-run on all studies. The leak survives the fix: on the identical 2,042 studies TF-IDF reaches 0.907 vs. MedGemma 0.673 (gap 0.234 [0.211, 0.256]); on all 8,166 studies MedGemma falls from 0.700 to 0.661. Elsewhere, results are on v1 unless marked v1.1.
Related work (§2). SHOVIR and ReXTrust are now discussed. We also use ReXErr, from the ReXTrust group, as an independent test of the protocol (Exp. 13, §6.6): a text-only probe reaches 0.876 on its same-report clean/edited pairs, and zero-shot MedGemma with no image 0.768.
Presentation. The paper follows one evaluation chain (probe → elimination with intervals → attribution → image ablation → v1.1 → external benchmark), with metric material moved to A.9.13. If any section remains hard to follow, we would be grateful to know which.