- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper addresses the safety issue of watermark usage in medical images, conducting a comprehensive evaluation of five watermarking schemes across 11 LLMs and 7 VLMs on multiple tasks covering both unimodal and multimodal clinical reasoning. The paper's findings show that watermarks can cause significant degradation under multiple failure modes, including lexical confusion, terminological hallucination, and amplified misattribution or omission of image findings.
Critical strengths
-
The safety of medical VLMs is a research problem that cannot be ignored;
-
The evaluated models are relatively comprehensive, including 11 large language models and 7 vision-language models;
-
The experiments and analysis are sufficient, the content coverage is comprehensive, and the evaluation metrics are reasonable.
Critical weaknesses
-
Using watermarks for medical image provenance is a development trend worth considering, is it necessary to use watermarks for medical image provenance? If medical institutions do not use watermarks as a basis for provenance, then the premise of this research topic does not hold, since embedding watermarks into medical images is itself unsafe;
-
The paper cites too many arXiv preprints;
-
Most of the experimental content is placed in the supplementary material, making it difficult for readers to follow;
-
Since there are already papers studying watermark attacks against medical VLMs, what is the significance of the evaluation experiments in this paper?
Questions for authors
If the authors can properly address the above shortcomings, I am inclined to raise the score.
4 comments
Thank you for the reviewer's careful response, which has resolved some of my concerns. However, I believe that although watermark attacks targeting medical VLMs are relatively scarce, the method proposed by the authors should still be further compared with other watermark attacks (outside the medical VLM domain), since the approaches are similar and only the domain differs. In addition, the current version of the paper still cites a considerable number of arXiv papers that have not undergone peer review. One last point: I would like to understand why, in the AI usage statement, the authors say they used AI to implement the method—why did the authors not write the code themselves to carry out the adversarial attack, given that injecting watermarks into medical images and optimizing them is not a particularly difficult task?
We thank the reviewer for engaging with our response. However, the reviewer’s comments indicate a fundamental misunderstanding of the subject of our work, which we would like to clarify.
On comparison with other watermark attacks. Our paper does not study watermark attacks, and no images are watermarked at any point. As stated in the abstract, Sec. 2 and Sec. 3.1, we evaluate generation-time text watermarks, such as KGW (Kirchenbauer et al., 2023), which embed a detectable signal into the text a model generates. Medical images are passed unchanged to the VLM; we measure how watermarking the generated text degrades its clinical quality. Attack methods are therefore not a meaningful baseline.
On arXiv citations. As stated earlier, only 9 of our 40 references are arXiv preprints. Six of them are the official technical reports of the models and benchmark we evaluate, which are released only on arXiv and are the standard citations for these artifacts, as common practice in ML papers. We have re-checked the remaining three as of today; none has a peer-reviewed version yet. We will update them if they appear before the camera-ready deadline.
On the LLM usage statement. As above, the paper contains no adversarial attack and no watermark injection into images. While a substantial portion of the code was written by hand, we disclose that parts of the implementation were AI-assisted, as required by the ICLR policy. We used it deliberately, as we believe that, used properly, it makes code better and less error-prone.
Thank you to the authors for their patient response to my comments; I will give them careful consideration.
We thank Reviewer mAtb for their valuable feedback, and address their individual points on weaknesses below.
Q1: Is watermarking for medical image provenance necessary, and is the premise valid? (W1)
We thank the reviewer for this question. First, we would like to make clear that we never watermark images (they are inputs only). The object of interest in this study is text-decoding watermarks. What we find is that watermarking the generated text tokens degrades how VLMs interpret the image, which is fed unchanged into the VLM encoder. The underlying question remains whether watermarking medical text is a realistic premise. We believe it is:
Text watermarking is now required by regulation in the EU. Article 50 of the EU AI Act has applied since 2 August 2026. Under the accompanying Code of Practice on Transparency of AI-Generated Content, providers must mark generated content in a machine-readable way. This implies that LLM/VLM-generated output is increasingly likely to be watermarked, and we study its medical-specific failures, which so far have only been evaluated on general-domain benchmarks, and are non-trivial to detect, as our work suggests.
Q2: Why does the paper cite so many arXiv preprints? (W2)
We thank the reviewer for this remark. Of our 40 citations, 24 are peer-reviewed publications. Of the remaining 16, six are the technical reports of the models and the benchmark we use, which are only released as preprints, and seven are non-academic sources (model cards, regulatory documents, etc.). The last three are recent research preprints, which we cite because the field moves fast and relevant work often appears on arXiv before peer review. We have replaced arXiv entries with their peer-reviewed versions where available.
Q3: Can more experimental content move from the appendix into the main paper? (W3)
We thank the reviewer for this feedback and agree. Using the additional page, we have moved the key results of Sec. 4.4 and 4.5 into the main paper: Table 3 compares answer-only and full watermarking of reasoning models (accuracy, detectability and output length), and Table 4 shows the detectability at which our audit vs. the HealthBench rubric score first detects degradation.
Q4: How does this work differ from existing papers on watermark attacks against medical VLMs? (W4)
We thank the reviewer for this question. As clarified in W1, we do not alter model inputs (images or text), nor do we study an adversarial setting. Instead, we study the effect of watermarking on output quality. To our knowledge, the only prior work on the effect of watermarks on generated medical text is Hastuti et al. (2025), which we discuss in Sec. 2.