- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduce SeMark, a model watermark method for LVLM. The key idea is not place watermark in vision encoder with the assumption that the fine-tuning doesn't usually update it. SeMark is implemented through optimization of visual perturbation and the watermarked vision encoder weights through clean-image features preservation, target features alignments given perturbation (semantic alignment and pairwise loss). The results show high watermark success rates across models, outperforming existing methods.
Critical strengths
- The motivation of placing watermark is practical for common fine-tuning settings of LVLM
- The loss design is reasonable choices
- The evaluation covers multiple architectures, methods with strong watermark success rates.
Critical weaknesses
- Although the assumption that vision encoder is usually frozen during finetuning, this is not otherwise a strong assumption for the threat model. For example, in adaptive attack, the adversary can just do a small fine-tuning of vision encoder (maybe through some feature alignments) can erase the watermarking. The Appx also shows the evidence of this adaptive attack.
- The evaluation lacks statistical test for a reliable ownership claims. Per-query FPR is not the same as the probability of falsely accusing an entire model.
- The technique contribution is merely limited as it is strongly overlapping with [1] that induce the trigger-condition changes in generated responses. The difference is about the application: [1] studies an attack, SeMark repurposes for model ownership verification. This should be discussed in the paper.
[1] Liu, Zhaoyi, and Huan Zhang. "Stealthy backdoor attack in self-supervised learning vision encoders for large vision language models." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
Questions for authors
- The adaptive attack design can be cleaner. Since the watermarking is mostly placed in vision encoder. Can we have a specific adaptive attack target this, say, updating full visual encoder without modifying the projector with similar training budgets and learning rates?
- The paper claims semantic watermarking, however, the verification is actually obtained by target keyword matching. Please justify or soften the claims