acceptodds
Withdrawn from ICLR 2027

Hidden in the Features: Semantic Backdoor Watermarking for Large Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) have become highly valuable intellectual property, making them prime targets for unauthorized copying and downstream reuse. Backdoor watermarking offers a promising route to copyright protection; however, existing LVLM watermarks typically encode ownership signals as specific output patterns or token-level statistics. Such signals produce distribution-shifted responses that are easy to detect, and their effectiveness can be weakened by downstream adaptation. In this work, we present SeMark, a stealthy semantic backdoor watermarking method for visual encoders in LVLMs. Specifically, SeMark employs a bounded universal perturbation as an imperceptible trigger and trains the watermarked encoder to steer triggered representations toward a target semantic neighborhood in the visual feature space. The watermark is embedded into the visual encoder via a bilevel optimization that rigorously balances feature alignment, clean representation preservation, and pairwise similarity. This ensures that triggered features approach the target semantic region, while clean representations remain close to those of the original encoder. Ownership is then verified via black-box queries by assessing whether private triggered samples elicit target-consistent responses from a suspect model. Extensive experiments demonstrate the effectiveness of SeMark, as well as its robustness against downstream fine-tuning while preserving clean-task performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Model Watermarking, Large Vision-Language Models, Copyright Protection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qq81OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce SeMark, a model watermark method for LVLM. The key idea is not place watermark in vision encoder with the assumption that the fine-tuning doesn't usually update it. SeMark is implemented through optimization of visual perturbation and the watermarked vision encoder weights through clean-image features preservation, target features alignments given perturbation (semantic alignment and pairwise loss). The results show high watermark success rates across models, outperforming existing methods.

Critical strengths
  • The motivation of placing watermark is practical for common fine-tuning settings of LVLM
  • The loss design is reasonable choices
  • The evaluation covers multiple architectures, methods with strong watermark success rates.
Critical weaknesses
  • Although the assumption that vision encoder is usually frozen during finetuning, this is not otherwise a strong assumption for the threat model. For example, in adaptive attack, the adversary can just do a small fine-tuning of vision encoder (maybe through some feature alignments) can erase the watermarking. The Appx also shows the evidence of this adaptive attack.
  • The evaluation lacks statistical test for a reliable ownership claims. Per-query FPR is not the same as the probability of falsely accusing an entire model.
  • The technique contribution is merely limited as it is strongly overlapping with [1] that induce the trigger-condition changes in generated responses. The difference is about the application: [1] studies an attack, SeMark repurposes for model ownership verification. This should be discussed in the paper.

[1] Liu, Zhaoyi, and Huan Zhang. "Stealthy backdoor attack in self-supervised learning vision encoders for large vision language models." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Questions for authors
  • The adaptive attack design can be cleaner. Since the watermarking is mostly placed in vision encoder. Can we have a specific adaptive attack target this, say, updating full visual encoder without modifying the projector with similar training budgets and learning rates?
  • The paper claims semantic watermarking, however, the verification is actually obtained by target keyword matching. Please justify or soften the claims
Reviewer_PyppOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a sematic backdoor watermarking method for Large Vision-Language Models (LVLMs). Different from works that encode ownership signals as output patterns or token-level statistics, the proposed SeMark method embeds watermark into the visual encoder. The encoder is tuned together with a private universal perturbation, under three coupled objects. Experiments show that SeMark achieves highly reliable ownership verification. Analyses and ablation studies are provided.

Critical strengths
  • The paper is studying a practical task. Given that LVLMs are widely used as foundation models in various tasks, protecting the copyright via watermarking is important, especially when the models can be finetuned in downstream tasks.
  • The idea of embedding a watermark in the visual encoder instead of relying on the output patterns or token-level statistics is reasonable. The paper validates this with high watermark success rates in the experiments.
Critical weaknesses
  • While the application of visual feature perturbation for this LVLMs copyright protection task sounds promising, the algorithm itself seems to be a collection of existing ideas on feature-spaced backdoor learning. Conceptually, it learns the universal perturbation and visual feature parameters via contrastive/similarity learning. The paper does not sufficiently explain how the proposed losses are unique to this task, therefore establishing the technical innovation.

  • The method highlights semantic backdoor watermarking. However, the correlation between feature similarities and semantic similarities is not quite straightforward. In the watermark verification, target keyword matching is used as the success rate metric. This cannot evaluate the cases where the responses are semantically identical but do not contain the target keywords.

Questions for authors
  1. Clarify how the proposed losses (3-5) are unique to this watermark task and different from existing feature-space backdoor attack.
  2. Explain the relationship between semantics and target keyword matching, and how the proposed method preserves the semantic metrics. Is there a contraction that the response is semantically similar but with no target keyword in the real experiments? Or, wouldn't this be a concern.
Reviewer_2yBfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SeMark, a semantic backdoor watermarking method for ownership verification of large vision-language models (LVLMs). It jointly optimizes a universal perturbation and the visual encoder to shift triggered features toward a target semantic region while preserving normal model behavior. Ownership is verified through black-box queries by checking for target keywords in generated responses. Experiments on LLaVA-1.5-7B and MiniGPT-4 show high watermark success rates after downstream fine-tuning, limited impact on clean-task performance, and improved robustness over several existing watermarking methods.

Critical strengths
  1. This paper proposes a semantic watermarking method that enables black box ownership verification using only the generated text.
  2. Experiments on two LVLMs and five downstream datasets show high watermark success rates after downstream fine tuning.
Critical weaknesses
  1. Insufficient novelty compared with existing semantic backdoor methods. This paper lacks discussion and comparison with closely related semantic backdoor methods, such as BadVision [1]. Both methods use universal perturbations and visual feature manipulation to induce target semantic responses while preserving normal model behavior. The motivation and advantages of the additional optimization objectives introduced by SeMark over existing methods remain unclear.

  2. The stealthiness claim lacks direct experimental support. This paper claims that mapping triggered visual features to a target semantic region produces more natural and less detectable responses, but the experiments mainly measure whether the target keyword appears in the generated text. Figure 5 shows that the triggered responses describe buses even when the original images contain a cat or a snowboarder, leading to clear conflicts with the visual content.

  3. Lack of systematic taxonomy of existing watermarking methods. This paper does not clearly distinguish existing LVLM watermarking methods in terms of their deployment requirements and verification settings. For example, SeMark requires modifying the visual encoder before model release, while PLA and SIF can construct fingerprints without modifying the original model parameters. These differences make it difficult to fully understand the practical advantages and tradeoffs of SeMark compared with existing methods.

  4. Unclear evaluation metrics. Although WSR is defined in Eq. (9), the definitions and calculation procedures of several other metrics are not sufficiently clear. For example, Table 4 uses FMR without explaining its relationship to WSR, while Table 2 reports only relative performance changes without showing the original scores or whether performance increases or decreases.

  5. Limited robustness under common image transformations. Although SeMark outperforms the baselines in Table 4, its FMR drops to 0.28 under JPEG compression and 0.57 under image resizing. These results suggest that common image processing operations can significantly weaken the watermark signal, which limits its reliability in practical deployment.

Questions for authors
  1. What are the main technical differences between SeMark and BadVision [1], and can the additional optimization objectives of SeMark demonstrate clear advantages under the same evaluation setting?

  2. How does SeMark avoid detection by Semantic-Divergence Attack (SDA) when the triggered responses are clearly inconsistent with the original images, as shown in Figure 5?

  3. How is FMR calculated in Table 4, and why does it differ from WSR used in the main experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.