acceptodds
Withdrawn from ICLR 2027

Beyond Inversion: Source-Anchored Forgery Attacks on Semantic Watermarks for Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Semantic watermarking encodes provenance signals in the initial noise of diffusion models without modifying model weights, enabling source attribution that is robust to common image transformations. Yet, malicious semantic edits can preserve these signals, allowing manipulated content to retain attribution to the original generator or user. Inversion-based forgery attacks first estimate an initial noise latent in an attacker-controlled model and then regenerate the image under a target prompt. Watermark retention along this path, however, is sensitive to mismatch between the attacker and victim models. We propose an inversion-free, source-anchored forgery attack that bypasses this reconstruction step. Using the observed image's latent encoding as a fixed anchor, the attack integrates differences between velocity predictions at paired source and target states to perform prompt-guided semantic manipulation. It requires neither the source prompt nor queries to the victim generator or watermark verifier. Experiments across five victim generators and multiple semantic watermark methods demonstrate consistently higher watermark retention than inversion-based baselines. The attack also produces semantic forgeries that pass watermark verification under a defense designed to bind watermarks to image semantics. These results expose a gap between robust source attribution and protection against semantic forgery.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
image watemarking, diffusion models, semantic watemark, forgery attack

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JdfrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a forgery attack that passes the verification of the original watermark even after semantic editing. It analyzes the model mismatch issue in existing inversion-based attacks and applies an inversion-free editing procedure anchored to the source image. GS is evaluated across five victim models, PRC and Spherical across four models, and results on SemBind are also presented.

Critical strengths
  1. It experimentally demonstrates the limitations of inversion-based attacks caused by model mismatch. It reports higher retention performance compared to baseline attacks across multiple victims and watermarks, providing analyses of victim-side latent drift and BitAcc loss. This is useful for understanding the impact of attacker model selection on security evaluations.

  2. It shows that semantic editing and passing watermark verification coexist even in SemBind. Extending the evaluation to multiple base watermarks is also meaningful. As the paper acknowledges, this vulnerability is not unique to the proposed attack.

Critical weaknesses
  1. The contributions are not sufficiently distinguished from existing methods and prior observations. The core update in Algorithm 1 coincides with FlowEdit, and key FLUX settings are identical. While FlowEdit is cited, methodological differences beyond simple application remain unclear. Furthermore, The Semantic Imprinting Hypothesis: How Semantic Watermarks Survive Prompt-based Editing has already reported that high detection and user attribution performance of certain semantic watermarks are retained even after InfEdit. Therefore, the paper needs to disentangle its additional contributions—namely the forgery attack, model mismatch, and SemBind analysis—from existing observations regarding watermark preservation after inversion-free editing.

  2. The superior watermark retention is not sufficiently disentangled from differences in editing extent. The proposed method exhibits lower CLIP-T and higher source fidelity than baseline attacks, a trade-off that the authors also acknowledge. While qualitative examples are provided, there is no comparison aligned by target edit success rate or editing extent, making it ambiguous whether the higher Det/Trace signifies a more effective forgery. Although style changes are included in the evaluation, the absence of category-specific results makes it difficult to assess performance under global modifications. Additionally, while the phenomenon where a larger RMS drift coincides with smaller BitAcc loss is discussed, the reason why the proposed trajectory preserves favorable verification directions is not directly verified.

Questions for authors
  1. What novel elements are introduced in Algorithm 1 compared to FlowEdit? Please clarify the scope of application of the existing algorithm and explain the added contributions relative to the observations of preservation after inversion-free editing in the aforementioned workshop paper.

  2. Can you report both editing success rate and watermark retention while varying parameters such as guidance and the number of update steps? Specifically, it would be informative to see whether the Det/Trace advantage persists at comparable levels of editing success, and what proportion of cases satisfies both the target edit and successful verification.

  3. Can you provide concrete examples of the included style changes (type 9) along with category-specific performance? While the COCO-to-I2P comparison in SemBind is noted, the boundary at which verification fails depending on the edit scope and strength remains unclear. Results from sequential editing would also be helpful.

  4. What is the rationale for excluding frequency-based semantic watermarks? It is necessary to verify the generalizability of the conclusions through comparisons with other representative embedding schemes and additional editing/attacks under the same cover-free setting. Is it also feasible to include comparisons using SDEdit as well as more accurate inversion techniques?

  5. Beyond the analyses in Chapter 4.5 and Appendix D.5, can you provide direct evidence linking the discrepancy between RMS drift and BitAcc loss to the verifier’s sensitivity and displacement direction? Please also clarify the extent to which the controlled residual comparison in the current theory explains the observed differences across actual attacks.

Reviewer_Kp3hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new method for the forgery of inversion-based watermarks for diffusion model generated images. Using a proxy model, the method enables targeted edits of images generated by a victim model with source prompts towards target prompts (both taken from the PIE-Bench dataset), which still verify as watermarked. The method does not rely on full inversion like a previous forgery attack (Reprompting). Instead, it uses flow-based editing with a proxy model (FLUX model) in order to move an encoded latent towards a target prompt’s semantic while anchoring to randomly sampled noise each step. The method is evaluated on Gaussian Shading and SemBind-protected Gaussian Shading. In both evaluations, the proposed method outperforms reprompting. In the appendix, further watermarking schemes are evaluated as well, namely spherical watermarks and PRC.

Critical strengths

S1: The method shows consistently high success rates across different victim models, which is a clear advantage over the baseline method (Reprompting). Furthermore, the qualitative examples clearly show that the edits preserve more of the original content of the watermarked reference image, enabling more targeted edits and hence, arguably more realistic use cases than the Reprompting baseline method.

S2: the question of whether image editing preserves the original watermark is interesting and the application of FlowEdit here is novel, to the best of my knowledge.

Critical weaknesses

W1: Clarify the scope of the forgery claim. At first, it appears to be targeting forgery in general, where a completely unrelated image can be generated that fools a watermark (which is the case for reprompting). However, this paper only considers edits to an existing watermarked image, which is more limited in scope, and uses the image editing tasks from the PIE-Bench dataset for evaluation.

W2: PIE-Bench’s source and target prompts shown as examples in the paper are very similar to each other, differing only in a few words between source and target prompt. While the proposed attack method itself does not require the source prompt, the closeness between source and target prompt may skew the results. In a realistic setting, there could be a bigger mismatch, which may affect the attack’s effectiveness. This could also affect the SemBind evaluation (Table 2): keeping source and target prompt similar can aid in bypassing the semantic check in SemBind. I believe the authors need to evaluate the attack with a target prompt automatically generated from the image, or use a rephrased version of the target prompt from PIE-Bench, which would also help understand robustness to prompt formulation. An automatic image caption or rephrased source prompt could also be used as a source prompt in the attack, which is currently missing this ablation.

W3: as it is written, the paper does not make clear to what extent the proposed attack follows the original approach in FlowEdit. In my current understanding, it appears to be identical, with the difference that an empty source prompt is used in the attack. If this is the case, I think the approach has to be described in a background section as something originally proposed by FlowEdit, and the approach should describe the differences and improvements made by the authors of this paper.

Questions for authors

Q1) Could the similarity between source and target prompt in the PIE dataset have affected the evaluation favourably? How does the method perform when the source prompt is truly unknown to the attacker? This is the central potential weak point and it should be addressed in order to improve the score.

Q2) Can the authors please explain how the proposed method for editing with source anchoring differs from flow-based editing? While the paper briefly states that flow-based editing is used as inspiration (line 196), the differences do not seem to be explained clearly.

Q3) It is unclear what is meant by watermark pattern mixing (depicted in Fig. 2c and mentioned a few times in the text). This point is not made sufficiently clear and there is very little text to explain it.

Q4) Have you compared your method to inpainting with pixel replacement outside the mask, so that large portions of the image are preserved? For Gaussian Shading, this could be highly effective, since it is very robust against dropping of image regions (compare Fig. 6d) in the original paper).

Reviewer_n7QAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates black-box forgery attacks against semantic watermarks in diffusion models. Existing attacks typically rely on a surrogate model to invert a watermarked image into its initial noise and subsequently regenerate the image under a target prompt. When the surrogate model differs from the original generator, this process may introduce additional reconstruction errors that compromise the embedded watermark information. To address this issue, the authors propose an inversion-free watermark forgery method that uses the clean VAE-encoded latent as a fixed anchor. By constructing source and target states with shared noise and accumulating their velocity-field differences under different text conditions, the method performs semantic editing while reducing reconstruction errors caused by model mismatch, thereby improving forgery effectiveness.

Critical strengths
  1. The paper highlights an important security gap between watermark-based source traceability and image semantic integrity.
  2. The proposed attack requires no access to the victim model, watermark keys, original prompts, or detector queries.
  3. The experiments cover five generative models, three semantic watermarking methods, and the SemBind defense, complemented by an analysis of watermark evidence drift.
Critical weaknesses
  1. The core design closely resembles FlowEdit [1] (ICCV 2025), including its figures, mathematical formulations, and algorithmic procedure. Moreover, the authors exclusively use FLUX.1-dev as the surrogate model, without evaluating other flow-matching or conventional diffusion-based surrogates. This limits the evidence for generalization across different surrogate architectures.

  2. The proposed method tends to preserve the original image and is primarily evaluated on localized edits. Its CLIP-T scores are lower than those of the baselines, and several qualitative examples fail to fully achieve the intended semantic modifications. These results suggest a potential trade-off between source-image preservation and semantic editing strength, which has not been adequately investigated. In contrast, prior work on Black-Box Forgery [2] (CVPR 2025) demonstrates substantially larger semantic modifications.

  3. In Table 1, SD2.1-based inversion performs relatively well on SD1.5 and SD2.1, which is expected given their shared conventional diffusion architecture. However, SDXL also belongs to this category but exhibits substantially worse performance, which requires further explanation. Conversely, FLUX.1-dev-based inversion achieves strong results on SD1.5 and SD2.1, despite the architectural mismatch that should theoretically lead to poor performance under the authors' hypothesis. These observations weaken the empirical support for the proposed Inversion Bottleneck.

  4. Although the proposed method avoids the additional inversion–regeneration stage, it does not eliminate reconstruction errors introduced by the surrogate VAE. More importantly, the theoretical analysis does not establish that removing the inversion residual necessarily improves watermark preservation under different editing displacements. The individual contributions of VAE reconstruction error, inversion residuals, and editing displacement are not experimentally isolated. Moreover, the method requires multiple steps of dual-branch velocity-field computation, yet no computational cost comparison is provided.

Questions for authors
  1. What are the fundamental algorithmic differences between the proposed method and FlowEdit [1] (ICCV 2025)? Beyond their different application scenarios, how do their core methodological contributions differ?

  2. Does the proposed method retain its advantage in watermark preservation when compared at the same semantic editing success rate or editing strength? Can the authors report the joint attack success rate and evaluate more substantial semantic modifications?

  3. How do the authors explain the substantial performance degradation of SD2.1-Inv on SDXL and the relatively strong performance of FLUX-Inv on SD1.5/SD2.1? What factors actually contribute to the proposed Inversion Bottleneck? The current experimental evidence appears insufficient to substantiate this hypothesis.

  4. Can the authors quantitatively disentangle the effects of VAE reconstruction error, inversion residuals, and editing displacement on watermark preservation? Additionally, can they provide a systematic comparison of inference time and computational cost across methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.