acceptodds
Under review as a conference paper at ICLR 2027

Freeze the Weights, Not the Graph: Prompt-Retrievable Watermarks for Frozen Diffusion Models with Certified Retrieval

69 views

Abstract

Prompt-retrievable watermarking asks a text-to-image model to embed a semantic feature of the prompt in the generated image so that a verifier holding a bank of candidate prompts can later return the one that produced it. We realise this on a frozen Stable Diffusion 1.5 with a trainable token selector, latent injector and ResNet decoder around the fixed sampler, and show that two implementation defaults decide whether it works. The memory-cheap freeze, a no-grad denoise with a detached output latent, removes the only gradient into the injector and pins retrieval at chance, whereas a differentiable frozen-weight rollout with per-step gradient checkpointing keeps the weights fixed but the graph intact and lifts held-out Top-1 to 13 to 87 times chance across bank sizes. The latent clamp inherited from bounded-template watermarks saturates a third of a standard-normal latent and replaces the image with a colour field; clamp-free inference restores image content at unchanged retrieval, 0.357±0.025 Top-1 at , and turns the injection amplitude into a dial between retrieval and prompt fidelity. We state a randomized-smoothing certificate for retrieval with an explicit finite-sample ceiling and certify the model with votes over three seeds. A control suite built around the unwatermarked image from the same seed establishes that the decoder reads the watermark rather than the image, that the signal survives the training attack suite, and that near-duplicate prompts and abstention remain weaknesses. Code and checkpoints are released.

Primary area:
generative models
Keywords:
diffusion models, watermarking, prompt retrieval, frozen generators, gradient checkpointing, certified robustness, randomized smoothing, image provenance, contrastive learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zE4eOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper embeds a CLIP-derived feature of the prompt in each generated image, so that a verifier holding a bank of N candidate prompts can retrieve the right one. On a frozen Stable Diffusion 1.5, a trainable selector computes the feature, an injector writes it into the initial latent, and a ResNet-18 decoder reads it back. The paper claims four contributions. (1) It backpropagates through the frozen sampler with gradient checkpointing, unlike a detached rollout, which cuts the gradient. (2) It removes an inherited clipping step on the initial latent and uses the injection strength to trade retrieval against fidelity. (3) It states a randomized-smoothing certificate for retrieval and the finite-sample ceiling on its radius. (4) It adds control experiments. The tests use held-out banks of 16 to 512 prompts, three seeds, the training distortions and a projected gradient descent (PGD) attacker.

Critical strengths

The paper reports its own failures and controlled experiments for the obvious limitation.

Critical weaknesses
  1. In a mature field of Watermarking Diffusion models, the paper compares with no existing watermark and omits the standard attacks.

    • No existing watermark appears in any table. The stated use case is telling which commission produced an image without "storing a key per generation" (l. 40–42). RingID or Gaussian Shading, keyed by a hash of the prompt, already serves it, since the verifier holds the prompts anyway. RingID reports 0.96 identification accuracy among 128 keys on Stable Diffusion 1.5, against 0.357 here (Table 10).

    • The distortion tests use only the training distortions, at training strengths (App. E). There is no regeneration attack, although Sec. 2 cites such attacks (Zhao et al., 2024). Nor is there crop, rotation or colour correction, although the watermark is "a strong tint" (l. 305).

    • The only adversarial test is weak: 5-step PGD over the five distortion parameters (App. F). It even raises the adversarially fine-tuned model's Top-1 from 0.315 to 0.359 (Table 8). There is no pixel-space attack, no forgery test (e.g. Müller et al., CVPR 2025) and no threat model. Yet the watermark is a keyless function of the prompt, and the checkpoints are released.

  2. Contributions 1 and 2 are published or fix the authors' pipeline. Backpropagating through a frozen sampler with per-step gradient checkpointing is DRaFT (Clark et al., ICLR 2024), and OptMark (Xing et al., AAAI 2026) does it for initial-noise watermarks. Neither is cited. A detached rollout gives the injector zero gradient by construction (Sec. 3.2), so the chance-level column of Table 1 only checks the implementation. The clipping step is "the configuration we inherited" (l. 186). The task is close to TokenTrace (cited) and SEAL (Arabi et al., ICCV 2025, not cited), which also ties the initial noise to the content.

  3. The formal results are borrowed, and the method itself has no theory. Theorem 1 restates Theorem 1 of Cohen et al. (2019) for a special case (l. 206–207). The paper claims the ceiling as contribution 3, but Cohen et al. derive and plot it (their Sec. 3.2.2 and Fig. 5). Jiang et al. (ECCV 2024), (not cited), already smooth a watermark decoder, with certificates against removal and forgery. In this work, nothing explains why, or how much, prompt information survives the sampler.

  4. The certified radius is too small to matter. A radius of 0.400.40 at noise level σ=0.25\sigma = 0.25 is a per-pixel root-mean-square change of 9×10−49 \times 10^{-4} (l. 356–357). Rounding a 256×256256 \times 256 RGB image to 88 bits alone changes it by about 0.500.50 in ℓ2\ell_2 norm. Certification is never run on unwatermarked images, where a verifier that cannot tell when to return no answer may certify a wrong prompt.

  5. The paper is poorly presented and very hard to read.

Questions for authors
  1. On the same banks of 128 and 512 prompts, how does RingID or Gaussian Shading keyed by a hash of the prompt compare? Please also report the method's detection TPR at 1% FPR. Matching the keyed baseline on exact prompts, or beating it on paraphrased ones, would resolve this concern.

  2. What is Top-1 at N = 128 under diffusion regeneration (Zhao et al., 2024) and under the imprinting forgery of (Müller et al. 2025)?

  3. For what fraction of unwatermarked images does certification return a prompt, and at what radius? A fraction near zero would show the certificate at least does not certify false attributions.

  4. Why does the base model's clean Top-1 differ between tables? Table 8 gives 0.396±0.0300.396 \pm 0.030 and Tables 2–4 give 0.357±0.0250.357 \pm 0.025, under the same checkpoint and protocol (Table 13). Without an answer, I cannot tell which runs back the other tables.

Reviewer_fDmgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes PRW (Prompt-Retrievable Watermarking) for image synthesis. Getting the prompt, a learnable injector makes a perturbation of the latent, and then the (frozen) Stable Diffusion 1.5 generate images via DDIM, then a learnable retriever (This paper calls it 'Decoder') reconstructs the initial prompt. Main contributions are,

  1. Do not detach the latents for DDIM sampling (in training stage)
  2. Do not clamp the latent because it distorts the watermark.
  3. The proposed framework is mathematically proved.
  4. Experimentally verified that the watermark detection really reads (interprets) wtaermarks, not the contents of the images.
Critical strengths

None

Critical weaknesses

I don't think 'Do not detach the latents' is a contribution. Most of the researchers would naturally do that if needed.

In figure 2, the quality of the output is not good (even in the first row). Colors are saturated, and there seems no semantic meaning in most outputs (even though counting on SD v1.5). Therefore, it is hard to believe that this framework produces useful outputs. Maybe the author may examine their idea with newer version of Diffusion Models (or flow matching models).

It is difficult to understand Theorem 1, because there are almost no relevant things in the other parts of the paper. Section 3.4 is not harmonized with the paper.

Questions for authors

Is there a reason why the line breaks in the title does not seem natural?

Reviewer_owg1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies prompt-retrievable watermarking for frozen text-to-image diffusion models. The proposed framework uses a trainable prompt selector, latent injector, and image decoder with Stable Diffusion 1.5. The authors identify two implementation issues: detaching the denoising graph prevents the injector from learning, while clipping the initial latent can damage image quality. They address these issues by preserving gradient flow and removing latent clipping. The paper also examines the trade-off between retrieval accuracy and image quality, and applies randomized smoothing to certify retrieval robustness. Experiments show improved retrieval accuracy but also reveal limitations in handling similar prompts and absent prompts.

Critical strengths
  1. The authors provide a relatively transparent discussion of the method's limitations, particularly regarding image fidelity, hard-negative prompts, and abstention.
  2. The paper is well organized.
Critical weaknesses

1. Limited technical novelty. The main contributions largely build on established techniques rather than introducing fundamentally new methodological ideas. Maintaining gradients through a frozen model is standard practice in differentiable optimization, while avoiding aggressive clipping of Gaussian noise is a relatively straightforward implementation correction. Similarly, the certification procedure directly follows conventional randomized smoothing. Although these observations are practically useful, their significance as independent research contributions remains unclear. I encourage the authors to better establish the novelty of the framework relative to recent watermarking approaches, particularly TokenTrace [1*], and clarify what generalizable insights extend beyond these implementation issues.

2. Unclear practical advantage over content-based retrieval. The proposed method introduces a noticeable trade-off between retrieval accuracy and image fidelity. In Table 3, reducing the injection amplitude to λ\lambda=0.045 preserves the CLIP score but decreases Top-1 retrieval to 0.109, while stronger injection substantially affects image content. More importantly, the λ\lambda=0 control uses a decoder trained with watermarked images, which is not necessarily a competitive content-only baseline. I suggest training a comparable decoder directly on unwatermarked images and evaluating whether the watermark provides additional retrieval benefits at matched image quality. Additional perceptual evaluations would also help establish the practical significance of the reported trade-off.

3. Limited practical significance of the robustness guarantees. The certified retrieval results follow directly from standard randomized smoothing, and the resulting certified radii are quite small in the pixel space. As acknowledged in Section 4.4, an ℓ2\ell_2 radius of 0.40 corresponds to a per-pixel RMS perturbation of approximately 9×10−49×10^{-4}. Such guarantees provide limited insight into robustness against realistic watermark-removal attacks. Furthermore, the empirical evaluation mainly considers transformations already represented in the training attack suite. I suggest evaluating stronger, adaptive, and image-quality-preserving attacks, following established watermark robustness benchmarks such as WAVES [2*] and adaptive attack studies [3*].

4. Limited reliability for realistic prompt attribution. The current evaluation reveals substantial difficulties when candidate prompts are semantically similar or the correct prompt is absent. In Table 6, Top-1 accuracy drops to 0.096 for the hard-negative bank, while the absent-prompt evaluation achieves an AUROC of only 0.545. Although the authors acknowledge these limitations, they substantially restrict the applicability of the proposed framework to practical provenance verification. I suggest further investigating realistic candidate-bank construction, reporting false-attribution rates under calibrated rejection thresholds, and clarifying the intended deployment assumptions. The evaluation on a single diffusion backbone at 256×256 resolution also leaves generalization to more capable generators uncertain.

References

[1*] Zhang et al., TokenTrace: Multi-Concept Attribution through Watermarked Token Recovery. CVPR 2026.

[2*] An et al., WAVES: Benchmarking the Robustness of Image Watermarks. ICML 2024.

[3*] Lukas et al., Leveraging Optimization for Adaptive Attacks on Image Watermarks. ICLR 2024.

Questions for authors
  1. Could the authors clarify the practical advantage over content-only retrieval? In particular, how does PRW compare with a decoder trained directly on unwatermarked images under a matched training budget and comparable image quality? This would help establish whether the learned watermark provides meaningful benefits beyond information already present in the generated image.
  2. How robust is the method against adaptive watermark-removal attacks? For instance, can targeted color correction, diffusion-based regeneration, or optimization against the learned decoder substantially reduce retrieval accuracy while preserving recognizable image content?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.