- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Motivation and claims:
-
Regeneration attacks work well for removing watermarks but have inconsistent effectiveness for latent-domain watermarks.
-
Paper theoretically explains why diffusion regeneration fails against latent-domain watermarks.
-
Paper proposes VARR, VAR-based regeneration that breaks the generation-inversion symmetry.
-
Why diffusion regeneration (add a bit of noise, denoise) can not remove latent-domain watermarks:
- Some upper bounds on zT reconstruction error after attack are given, as well as detector score.
- Conclusion is that magnitude of the injected noise is not important but the reconstruction error of attacker’s VAE and denoiser.
Approach:
-
Threat model is no-box (no API queries), goal is to remove watermark
-
Attack consists of encoding the watermarked image into a scale pyramid, then dropping the last m scales (at highest resolutions) and re-generating them using the VAR backbone. Final image is the VAR decoder output.
Results:
-
Wide selection of attacked watermarks, including PRC, GS, TreeRing, RingID, Stable Signature, AquaLoRA. Evaluation could be extended with TrustMark and PixelSeal as baselines as well.
-
Attack baselines: simple perturbations, UnMarker, VAE regeneration, diffusion regeneration, next-frame prediction attack and CtrlRegen.
-
5k images are watermarked for every method for each (of the two) datasets.
-
Only last 2 (out of 10) scales are re-generated. Note that the last m=2 scales are roughly 70% of total tokens during generation.
-
Evaluation metrics: removal success and PSNR, SSIM and FID (and CLIP for alignment).
-
The methods shows strong removal results, on par with UnMarker, except for AquaLoRA and TreeRing, where it is better, and for HSTR and Gaussian shading where UnMarker is better. However, it performs better at image fidelity than UnMarker, with high PSNR, and better FID. However, CtrlRegen has better PSNR and SSIM. Speed-wise, VARR is 200x as fast as UnMarker.
-
The method reports ~93% removal success across the tested latent-domain watermarks, but only ~60% for Gaussian Shading.
Critical strengths
S1: mostly thorough and reasonable evaluation, reasonable ablations (method is very simple so there isn’t much to ablate)
S2: simple training-free approach that should be easy to implement, no per-image optimization or detector access
S3: effective removal performance across tested watermarks, while being 200x faster and having slightly better image preservation than UnMarker
Critical weaknesses
W1: not clear how significant or practically consequential the theoretical analysis is. There is no empirical analysis to investigate theoretical claims in Section 3. Section 3 establishes that the detection score difference between the original watermarked image and the attacked image is upper-bounded (Eq. 3). Eq. 4 then concludes that the watermark remains detectable if the upper bound on detection score change is below the original image’s detection margin \gamma. However, it is never established whether the upper bound tends to be smaller than the margin so it’s not clear whether this theoretical treatment explains the robustness to diffusion regeneration.
W2: missing control and ablations. a) m=0 needs to be included as a control on how much using a different autoencoder and tokenizer already disrupts the watermark (quantization error in tokenization could have a substantial effect on watermark detectability) and b) m=1 is missing from ablations.
Other important weaknesses:
W3: BitMark is not included in the experiments. It is a watermarking method specifically designed for bit-wise generators (including infinity, VAR’s successor). It is very robust and works on the same generator family as VARR. I believe before this paper is accepted, BitMark needs to be included in the experiments to understand how the attack works against watermarks on other scale-autoregressive generators. It would also be interesting to see how stable the coarser scales in BitMark are when the finer scales are attacked by VARR.
W4: attack success metric has issues
a) it is unclear how the removal success thresholds have been calibrated.
b) it is unclear how separable attacked images are from unwatermarked images. It could be that even though the chosen thresholds are defeated, other thresholds still allow to separate attacked watermarked from unwatermarked (attacked or clean) images. For this reason, TPR at a fixed FPR should be reported.
W5: Limited novelty. While simple, the presented approach is a straightforward adaptation of an existing scale-autoregressive generative model to use in a regeneration attack.
Questions for authors
Q1: i wonder how much sense it makes to regenerate earlier scales while keeping the other scales fixed (in ablations). How bad are the image artifacts there? Could you include examples in appendix?
Remarks:
- Citations should be formatted properly. Use \citep
- By latent-domain watermarks, the authors assume watermarks relying on full inversion (treering-style)
- RAVEN (Erasing Invisible Watermarks via Novel View Synthesis) at least be cited. It’s similar to NFPA.
- Infinity and BitMark should be cited.
2 comments
Response to Reviewer 4
For W4: (b) In the revised manuscript, we have added a Separability Analysis of Attacked and Unwatermarked Images section to evaluate the separability between VARR-attacked images and unwatermarked images using AUROC. The results show that the AUROC values for all watermarking methods except Gaussian Shading are close to 0.5, indicating that VARR-attacked images are nearly indistinguishable from unwatermarked images. These findings further support that VARR effectively removes watermark evidence and that simply adjusting the detection threshold is unlikely to substantially improve the separability between attacked and unwatermarked images.
Table3: Separability analysis of attacked and unwatermarked images.
| Method | Unwatermarked Bit Acc (%) | Attacked Bit Acc (%) | TPR (%) | AUROC |
|---|---|---|---|---|
| Tree-Ring | N/A | N/A | 1.40 | 0.518 |
| ROBIN | N/A | N/A | 1.20 | 0.509 |
| HSTR | N/A | N/A | 12.40 | 0.604 |
| GaussMarker | 2.60 | 0.531 | ||
| PRC | 1.00 | 0.504 | ||
| Gaussian Shading | 42.60 | 0.742 |
For W5: We acknowledge that VARR directly applies an existing autoregressive model to regeneration attacks. However, this does not necessarily imply a lack of novelty. Prior work [1] also directly applied diffusion and VAE models to regeneration attacks without modifying anything, yet was accepted at NeurIPS. Furthermore, we emphasize that our contributions lie in the theoretical analysis of why diffusion regeneration attacks fail against latent-domain watermarks, the simultaneous improvements in watermark removal success rate, visual fidelity, and computational efficiency, and the insights into why VARR is effective, including the advantages of the VAR domain for watermark removal and VAR's strong image generation prior for preserving visual fidelity. Together, these contributions establish VARR as a state-of-the-art regeneration attack and reveal a significant threat to existing diffusion model watermarks.
[1] Xuandong Zhao, Kexun Zhang, Zihao Su, Saastha Vasan, Ilya Grishchenko, Christopher Kruegel, Giovanni Vigna, Yu-Xiang Wang, and Lei Li. Invisible image watermarks are provably removable using generative ai. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 8643–8672. Curran Associates, Inc., 2024. doi: 10.52202/079017-0276.
For Q1: We would like to clarify that regenerating earlier scales while keeping subsequent scales fixed is not a reasonable setting, as VAR follows a coarse-to-fine autoregressive process, where each scale is generated based on the preceding scales. Modifying earlier scales without updating subsequent scales would break the dependencies between scales and potentially introduce significant visual artifacts.
For Remarks: We believe that focusing on watermarks relying on full inversion is reasonable, as existing latent-domain watermarking methods that embed watermarks into the initial Gaussian noise rely on inversion for watermark detection. Additionally, we have added the suggested references, including RAVEN, Infinity, and BitMark in the revised manuscript.
We welcome any further questions from the reviewer and would be pleased to provide additional clarifications in further discussion.
Response to Reviewer 4
We thank the reviewer for the helpful suggestions.
For W1: Due to the use of Lipschitz continuity in our theoretical derivation, it is difficult to empirically evaluate the tightness of the derived upper bound. We emphasize that our theoretical analysis primarily aims to provide an insight into why diffusion regeneration fails against latent-domain watermarks. Previous studies generally attributed this robustness to the semantic nature of latent-domain watermarks, arguing that diffusion regeneration mainly removes noise-like information. In contrast, our analysis provides a more fundamental explanation by identifying generation–inversion symmetry as the underlying mechanism responsible for this robustness.
For W2: We have added the corresponding experiments in the revised ablation study section. Overall, the parameter (m) controls the trade-off between watermark removal success rate and visual fidelity. Increasing (m) improves the watermark removal success rate but leads to greater degradation in visual fidelity.
Table1: Ablation study on the number of regenerated scales (m) for watermark removal and visual fidelity.
(a) m = 0
(b) m = 1
For W3: We emphasize in the title that the scope of our proposed VARR attack is limited to diffusion model watermarks, whereas BitMark is specifically designed for bit-wise generators and thus falls outside the scope of this paper. We leave the evaluation of VARR's effectiveness against watermarks in autoregressive generative models for future work.
For W4: (a) The detection thresholds used to determine watermark removal success are calibrated at a fixed false-positive rate (FPR), assuming that the decoded bits from unwatermarked images follow a binomial distribution, consistent with the settings adopted in the original watermarking papers. To further improve the clarity and reproducibility of our experimental results, we additionally report the watermark detection configurations and unattacked watermark detection rates in the revised manuscript.
Table2: Watermark Detection Configurations.