- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to detect watermarks in images generated by autoregressive models. Current watermarks for these models, like WMAR and ClusterMark, bias the generator toward a secret set of tokens. To check for the watermark, the detector tokenizes the image and statistically tests the token distribution. The paper shows that after edits like JPEG compression, some token positions are reliably recovered and others are not. It then proposes RAV, which is a small network that predicts how likely each token is to be recovered correctly, and gives more weight to the more reliable tokens during detection. On three ImageNet generators, RAV improves detection by a few percentage points in most settings.
Critical strengths
- The method is simple and cheap. It changes only the detector, needs no secret key to compute the weights, works with different watermarks, and adds less than 20 ms per image.
- The detection improvement transfers to neural compression, which was not used during training. On Taming, RAV outperforms a simple weighting for all six compression models (Table 11).
Critical weaknesses
- The learned network improves only modestly on simpler baselines. Weighting tokens by reliability is already used in text watermarking (SWEET [1], EWD [2]), and simpler weightings that need no extra training get much of RAV's gain (Tables 2 and 4).
- RAV does not fix the cases where these watermarks cannot be detected reliably. After a 15-degree rotation, it detects only 3.0% of watermarked images at a 1% false-positive rate (Table 8), which the authors acknowledge as a limitation. The paper also cites forgery attacks but does not test against them.
References
[1] Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. Who Wrote this Code? Watermarking for Code Generation. ACL 2024.
[2] Yijian Lu, Aiwei Liu, Dianzhi Yu, Jingjing Li, and Irwin King. An Entropy-based Text Watermarking Detection Method. ACL 2024.
Questions for authors
- On ClusterMark, is RAV's gain over simple confidence weighting statistically significant?
- What is the false-positive rate on unwatermarked images from other sources, such as real photos or other generators? The weights depend on image content, so this rate could differ.
- All experiments use models that generate 256×256 ImageNet images from a class label. Does RAV also improve detection on a text-to-image autoregressive model?
- The paper shows that RAV transfers to neural compression, but it does not say whether the removal attacks it tests were used in training. How much of RAV's gain remains on unseen removal attacks?
3 comments
Thank you for the response and the new experiments. However, the learned predictor is only slightly better than simple weighting without training, and in some settings, the gain is not significant. Therefore, I keep my score.
Response to Reviewer 8DpN
We thank the reviewer for the follow-up feedback. We would like to further clarify that RAV's main contribution goes beyond the incremental gain of the learned CNN. Our primary contribution is a reliability-aware, non-uniform evidence-weighting framework for autoregressive image watermark verification, which explicitly accounts for the unequal survival reliability of recovered symbols.
Compared with the original WMAR detector (NeurIPS 2025), RAV improves TPR by 2.17–3.22 percentage points (pp) at 1% FPR and 3.07–4.56 pp at 0.1% FPR across RAR-XL and TAMING. For ClusterMark (CVPR 2026), using the same frozen Cluster Predictor, RAV improves six-attack macro TPR over hard, uniform-weighted verification by 6.34 pp on RAR-XL and 6.48 pp on GPT-B at the empirical 1% FPR operating point. These results demonstrate the benefits of reliability-aware evidence aggregation across two established watermarking schemes.
Beyond this framework-level contribution, we further explore learned spatial reliability prediction. Although training-free weighting captures much of the benefit, RAV achieves an average additional gain of 1.04 pp over Margin/Confidence across four generator-watermark settings at nominal 1% FPR, with statistically significant improvements in three of the four settings.
We believe our contribution lies in both demonstrating the value of reliability-aware verification and exploring the potential of learned reliability estimation. We hope this framework can inspire further research on more effective reliability prediction and watermark evidence aggregation.
Response to Reviewer 8DpN
Question 1:
Thank you for this valuable question. We added image-paired significance analyses comparing RAV with simple confidence weighting on ClusterMark. RAV improves TPR on both RAR-XL and GPT-B; the gain is statistically significant on RAR-XL but not on GPT-B. The smaller GPT-B gain may reflect the strong Confidence baseline and limited headroom.
Specifically, we performed 5,000 image-paired bootstrap resamples across six attack groups. On ClusterMark/RAR-XL, RAV improves TPR over Confidence by 1.22 percentage points (pp), with a 95% CI of [0.35, 2.00], excluding zero. On ClusterMark/GPT-B, the gain is 0.46 pp (95% CI [-0.23, 0.70]), including zero. A possible explanation is that Confidence already reaches 96.55% six-group macro TPR on GPT-B, leaving limited headroom for RAV's reliability-prediction CNN, versus 91.68% on RAR-XL.
Changes: Sec. 5.3 and Appendix F.3.
Question 2:
Thank you for highlighting possible false positives under source shift. We evaluated genuinely unwatermarked images from another generator and real photographs. RAV maintains low FPRs across both external sources.
Without retraining or threshold recalibration, we applied the original detector keys and previously frozen nominal-1%-FPR thresholds to 1,000 unwatermarked LlamaGen GPT-B images and 1,000 real COCO photographs. Observed FPRs on GPT-B/COCO were 1.00%/1.50% for WMAR-RAR-XL and 1.30%/0.50% for ClusterMark-RAR-XL. FPR remains low despite modest source-dependent variation.
Changes: Sec. 5.5 and Appendix C.
Question 3:
Thank you for suggesting evaluation on text-to-image AR generation. We added watermark verification experiments on Janus-Pro-1B. RAV improves detection over both Equal and Margin, supporting its applicability to an additional AR image-generation setting.
We adapted a WMAR-style keyed-token watermark and RAV to Janus-Pro-1B, using disjoint prompt splits. For each condition, 350 genuinely unwatermarked images calibrated the threshold; another 350 watermarked and 350 genuinely unwatermarked images formed the test set. Across PNG round-trip, JPEG Q50, and blur R1, macro TPRs were 94.00%, 93.43%, and 89.90% for RAV, Margin, and Equal, respectively. RAV gains were 4.10 pp over Equal and 0.57 pp over Margin.
Changes: Sec. 5.6 and Appendix D.
Question 4:
Thank you for this suggestion. We evaluated two held-out removal-attack families, neural compression and regeneration, and checked the available training and model-selection records. RAV retains gains over Margin or Confidence on these unseen attacks.
For WMAR, six CompressAI q=3 neural-compression models were absent from predictor training. At frozen nominal-1%-FPR thresholds, RAV improved TPR over Margin by 0.58 pp on RAR-XL and 4.23 pp on TAMING. For ClusterMark, we examined three regeneration attacks absent from recorded training conditions. On RAR-XL, RAV attained 95.67% TPR versus 93.92% for Confidence, a 1.75 pp gain. On strict GPT-B regeneration, RAV exceeded Confidence by 0.50 pp at their respective frozen thresholds.
Changes: Sec. 5.6 and Appendix E.
Weakness 1:
We thank the reviewer for asking about the practical benefits of learned weighting. We added comparisons with training-free weighting methods and computational-cost analyses. The results show that RAV further improves autoregressive image-watermark detection over simple weighting methods while maintaining manageable online verification overhead.
We also analyzed statistical significance against simple weighting. For WMAR, RAV improves TPR over training-free Margin at nominal 1%/0.1% FPR by 0.94/1.11 pp on RAR-XL (95% CIs [0.65, 1.24]/[0.82, 1.42]) and 1.53/2.32 pp on TAMING ([1.19, 1.86]/[1.93, 2.71]). See Question 1 for ClusterMark significance results.
The additional WMAR online cost is approximately 18.95 ms/image. ClusterMark verification takes 8.534 ms/image with RAV versus 3.945 ms/image with Hard. Overall, the cost is manageable, with statistically significant gains in three of four generator-watermark settings.
Changes: Sec. 2, Secs. 5.2 and 5.3, and Appendix F.3.
Weakness 2:
Thank you for identifying severe rotation and targeted-forgery limitations. Under a 15-degree rotation at nominal 1% FPR, RAV increases TPR from 1.75% with Margin to 3.00%. Although absolute detection remains low, this suggests that reliability weighting may exploit some residual watermark evidence; weighting cannot replace failed symbol recovery or synchronization.
The cited studies by Müller et al. (CVPR 2025 on diffusion-watermark forgery and a 2026 study of AR-watermark removal and forgery) provide threat context, not evidence that RAV resists these attacks. We explored feasible targeted-forgery evaluations but have not completed them owing to time and computational constraints.
Changes: Sec. 2, Sec. 6, and the geometric-attack analysis in Appendix Additional WMAR Results.