- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how much of a token-level watermark is preserved when an AR image model's tokens are decoded into an image, possibly edited, and converted back into tokens for detection. It defines a retention score that compares the average watermark detection score before and after this process and shows that, under a Gaussian assumption, higher retention yields better detection for the same model and watermark. It then proposes RenderGap, which estimates retention from unwatermarked images by checking how well tokens keep their watermark label after re-tokenization. The estimates match measured retention under JPEG and blur on Janus-Pro and LlamaGen, and across tokenizer variants on RAR-XL, but fail under cropping.
Critical strengths
- The method is simple and works in most tested settings. RenderGap needs only unwatermarked images, and its estimates match measured retention under JPEG, blur, and different tokenizers.
- Retention is more informative than the detection rate alone. For example, an adapted IndexMark is still detected in every image, but retention shows that re-tokenization weakens it (Table 22). Similarly, two RAR-XL tokenizers under JPEG have almost the same detection rate, but retention shows that one preserves more of the watermark (Table 10).
Critical weaknesses
- The full method does not clearly outperform the paper's own simpler baseline. Squaring the fraction of tokens that stay unchanged after re-tokenization gives errors close to RenderGap's, except for the cluster-based green list (Table 3a). The paper's own statistical tests find no significant difference for any of the six model and edit combinations (Table 14). The extra variance correction even makes estimates worse with small samples (Table 4a).
- The practical benefit is small. Watermarked images cost about as much to generate as unwatermarked ones, so measuring retention directly is already cheap. In the paper's cost comparison, RenderGap saves under two GPU-minutes when evaluating ten JPEG levels or three tokenizers (Table 4a). The paper argues the savings grow when testing many watermark keys, but on Janus-Pro and LlamaGen its estimates cannot reliably tell which key preserves more of the watermark (Table 18). RenderGap also fails under cropping, and stronger attacks such as diffusion-based regeneration are not tested.
Questions for authors
- Table 27 is the clearest case where the variance correction helps. Does this hold when prediction and measurement use separate unwatermarked images, as in Table 2?
- Given the small cost difference, when would someone evaluating a watermark prefer RenderGap over measuring retention directly with watermarked images?
- Do the estimates still match measured retention with classifier-free guidance, which all three models use by default, or with LlamaGen's default top-k sampling? All experiments use plain sampling at temperature 1.
4 comments
Response to Reviewer 6B9f (2/2)
W2. its estimates cannot reliably tell which key preserves more of the watermark
The ten keys of Table 19, which was Table 18 in the submission, differ in retention by only about one bootstrap SD. Hence neither RenderGap nor direct measurement can rank them at these sample sizes. In contrast, RenderGap orders candidates that differ by more than the noise. Under RAR-XL's official sampler with JPEG 75, it ranks the three tokenizers in their measured order and selects the one fine-tuned with augmentations (Table 33).
W2. RenderGap also fails under cropping
We agree that crop marks the boundary of the same-position transition approximation. On an absolute scale, however, prediction and measurement agree that crop, rotation by 5 degrees and horizontal flip leave almost no watermark signal on the three models. In these nine spatial settings the measured is at most 0.0553, and RenderGap predicts at most 0.0203 (Tables 34–37). The new Proposition 3 in Appendix A.2 derives the exact small-bias limit of under AR sampling, the AR limit, which also counts the signal that moves to other token positions. Its estimate has a lower error than RenderGap in all nine spatial settings (Table 42).
W2. stronger attacks such as diffusion-based regeneration are not tested.
Regeneration removes more of the watermark as its strength grows, and RenderGap predicts this decline from unwatermarked images on LlamaGen-L at all three strengths and on Janus-Pro-1B at strengths 0.1 and 0.2. We regenerated every decoded image with Stable Diffusion v1.5 image-to-image at strengths 0.1, 0.2 and 0.3 and re-encoded the result (Tables 34–37).
| Model | Strength | Measured | error |
|---|---|---|---|
| Janus-Pro-1B | 0.1 | 0.0693 | 0.0754 |
| Janus-Pro-1B | 0.2 | 0.0104 | 0.0242 |
| Janus-Pro-1B | 0.3 | 0.0029 | 0.4452 |
| LlamaGen-L | 0.1 | 0.1109 | 0.0767 |
| LlamaGen-L | 0.2 | 0.0278 | 0.0175 |
| LlamaGen-L | 0.3 | 0.0065 | 0.0322 |
| RAR-XL | 0.1 | 0.2114 | 0.1902 |
| RAR-XL | 0.2 | 0.1126 | 0.3593 |
| RAR-XL | 0.3 | 0.0538 | 0.3951 |
On Janus-Pro-1B at strength 0.3, the measured is 0.0029 and the prediction is 0.0016 (Table 34), so the large relative error of 0.4452 comes from two values that are both near zero.
On RAR-XL at strengths 0.1 and 0.2, the estimate lowers the errors to 0.0299 and 0.0077 (Table 42). Regeneration at these strengths therefore also moves part of the watermark signal to other token positions.
Q3. Do the estimates still match measured retention with classifier-free guidance
Yes. RenderGap still matches the measured retention under each model's official sampler, which uses classifier-free guidance on all three models and top-k 2000 on LlamaGen-L (Table 30). Under these samplers the median errors of RenderGap range from 0.0097 to 0.0562 across the five families without crop, close to its errors in Table 2 (Table 5).
Thank you again for the specific questions, and we are happy to answer any further questions during the discussion.
Thank you for the response. It addressed my concerns about the samplers and the separate test set. However, the full method is rarely better than the simpler baselines, and the practical saving remains small. Therefore, I keep my score.
Thank you for reading the response and for confirming that the concerns about the samplers and the separate test set are addressed. As for your remaining concerns:
1. the full method is rarely better than the simpler baselines
Both simpler baselines are special cases of the law , and the law accounts for what each one misses.
- The gap-only sets the variance factor to 1 and thus misses render channels that change the variation of green counts. The full prediction accounts for these render channels through , which is the second factor of the exact identity for retention (Eq. 10).
- The squared exact-token agreement also replaces the green-preservation gap by the fraction of tokens that are recovered unchanged (Section 5.2). So misses changed tokens that keep their green label, which captures.
The contribution is the law , which yields all three predictors together with the condition under which each one holds. Each simpler baseline therefore matches the full prediction wherever its condition holds, and the agreement in most tested settings supports the law. But where a condition fails, so does the simpler baseline built on it, whereas the full prediction stays far more accurate. Hence underpredicts retention on the cluster-based green list that you noted (Table 3(a)). Similarly, under a nearest-code search that changes the variation of green counts, the predicted-to-measured ratio is 2.80 for and 0.82 for (Table 3(b)).
2. the practical saving remains small
RenderGap halves the GPU time of an evaluation, and we agree that this saving is small in absolute terms, because watermarked images cost about as much to generate as unwatermarked ones. As you noted, however, retention is more informative than the detection rate alone. RenderGap explains this retention through the green-preservation gap and the variance factor of unwatermarked round trips. A small shows that the render channel changes the green labels of many tokens, and the choice of tokenizer can then raise retention, as the fine-tuned RAR-XL tokenizers do (Table 1). In contrast, a far from 1 shows that the re-encoding changes the variation of green counts, as the unnormalized nearest-code search does on Janus-Pro-1B (Table 3(b)). Hence, before any watermarked image is generated, RenderGap explains why an AR image model keeps or loses a watermark.
Thank you again for the careful reading, and we are glad to answer any further questions.
Response to Reviewer 6B9f (1/2)
Thank you very much for your thoughtful review and the precise questions. We appreciate your note that RenderGap is simple and that retention is more informative than the true-positive rate alone.
W1. The full method does not clearly outperform the paper's own simpler baseline.
Under each model's official sampler, RenderGap has a lower median error than the gap-only Δ2 and the squared exact-token agreement psame2 in all five families without crop (Table 5). The table below gives absolute relative errors for two cases in which the variance factor ν departs from 1. In both cases, the full prediction Δ2ν has a far lower error than both simpler predictors.
Exact-token agreement psame never uses the green list, so psame2 gives the same prediction for every key and construction and cannot compare watermark designs, as you note for the cluster-based green list of Table 3(a).
We agree that the paired tests of Table 15, which was Table 14 in the submission, find no significant difference between the three predictions. In all 48 settings of Table 15, ν lies between 0.88 and 1.14, so Δ2 stays close to Δ2ν. Moreover, the key-selected green list of the Global construction leaves a changed token's green label nearly independent of its original label, so psame2 stays close to Δ2. In contrast, for brightness 1.2 on Janus-Pro-1B with ν=0.67, Δ2ν is more accurate than psame2 in 1,992 of the 2,000 draws of the paired bootstrap (Table 47).
As you note, ν is noisy at 128 fitting images and then lowers accuracy (Table 4(a)). Our selection rule therefore uses Δ2ν only when a bootstrap test on ν rejects ν=1. At 128 fitting images on Janus-Pro JPEG, the rule lowers the mean error from 0.1366 for Δ2ν to 0.0901 (Table 45).
Q1. Does this hold when prediction and measurement use separate unwatermarked images, as in Table 2?
Yes. We split the 1,024 unwatermarked images of Table 28, which was Table 27 in the submission, 200 times at random into fitting and reference halves. In the unnormalized BF16 row the median error is 0.1811 for Δ2ν against 1.8210 for Δ2 (Table 46). The gain of ν thus holds on separate fitting and reference images.
W2 & Q2. The practical benefit is small; when would someone evaluating a watermark prefer RenderGap over measuring retention directly
The main benefit is precision, which matters most for weak watermarks. On RAR-XL with 128 unwatermarked images, the full prediction Δ2ν has a bootstrap SD of 0.0422 at every watermark bias δ because it uses no watermarked image (Table 48). Direct measurement adds 128 watermarked images, and its bootstrap SD rises from 0.0453 at bias 2 to 0.0710 at bias 0.2. The SD of Δ2ν thus stays below the SD of direct measurement at every tested bias, with half the images.
RenderGap is preferable in two further situations.