acceptodds
Under review as a conference paper at ICLR 2027

Predicting Token-Level Watermark Retention from Unwatermarked Images for Autoregressive Image Models

Abstract

Token-level watermarks make autoregressive (AR) image models more likely to sample tokens selected by a secret key, creating a signal that identifies watermarked images. However, the signal can remain detectable after image processing on one AR image model and become difficult to detect on another. Existing evaluation requires generating watermarked images for each model and watermark. In this paper we propose a metric for the signal retained when generated tokens are rendered into an image and recovered for detection. We prove that higher gives higher detection success for a fixed model, watermark, and sampling bias at a fixed false-positive rate under stated statistical assumptions. We then propose RenderGap, an evaluation method that predicts through a derived approximation using only unwatermarked generations. RenderGap compares generated tokens with tokens recovered from the images and scores both using the watermark key. Experiments with independent calibration on various AR image models show median absolute relative prediction errors of – across JPEG, blur, and tokenizer variants. RenderGap therefore evaluates retention for a given AR image model and watermark before watermarked images are generated.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
token-level watermarking, autoregressive image generation, watermark evaluation, vector quantization, image tokenizers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6B9fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how much of a token-level watermark is preserved when an AR image model's tokens are decoded into an image, possibly edited, and converted back into tokens for detection. It defines a retention score that compares the average watermark detection score before and after this process and shows that, under a Gaussian assumption, higher retention yields better detection for the same model and watermark. It then proposes RenderGap, which estimates retention from unwatermarked images by checking how well tokens keep their watermark label after re-tokenization. The estimates match measured retention under JPEG and blur on Janus-Pro and LlamaGen, and across tokenizer variants on RAR-XL, but fail under cropping.

Critical strengths
  1. The method is simple and works in most tested settings. RenderGap needs only unwatermarked images, and its estimates match measured retention under JPEG, blur, and different tokenizers.
  2. Retention is more informative than the detection rate alone. For example, an adapted IndexMark is still detected in every image, but retention shows that re-tokenization weakens it (Table 22). Similarly, two RAR-XL tokenizers under JPEG have almost the same detection rate, but retention shows that one preserves more of the watermark (Table 10).
Critical weaknesses
  1. The full method does not clearly outperform the paper's own simpler baseline. Squaring the fraction of tokens that stay unchanged after re-tokenization gives errors close to RenderGap's, except for the cluster-based green list (Table 3a). The paper's own statistical tests find no significant difference for any of the six model and edit combinations (Table 14). The extra variance correction even makes estimates worse with small samples (Table 4a).
  2. The practical benefit is small. Watermarked images cost about as much to generate as unwatermarked ones, so measuring retention directly is already cheap. In the paper's cost comparison, RenderGap saves under two GPU-minutes when evaluating ten JPEG levels or three tokenizers (Table 4a). The paper argues the savings grow when testing many watermark keys, but on Janus-Pro and LlamaGen its estimates cannot reliably tell which key preserves more of the watermark (Table 18). RenderGap also fails under cropping, and stronger attacks such as diffusion-based regeneration are not tested.
Questions for authors
  1. Table 27 is the clearest case where the variance correction helps. Does this hold when prediction and measurement use separate unwatermarked images, as in Table 2?
  2. Given the small cost difference, when would someone evaluating a watermark prefer RenderGap over measuring retention directly with watermarked images?
  3. Do the estimates still match measured retention with classifier-free guidance, which all three models use by default, or with LlamaGen's default top-k sampling? All experiments use plain sampling at temperature 1.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6B9f (1/2)

Thank you very much for your thoughtful review and the precise questions. We appreciate your note that RenderGap is simple and that retention is more informative than the true-positive rate alone.

W1. The full method does not clearly outperform the paper's own simpler baseline.

Under each model's official sampler, RenderGap has a lower median error than the gap-only Δ2\Delta^2 and the squared exact-token agreement psame2p_{\mathrm{same}}^2 in all five families without crop (Table 5). The table below gives absolute relative errors for two cases in which the variance factor ν\nu departs from 1. In both cases, the full prediction Δ2ν\Delta^2\nu has a far lower error than both simpler predictors.

Settingν\nuΔ2ν\Delta^2\nuΔ2\Delta^2psame2p_{\mathrm{same}}^2
LlamaGen-L, official sampler, median over 10 JPEG levels (Tables 5 and 32)1.02–1.200.03780.13590.1417
Janus-Pro-1B, brightness 1.2 (Table 34)0.670.04540.41790.4078

Exact-token agreement psamep_{\mathrm{same}} never uses the green list, so psame2p_{\mathrm{same}}^2 gives the same prediction for every key and construction and cannot compare watermark designs, as you note for the cluster-based green list of Table 3(a).

We agree that the paired tests of Table 15, which was Table 14 in the submission, find no significant difference between the three predictions. In all 48 settings of Table 15, ν\nu lies between 0.88 and 1.14, so Δ2\Delta^2 stays close to Δ2ν\Delta^2\nu. Moreover, the key-selected green list of the Global construction leaves a changed token's green label nearly independent of its original label, so psame2p_{\mathrm{same}}^2 stays close to Δ2\Delta^2. In contrast, for brightness 1.2 on Janus-Pro-1B with ν=0.67\nu=0.67, Δ2ν\Delta^2\nu is more accurate than psame2p_{\mathrm{same}}^2 in 1,992 of the 2,000 draws of the paired bootstrap (Table 47).

As you note, ν\nu is noisy at 128 fitting images and then lowers accuracy (Table 4(a)). Our selection rule therefore uses Δ2ν\Delta^2\nu only when a bootstrap test on ν\nu rejects ν=1\nu=1. At 128 fitting images on Janus-Pro JPEG, the rule lowers the mean error from 0.1366 for Δ2ν\Delta^2\nu to 0.0901 (Table 45).

Q1. Does this hold when prediction and measurement use separate unwatermarked images, as in Table 2?

Yes. We split the 1,024 unwatermarked images of Table 28, which was Table 27 in the submission, 200 times at random into fitting and reference halves. In the unnormalized BF16 row the median error is 0.1811 for Δ2ν\Delta^2\nu against 1.8210 for Δ2\Delta^2 (Table 46). The gain of ν\nu thus holds on separate fitting and reference images.

W2 & Q2. The practical benefit is small; when would someone evaluating a watermark prefer RenderGap over measuring retention directly

The main benefit is precision, which matters most for weak watermarks. On RAR-XL with 128 unwatermarked images, the full prediction Δ2ν\Delta^2\nu has a bootstrap SD of 0.0422 at every watermark bias δ\delta because it uses no watermarked image (Table 48). Direct measurement adds 128 watermarked images, and its bootstrap SD rises from 0.0453 at bias 2 to 0.0710 at bias 0.2. The SD of Δ2ν\Delta^2\nu thus stays below the SD of direct measurement at every tested bias, with half the images.

RenderGap is preferable in two further situations.

  • A design search over many keys, partitions or biases reuses one cached set of unwatermarked round trips, whereas direct measurement generates new watermarked images for every candidate (Table 4(b)).
  • A diagnosis uses the pooled gap Δ\Delta and ν\nu to separate green-label changes from variance changes.
AuthorsOpenReview ↗

Response to Reviewer 6B9f (2/2)

W2. its estimates cannot reliably tell which key preserves more of the watermark

The ten keys of Table 19, which was Table 18 in the submission, differ in retention by only about one bootstrap SD. Hence neither RenderGap nor direct measurement can rank them at these sample sizes. In contrast, RenderGap orders candidates that differ by more than the noise. Under RAR-XL's official sampler with JPEG 75, it ranks the three tokenizers in their measured order and selects the one fine-tuned with augmentations (Table 33).

W2. RenderGap also fails under cropping

We agree that crop marks the boundary of the same-position transition approximation. On an absolute scale, however, prediction and measurement agree that crop, rotation by 5 degrees and horizontal flip leave almost no watermark signal on the three models. In these nine spatial settings the measured η\eta is at most 0.0553, and RenderGap predicts at most 0.0203 (Tables 34–37). The new Proposition 3 in Appendix A.2 derives the exact small-bias limit of η\eta under AR sampling, the AR limit, which also counts the signal that moves to other token positions. Its estimate ηAR\eta_{\mathrm{AR}} has a lower error than RenderGap in all nine spatial settings (Table 42).

W2. stronger attacks such as diffusion-based regeneration are not tested.

Regeneration removes more of the watermark as its strength grows, and RenderGap predicts this decline from unwatermarked images on LlamaGen-L at all three strengths and on Janus-Pro-1B at strengths 0.1 and 0.2. We regenerated every decoded image with Stable Diffusion v1.5 image-to-image at strengths 0.1, 0.2 and 0.3 and re-encoded the result (Tables 34–37).

ModelStrengthMeasured η\etaΔ2ν\Delta^2\nu error
Janus-Pro-1B0.10.06930.0754
Janus-Pro-1B0.20.01040.0242
Janus-Pro-1B0.30.00290.4452
LlamaGen-L0.10.11090.0767
LlamaGen-L0.20.02780.0175
LlamaGen-L0.30.00650.0322
RAR-XL0.10.21140.1902
RAR-XL0.20.11260.3593
RAR-XL0.30.05380.3951

On Janus-Pro-1B at strength 0.3, the measured η\eta is 0.0029 and the prediction is 0.0016 (Table 34), so the large relative error of 0.4452 comes from two values that are both near zero.

On RAR-XL at strengths 0.1 and 0.2, the estimate ηAR\eta_{\mathrm{AR}} lowers the errors to 0.0299 and 0.0077 (Table 42). Regeneration at these strengths therefore also moves part of the watermark signal to other token positions.

Q3. Do the estimates still match measured retention with classifier-free guidance

Yes. RenderGap still matches the measured retention under each model's official sampler, which uses classifier-free guidance on all three models and top-k 2000 on LlamaGen-L (Table 30). Under these samplers the median errors of RenderGap range from 0.0097 to 0.0562 across the five families without crop, close to its errors in Table 2 (Table 5).

Thank you again for the specific questions, and we are happy to answer any further questions during the discussion.

Thank you for the response. It addressed my concerns about the samplers and the separate test set. However, the full method is rarely better than the simpler baselines, and the practical saving remains small. Therefore, I keep my score.

Thank you for reading the response and for confirming that the concerns about the samplers and the separate test set are addressed. As for your remaining concerns:

1. the full method is rarely better than the simpler baselines

Both simpler baselines are special cases of the law η≈Δ2ν\eta\approx\Delta^2\nu, and the law accounts for what each one misses.

  • The gap-only Δ2\Delta^2 sets the variance factor ν\nu to 1 and thus misses render channels that change the variation of green counts. The full prediction Δ2ν\Delta^2\nu accounts for these render channels through ν\nu, which is the second factor of the exact identity for retention η\eta (Eq. 10).
  • The squared exact-token agreement psame2p_{\mathrm{same}}^2 also replaces the green-preservation gap Δ\Delta by the fraction of tokens that are recovered unchanged (Section 5.2). So psame2p_{\mathrm{same}}^2 misses changed tokens that keep their green label, which Δ\Delta captures.

The contribution is the law η≈Δ2ν\eta\approx\Delta^2\nu, which yields all three predictors together with the condition under which each one holds. Each simpler baseline therefore matches the full prediction Δ2ν\Delta^2\nu wherever its condition holds, and the agreement in most tested settings supports the law. But where a condition fails, so does the simpler baseline built on it, whereas the full prediction Δ2ν\Delta^2\nu stays far more accurate. Hence psame2p_{\mathrm{same}}^2 underpredicts retention on the cluster-based green list that you noted (Table 3(a)). Similarly, under a nearest-code search that changes the variation of green counts, the predicted-to-measured ratio is 2.80 for Δ2\Delta^2 and 0.82 for Δ2ν\Delta^2\nu (Table 3(b)).

2. the practical saving remains small

RenderGap halves the GPU time of an evaluation, and we agree that this saving is small in absolute terms, because watermarked images cost about as much to generate as unwatermarked ones. As you noted, however, retention η\eta is more informative than the detection rate alone. RenderGap explains this retention through the green-preservation gap Δ\Delta and the variance factor ν\nu of unwatermarked round trips. A small Δ\Delta shows that the render channel changes the green labels of many tokens, and the choice of tokenizer can then raise retention, as the fine-tuned RAR-XL tokenizers do (Table 1). In contrast, a ν\nu far from 1 shows that the re-encoding changes the variation of green counts, as the unnormalized nearest-code search does on Janus-Pro-1B (Table 3(b)). Hence, before any watermarked image is generated, RenderGap explains why an AR image model keeps or loses a watermark.

Thank you again for the careful reading, and we are glad to answer any further questions.

Reviewer_Lwn3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates whether watermark retention in autoregressive image models can be predicted without generating watermarked images. It distinguishes watermark signal preservation from detection success and introduces RenderGap, a novel evaluation approach based on unwatermarked token round trips. Experiments show that its predictions closely match independently measured retention across tested JPEG, blur, and tokenizer settings, supporting more efficient evaluation before watermarked

Critical strengths

Predicting watermark retention without generating watermarked images is a practically valuable problem. The paper clearly distinguishes signal retention from detection success and develops an interpretable prediction framework using unwatermarked token pairs. Independent calibration and experiments across multiple models, processing conditions, and tokenizer variants strengthen the empirical evaluation.The paper is clearly organized and well written, with helpful illustrations of the evaluation pipeline.

Critical weaknesses

1.Spatial transformations substantially limit applicability. Crop-and-resize produces a median relative prediction error of approximately 55%, revealing a structural limitation of the same-position transition approximation. 2.Robustness to changes in the conditioning distribution remains unclear. Experiments primarily use limited, fixed subjects or classes. Prediction accuracy and configuration rankings may change when cached unwatermarked samples do not represent the deployment-time generation distribution. 3.Proposition 2 assumes fixed contexts and token-local transitions, whereas watermarking changes the prefix distribution during AR sampling. Appendix A acknowledges that the proposition provides no bounds on the resulting cross-position contributions. Although experiments support the approximation in the tested settings, its reliability beyond these settings lacks sufficient theoretical characterization.

Questions for authors

1.When initial watermark strength varies across models or watermark constructions, could the authors clarify which configuration choices can be made solely from RenderGap’s predicted retention and which require additional evaluation of initial signal strength and final detection performance?

2.The existing experiments have validated the accuracy of the pooled predictor. Could the authors clarify what proportion of the overall prediction residual is attributable to replacing the weighted gap with the pooled gap?

3.Could the authors explain how RenderGap’s prediction statistics could guide targeted improvements to watermark or tokenizer design, beyond comparing existing configurations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Lwn3 (1/2)

Thank you very much for your positive and thoughtful review.

W1. Spatial transformations substantially limit applicability.

We agree that spatial operations reveal a structural limitation of the same-position transition approximation. On an absolute scale, however, prediction and measurement agree that crop, rotation by 5 degrees and horizontal flip leave almost no watermark signal on the three models. In these nine spatial settings the measured retention η\eta is at most 0.0553, and RenderGap predicts at most 0.0203 (Tables 34–37). Our answer to W3 derives the AR limit, which also counts the watermark signal that spatial operations move to other token positions. Its estimate ηAR\eta_{\mathrm{AR}} has a lower error than RenderGap in all nine spatial settings (Table 42).

W2. Robustness to changes in the conditioning distribution remains unclear.

RenderGap predicts the retention of the conditioning distribution that its fitting images come from. We split the 60 prompts or classes of each model by id into a lower half and an upper half, fit RenderGap on one half and compared its prediction with the measured η\eta of the reference images of either half.

Mean error, fit → referencelower → lowerupper → upperlower → upperupper → lower
Janus-Pro-1B, JPEG0.04790.04060.05300.0652
LlamaGen-L, JPEG0.07260.03770.44200.2733
RAR-XL, without crop0.04450.02560.02960.0627

The prediction transfers across halves on Janus-Pro-1B and on RAR-XL without crop (Table 44). In contrast, the retention of LlamaGen-L itself depends on the classes. Under JPEG 90 its measured η\eta is 0.6570 for the lower half and 0.4951 for the upper half, while fitting images from the same half predict 0.7099 and 0.5029. Hence LlamaGen-L's errors across halves come from this change of retention. As you anticipated, the fitting images must follow the deployment's conditioning distribution, just like the watermarked images used for direct measurement. The new Section 5.6 states the same.

AuthorsOpenReview ↗

Response to Reviewer Lwn3 (2/2)

W3. watermarking changes the prefix distribution during AR sampling

The new Proposition 3 in Appendix A.2 handles changing prefixes in the small-bias limit. Proposition 3 uses the green innovation of a token sequence xx, ξ(x)=∑t(1[xt∈Gt]−πt(x<t)),\xi(x)=\sum_t\big(\mathbb{1}[x_t\in G_t]-\pi_t(x_{\lt t})\big), where πt(x<t)\pi_t(x_{\lt t}) is the green probability given the prefix. In terms of the green count SS of the generated tokens and the green count S^\hat S of the re-read tokens, the exact AR limit is η→β2ν,β=Cov(un)(S^,ξ)Cov(un)(S,ξ),as δ→0+.\eta\to\beta^2\nu,\qquad \beta=\frac{\mathrm{Cov}^{(\mathrm{un})}(\hat S,\xi)}{\mathrm{Cov}^{(\mathrm{un})}(S,\xi)},\qquad \text{as }\delta\to 0^{+}. Here δ\delta is the watermark bias, ν\nu is the variance factor and the covariances are taken under unwatermarked generation. The AR limit holds with changing prefixes and for any render channel that does not depend on δ\delta, including crop, provided Cov(un)(S,ξ)>0\mathrm{Cov}^{(\mathrm{un})}(S,\xi)\gt 0. The covariances in β\beta therefore count exactly the cross-position contributions that Proposition 2 leaves unbounded.

The log ratio log⁡(Δ2ν/η)\log(\Delta^2\nu/\eta) of RenderGap's prediction then splits exactly into a weighting term, a cross-position term, a finite-bias term and a variance term (Eq. 17). The cross-position term measures watermark signal at other token positions, and the finite-bias term measures the departure from the small-bias limit at the deployed bias. In all nine spatial settings the finite-bias term stays within one bootstrap SD of zero, while the cross-position term is negative (Tables 39–41). RenderGap thus underpredicts the spatial operations because the watermark signal moves to other token positions.

Q1. which configuration choices can be made solely from RenderGap's predicted retention

For a fixed AR image model, construction, key and bias, η\eta alone orders the TPR by Proposition 1. Choices among tokenizers, decoders, VQ implementations and image pipelines therefore need only the predicted η\eta. When the model, the construction or the bias changes, however, the TPR depends on the mean post-render score z^=η z\hat z=\sqrt{\eta}\ z (Table 21). Here zz is the generated-token mean score, and measuring zz needs watermarked tokens but no rendering.

Q2. what proportion of the overall prediction residual is attributable to replacing the weighted gap with the pooled gap?

Pooling explains a small part of the residual on Janus-Pro-1B and almost none on LlamaGen-L. The weighting term of Eq. 17 gives this share, and under uint8 it is 0.0064 on Janus-Pro-1B (Table 39) and about 0.0002 on LlamaGen-L (Table 40). As log ratios, these values change the prediction by less than 1%. In contrast, on RAR-XL under uint8 the weighting term is 0.0779 (Table 41), which has the opposite sign of the cross-position and finite-bias terms and offsets almost half of their sum.

Q3. how RenderGap's prediction statistics could guide targeted improvements to watermark or tokenizer design

The prediction separates causes of reduced retention, and each cause points to a different design change. First, a variance factor ν\nu far below 1 shows that the render channel inflates the green-count variance of unwatermarked images, as the unnormalized VQ search of Table 28 does. Restoring the native L2L_2 normalization raises η\eta there from 0.0075 to 0.7161. Second, a low exact-token agreement psamep_{\mathrm{same}} calls for a more robust tokenizer, as the RCC-ft tokenizer raises RAR-XL's f32 retention from 0.4732 to 0.9833 (Table 1). Third, the excess of Δ\Delta over psamep_{\mathrm{same}} measures how often a changed token keeps its green label. It rises when the green list groups similar codebook vectors, as Cluster-64 does in Table 3(a).

Thank you again for your comments on AR prefixes and on the conditioning distribution, which led to Proposition 3 and to Appendix F.4. We are happy to answer any further questions during the discussion.

Dear Reviewer Lwn3,

We thank you once more for your constructive feedback on our submission. Because the discussion period concludes on Nov 18, we wanted to check whether our response and the revised paper have resolved your concerns. Please let us know if you have any remaining or new concerns. We look forward to our discussion with you.

Reviewer_Y2UNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on estimating how well the token-level watermark is preserved in autoregressive image models when the tokens are converted into an image and then recovered, possibly after image processing. After defining watermark retention, the authors introduce the RenderGap method to estimate retention without generating watermarked images, so only unwatermarked generation is used. The main goal of such estimation is to make the process of configuring watermarking pipelines, such as choosing the tokenizer, faster by avoiding additional watermarked generation for evaluation. The method compares how green-label membership changes between original and recovered tokens of unwatermarked images. The paper provides a theoretical justification under simplifying assumptions and shows how retention affects successful watermark detection.

Critical strengths

The paper presents credible evidence that RenderGap can predict retention sufficiently well for the tested JPEG compression levels, blur strengths, and tokenizer variants. Notably, the reference retention was measured on separate samples.

The results in Table 20 show a useful separation of retention from detection strength. It was shown that lower retention can be compensated by a stronger initial watermark signal, making the detection rate higher. The experiment results also distinguish exact token recovery and green-label preservation.

Critical weaknesses

The benefit over simpler predictors remains insufficiently demonstrated. It is not clear when the full estimator provides a meaningful advantage in practice. The experiments show that estimating the variance correction can worsen the accuracy. Moreover, all paired mean-error differences reported in Table 14 remain statistically unresolved.

Crop-and-resize transformations expose a practical limitation of the estimator, producing 55.31% median absolute relative error in the RAR-XL comparison. On top of that, the Gaussian detection formula predicts a true positive rate equal to 0.9628 compared with the actual observed 0.7617 for Janus-Pro at JPEG quality 30. These limitations restrict the estimator’s practical applicability

Questions for authors

Is there an observable criterion for deciding when to use the full predictor rather than gap-only alternative?

Could you evaluate the estimator on additional image processing operations to assess its real-world applicability?

How much do the weighted gap in Proposition 2 and the pooled gap that is used in the experimental estimator actually differ?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Y2UN (1/2)

Thank you very much for your thoughtful review and constructive questions. We appreciate your note that the paper gives credible evidence for RenderGap and usefully separates retention from detection strength.

W1. The benefit over simpler predictors remains insufficiently demonstrated.

With 1,000 fitting images the full prediction Δ2ν\Delta^2\nu is our default. It has the lowest median error in all five official-sampler families without crop and over the aligned operations on Janus-Pro-1B and LlamaGen-L, which are operations that keep image content at its pixel positions (Table 5). RAR-XL is the one exception, where Δ2\Delta^2 has the lower median error, and our answer to Q2 explains why.

The gap-only Δ2\Delta^2 and the squared exact-token agreement psame2p_{\mathrm{same}}^2 are special cases of Δ2ν\Delta^2\nu. Δ2\Delta^2 assumes that the variance factor ν\nu equals 1, and psame2p_{\mathrm{same}}^2 further assumes that a changed token's green label is independent of its original label. The derivation therefore predicts that Δ2ν\Delta^2\nu is more accurate where ν\nu departs from 1.

Settingν\nuΔ2ν\Delta^2\nuΔ2\Delta^2psame2p_{\mathrm{same}}^2
LlamaGen-L, official sampler, median over 10 JPEG levels (Tables 5 and 32)1.02–1.200.03780.13590.1417
Janus-Pro-1B, brightness 1.2 (Table 34)0.670.04540.41790.4078

The entries are absolute relative errors. In both cases ν\nu departs from 1, so Δ2ν\Delta^2\nu has a far lower error than the special cases.

We agree that no paired difference in Table 15, which was Table 14 in the submission, is significant. There ν\nu lies between 0.88 and 1.14 in all 48 settings, so the predictions nearly coincide and Table 15 cannot show the gain that ν\nu brings.

In contrast, the paired bootstrap finds a significant gain of Δ2ν\Delta^2\nu where ν\nu departs furthest from 1. For brightness 1.2 on Janus-Pro-1B, at ν=0.67\nu=0.67, Δ2ν\Delta^2\nu is more accurate than Δ2\Delta^2 in 1,993 of the 2,000 draws (Table 47).

Q1. Is there an observable criterion for deciding when to use the full predictor

Yes. Since ν\nu uses only unwatermarked images, a bootstrap test on ν\nu is an observable criterion that needs no watermarked image (Table 45). Our selection rule uses Δ2ν\Delta^2\nu when this test rejects ν=1\nu=1 at the 5% level and Δ2\Delta^2 otherwise. The rule selects Δ2ν\Delta^2\nu at 8 of the 10 JPEG levels of LlamaGen-L's official sampler and for brightness 1.2 on Janus-Pro-1B. The rule thus uses Δ2ν\Delta^2\nu where the sampler or the render channel changes the green-count variance of unwatermarked images. When ν\nu is noisy at 128 fitting images on Janus-Pro JPEG, the rule lowers the mean error from 0.1366 for Δ2ν\Delta^2\nu to 0.0901.

AuthorsOpenReview ↗

Response to Reviewer Y2UN (2/2)

W2. Crop-and-resize transformations expose a practical limitation of the estimator

We agree that crop marks the boundary of the same-position transition approximation. On an absolute scale, however, prediction and measurement agree that crop, rotation by 5 degrees and horizontal flip leave almost no watermark signal on the three models. In these nine spatial settings the measured retention η\eta is at most 0.0553, and RenderGap predicts at most 0.0203 (Tables 34–37). The new Proposition 3 in Appendix A.2 derives the exact small-bias limit of η\eta under AR sampling, the AR limit, which also counts the watermark signal that moves to other token positions. In all nine settings its estimate ηAR\eta_{\mathrm{AR}} has a lower error than RenderGap (Table 42). RenderGap therefore underpredicts the spatial operations because the watermark signal moves to other token positions.

W2. the Gaussian detection formula predicts a true positive rate equal to 0.9628

You are right that the Gaussian formula overpredicts the TPR at JPEG 30. This formula relates η\eta to the TPR in Proposition 1, whereas RenderGap predicts η\eta itself from unwatermarked token pairs. The observed TPR still rises with η\eta in all Janus-Pro-1B and LlamaGen-L sweeps, so the error lies in the level of the predicted TPR and comes from the unit-variance assumption. Fitting the ratio of the SD of the watermarked scores to the SD of the unwatermarked scores lowers the mean TPR error over the 17 Janus-Pro-1B settings from 0.0652 to 0.0142 (Table 7). The fitted formula gives 0.7780 against the observed 0.7617 at JPEG 30 (Appendix C.1).

Q2. Could you evaluate the estimator on additional image processing operations

Yes. On Janus-Pro-1B and LlamaGen-L, RenderGap has the lowest median error over 14 new aligned operations, which keep image content at its pixel positions (Table 5). They range from brightness and noise to Stable Diffusion v1.5 regeneration at three strengths. On RAR-XL, the gap-only Δ2\Delta^2 has the lower median error, 0.0595 against 0.0792 for Δ2ν\Delta^2\nu. The reason is that ν\nu lies below 1 in 13 of the 16 aligned settings, where omitting ν\nu raises the prediction and offsets the underprediction of Δ2ν\Delta^2\nu (Table 36).

Q3. How much do the weighted gap in Proposition 2 and the pooled gap

Pooling costs RenderGap almost no accuracy on Janus-Pro-1B and LlamaGen-L, where the pooled gap Δ\Delta agrees with the weighted gap Δ∗\Delta_* of Proposition 2 within 0.6% (Table 38). The weighted gap needs next-token probabilities from a teacher-forced pass, whereas the pooled gap needs only the unwatermarked round trips. On RAR-XL, Δ\Delta exceeds Δ∗\Delta_* by about 4%, which raises the predicted η\eta (Table 38).

Thank you again for your careful review, and we are happy to answer any further questions during the discussion.

Dear Reviewer Y2UN,

Thank you again for the time and care you gave to our paper. As the discussion ends on Nov 18, we are writing to ask whether our response and the revised paper have addressed your concerns. Any further questions or new concerns are very welcome. We look forward to discussing the paper with you.

Comments on the paper

2 comments
AuthorsOpenReview ↗

New experiments for all reviewers

We thank all reviewers for their thoughtful and constructive reviews. In three new experiments RenderGap predicts from 1,000 unwatermarked fitting images, while the measured retention η\eta comes from separate reference images, as in Table 2.

1. RenderGap stays accurate under each model's official sampler, with the lowest median error in all five families without crop (Table 5).

2. On Janus-Pro-1B and LlamaGen-L, RenderGap has the lowest median error over 14 new operations that keep image content at its pixel positions, including Stable Diffusion v1.5 regeneration. On RAR-XL, Δ2\Delta^2 has the lower median error (Table 5). Crop, rotation by 5 degrees and horizontal flip instead move image content between pixel positions. For these spatial operations on the three models, the measured η\eta is at most 0.0553, and RenderGap predicts at most 0.0203 (Tables 34–37). RenderGap thus leads to the same conclusion as direct measurement, that these operations remove almost all of the watermark.

3. The new Proposition 3 derives the exact small-bias limit of η\eta under AR sampling, the AR limit, which also counts the watermark signal that moves to other token positions. In all nine spatial settings, its estimate ηAR\eta_{\mathrm{AR}} has a lower error than RenderGap (Table 42). In the exact decomposition of RenderGap's error in Eq. 17, the term that measures watermark signal at other token positions is negative in all nine settings (Tables 39–41). Hence RenderGap underpredicts the spatial operations because the watermark signal moves to other token positions.


Revised text is highlighted in blue in the paper. Section 5.6 and Table 5 summarize items 1 and 2, so every appendix table number is one higher than in the submission, and our replies cite the revised numbers. Appendix A.2 states and proves Proposition 3, and Appendix F (Tables 30–48) reports all new experiments. Appendix D recomputes Table 22 on the image sets of Table 1.

AuthorsOpenReview ↗

Summary of the discussion

Dear Area Chair and Reviewers,

We thank the Area Chair and Reviewers Y2UN, Lwn3 and 6B9f for their time and discussion.

Strengths noted by the reviewers

  • RenderGap predicts watermark retention from only unwatermarked images, which addresses a "practically valuable problem" (Lwn3, 6B9f).
  • The paper clearly separates retention from detection success (6B9f, Lwn3, Y2UN).
  • Tests on separate samples give "credible evidence" that RenderGap matches measured retention across JPEG levels, blur strengths and tokenizers (Y2UN, 6B9f).

Main concerns and our responses

  • Official samplers and separate fitting images (6B9f). Under each model's official sampler with classifier-free guidance, RenderGap has the lowest median error in all five families without crop (Table 5). The gain of the full prediction Δ2ν\Delta^2\nu also holds when the fitting images and the reference images are separate (Table 46), and Reviewer 6B9f confirmed that the response addressed both concerns.
  • More image operations, including diffusion regeneration (Y2UN, 6B9f). On Janus-Pro-1B and LlamaGen-L, RenderGap has the lowest median error over 14 new operations that keep image content at its pixel positions, including Stable Diffusion v1.5 regeneration (Table 5). Reviewer Y2UN thanked us for the additional experiments and raised the rating.
  • Spatial operations such as crop (Y2UN, Lwn3, 6B9f). Prediction and measurement agree that crop, rotation by 5 degrees and horizontal flip remove almost all of the watermark on the three models (Tables 34-37). The new Proposition 3 derives the AR limit, whose estimate has a lower error than RenderGap in all nine spatial settings (Table 42), while RenderGap underpredicts these operations because the watermark signal moves to other token positions (Tables 39-41).
  • Changing AR prefixes and the conditioning distribution (Lwn3). Proposition 3 holds with changing prefixes in the small-bias limit (Appendix A.2). A split of the prompts or classes shows that the fitting images must follow the deployment's conditioning distribution, just like the watermarked images used for direct measurement (Table 44, Section 5.6).
  • Benefit over the simpler predictors and the practical saving (6B9f, Y2UN). As special cases of the law η≈Δ2ν\eta\approx\Delta^2\nu, the gap-only Δ2\Delta^2 and the squared exact-token agreement psame2p_{\mathrm{same}}^2 match the full prediction Δ2ν\Delta^2\nu where their conditions hold, while Δ2ν\Delta^2\nu is far more accurate where a condition fails (Tables 3 and 47). Moreover, RenderGap halves the GPU time of an evaluation and uses the green-preservation gap Δ\Delta and the variance factor ν\nu to explain why an AR image model keeps or loses a watermark before any watermarked image is generated (Tables 1 and 3(b)). Our response on Nov 16 answers both points of Reviewer 6B9f's follow-up comment.

Section 5.6 and Appendix F of the revised PDF report all new experiments, and Appendix A.2 states and proves Proposition 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.