acceptodds
Under review as a conference paper at ICLR 2027

A Practical and Undetectable Watermark for Language Models Using Pseudorandom Codes

Abstract

The strongest quality notion for language model watermarks is undetectability, which ensures that the watermarked model is computationally indistinguishable from the original model. This notion is powerful, implying that the watermark preserves the model’s performance on any task that can be run in polynomial time. However, until now all undetectable watermarks either lacked nontrivial robustness (Christ et al., 2024) or had no practical implementation (Christ and Gunn, 2024). The latter construction uses pseudorandom codes (PRCs), whose codeword length corresponds to the number of tokens required for detection. Achieving nontrivial robustness required infeasibly long codewords, on the order of thousands of tokens. In this work, we significantly reduce the number of tokens required for detection in the PRC watermarking scheme of Christ and Gunn (2024) by introducing a new entropy-aware detector. We show that on the Qwen family of models, reliable detection is possible given only approximately 450 tokens. For the first time, this brings robust and undetectable LLM watermarks into the realm of practicality. Empirically, we find that the PRC watermark resists black-box detection and preserves quality, as measured by benchmark performance and output diversity. Compared with state-of-the-art baselines, our watermark offers greater resistance to the evaluated watermark-stealing attacks than SynthID-Text (Dathathri et al., 2024) and more diverse responses than TextSeal (Sander et al., 2026). Thus, we provide the first practical implementation of PRC watermarking for language models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Watermarking, Cryptography, Provenance, Information Integrity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7jdmOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a practical implementation of a robust, undetectable language model watermark based on Pseudorandom Codes (PRCs). By introducing an "entropy-aware detector" (using a posterior-mean scoring rule) that weights the watermark signal according to the model's token-level prediction uncertainty, the method drastically reduces the required detection length to approximately 450 tokens. Experimental validation on the Qwen model family demonstrates that this approach achieves reliable detection (90% TPR at 0.1% FPR) while preserving text quality and diversity, evading black-box detection tests, and resisting both watermark-stealing attacks and random token substitutions.

Critical strengths
  • Solves a Critical Bottleneck: Drastically reduces the required sequence length from thousands of tokens to ~450, making PRC watermarks viable for real-world deployment.

  • Novel Algorithmic Approach: Introduces an "entropy-aware detector" that leverages the model's token-level prediction uncertainty to maximize watermark signal detection.

  • Maintains Undetectability: Keeps the watermarked output distribution mathematically identical to the baseline, fully preserving text quality and diversity.

  • Comprehensive Validation: Provides strong empirical evidence on Qwen models, proving high detection efficiency, evasion of black-box tests, and robust resistance to stealing attacks.

Critical weaknesses
  • Critical Vulnerability to Desynchronization: The watermark relies entirely on token positional indices. As a result, it is highly susceptible to insertion and deletion attacks; even minor edits that shift the token order can desynchronize the parity checks and completely erase the watermark.

  • Fragility in Short Texts: Although the paper claims practicality at ~450 tokens, the robustness experiments (Appendix D.13) reveal that for 400-token outputs, a 30% substitution rate plummets detection to just 2.6%. The robustness claims are only practically valid for much longer texts (e.g., 4,000+ tokens).

  • Susceptibility to Paraphrasing Attacks: Because the method cannot survive insertions or deletions, standard real-world attacks, such as asking another LLM to rewrite or paraphrase the text (semantic-preserving transformations), will trivially and completely remove the watermark.

  • Limitation to Zero-bit Information (No User Identification): The proposed PRC encoding only signals the presence or absence of a watermark (zero-bit). It cannot embed multi-bit payloads, meaning it is impossible to identify specific users, sessions, or model versions from the watermarked text, severely limiting its utility for tracing misuse.

  • Compromised Theoretical Guarantees: To achieve shorter, practical detection lengths, the authors reduced the parity-check weight to t=3t=3. This parameter choice sacrifices the rigorous asymptotic undetectability guarantees established in the original PRC framework, leaving the scheme reliant entirely on empirical, rather than theoretical, validation.

Questions for authors
  • The robustness evaluation heavily focuses on token substitutions. How does the proposed watermark perform against desynchronization attacks (insertions/deletions) or semantic-preserving paraphrasing? Are there any mitigation strategies planned for these?

  • Table 15b shows that for 400-token outputs, the detection rate plummets to 2.6% under a 30% substitution rate. Given this fragility, what is the realistic minimum sequence length required for this watermark to be robust in real-world deployments?

7 comments

Thank you for your detailed feedback and for recognizing the practical improvements achieved by our work! We address your two questions first, followed by the remaining concerns.

Question 1. We agree that our current implementation is vulnerable to desynchronization, since the code that we use involves parity checks that depend on token positions. However, this is not an inherent limitation of PRC watermarking: Golowich and Moitra (2024) and Christ et al. (2026) construct PRCs that are robust to edits (insertions and deletions). Watermarks using these edit-robust PRCs are not yet practical, as their rate of error tolerance is too low. Our work improves the error tolerance of PRC-based watermarks, making the parity-check-based PRC that we use practical. While further error tolerance is required to use these edit-robust PRCs in practice, we make progress on this front.

Extensive paraphrasing is also a general challenge for text watermarks, including deployed systems. Both Google's SynthID-Text and Anthropic's Claude watermark acknowledge that substantial rewriting can weaken or remove their watermark signals. We have not evaluated paraphrasing attacks on our implementation and leave robustness to these transformations to future work.

Question 2. The minimum sequence length required for robust detection depends on the substitution rate and desired detection power. As discussed in our response to Reviewer V3rL, our existing substitution experiments only evaluate 400 and 4,096 tokens, so they do not establish that 4,096 tokens are necessary. We are extending these experiments to intermediate lengths to better identify practical operating points under different substitution rates.

As a practical reference point, the recent evaluation of OpenAI's text watermark reports approximately 66% detection under 10% synonym replacement and 17% under 25% replacement for 400-token ELI5 responses, at a target FPR of 1%. We are also evaluating PRC robustness on ELI5 to help determine the sequence lengths needed to achieve comparable detection rates. We expect to share these additional results in the next few days.

Zero-bit encoding and user identification. Our implementation indeed uses a zero-bit PRC, but this is not a fundamental limitation of PRCs. Christ and Gunn (2024), Section 6.1, provide a generic transformation from zero-bit to multi-bit PRCs, and Gunn et al. (2025) demonstrate multi-bit PRC watermarking for images. These techniques could in principle support user or session identification in LLM watermarking, although additional payloads may require longer outputs for reliable detection. We leave their practical implementation to future work.

Theoretical guarantees. Indeed, as discussed in Appendix B.5, our practical choice of parity-check weight (t=3) falls outside the parameter regime of the original asymptotic undetectability guarantee from Christ and Gunn (2024). Our evidence for undetectability at these parameters is therefore empirical, supported by quality-preservation experiments and black-box distinguishing tests. Separately, the statistical FPR guarantee of Theorem 3.3 remains valid at t=3, under its stated assumptions. We will make this distinction clearer in the main text.

Thank you for the response. It confirms rather than resolves the main concerns. The evaluated scheme remains vulnerable to insertion and deletion, and paraphrase robustness has not been tested. Citing other edit-robust PRC constructions does not establish robustness for this implementation.

The claim of practical detection at around 450 tokens is also unsupported under the reported attacks. For example, at 400 tokens, detection falls to 2.6% with 30% substitutions, while the proposed intermediate-length results are still pending. The comparison to OpenAI’s watermark uses different attack rates and FPR targets, so it does not establish comparable robustness.

Finally, the authors acknowledge that t=3t=3 falls outside the asymptotic undetectability guarantee. Black-box tests provide empirical evidence, not an equivalent guarantee; the possibility of extending the scheme to multi-bit payloads is likewise not a demonstrated capability.

These are central limitations for a practical, robust watermark, not peripheral issues. I do not consider the response sufficient to change my rating.

Thank you for your follow-up and your patience while we completed the additional robustness experiments! We’re excited to share new results demonstrating strong substitution robustness at substantially shorter, more practical response lengths. With the discussion deadline approaching, we’d really appreciate your time and consideration of this new evidence and our clarifications. We address your main concerns point by point below.

1. Practical robustness at shorter response lengths

The claim of practical detection at around 450 tokens is also unsupported under the reported attacks. For example, at 400 tokens, detection falls to 2.6% with 30% substitutions, while the proposed intermediate-length results are still pending.

Our intermediate-length experiments are now complete, and they demonstrate strong substitution robustness at substantially shorter, more practical lengths than 4,096 tokens. Our PRC watermark with the posterior-mean detector achieves 92.6% TPR at 1,024 tokens under 10% random substitutions, and 93.2% TPR at 2,048 tokens under 20% substitutions, at a target FPR of 0.1%.

The new detector comparisons also demonstrate the benefit of entropy-aware detection under substitution attacks. At 1,024 tokens under 10% substitutions, the naïve detector achieves 79.6% TPR on the same attacked responses, at the same target FPR. Our posterior-mean detector therefore provides a 13-percentage-point improvement.

We’d like to clarify that the approximately 450-token result concerns unmodified text, not text subjected to edit attacks. For context, OpenAI’s deployed watermark also shows reduced detection after substitution attacks at 400 tokens, as discussed in the comparison below. We’re finalizing revisions to the main text to distinguish this from the lengths required for strong detection after substitutions, which our new experiments characterize.

These results also support our broader goal of making robust, undetectable watermarking practical. Our comparison with CGZ, the earlier undetectable watermark of Christ, Gunn, and Zamir (2024), demonstrates a practical advance: our PRC watermark achieves substantially higher detection after substitutions with much less text. Under 10% random substitutions, CGZ achieves only 14.4% TPR even at 4,096 tokens, at the same target FPR of 0.1%. Thus, under this attack, our PRC watermark achieves much higher detection with one-quarter as much text.

An overview of all three experiments and their main findings is available in this update.

Reviewer_jZ4iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

My understanding for the main idea of this paper is, it proposes a detector that weights token observations according to how reliably they are able to reveal the underlying hidden code bits. This seems to help address the difficulty of detecting the pseudorandom-code watermarks of Christ and Gunn (2024) from texts with shorter detection length. The approach is supported by a theorem that provides false-positive-rate guarantee and experiments demonstrating improved detection power with fewer tokens compared with the baseline.

Critical strengths
  1. The high level intuition of the idea is very natural, however also impressive. The authors explain the method clearly, allowing me to quickly grasp the main idea and understand why it should improve detection.

  2. Although the core idea is simple, the reported experimental results seem to suggest that it can significantly improve detection performance over the baseline, as shown in section 4.1 of the paper.

Critical weaknesses

Of the two proposed detectors, the posterior-mean detector appears to perform better. However, the core idea of it already appears in the paper An Undetectable Watermark for Generative Image Models (Appendix B, Algorithm 3), though this paper focus on image watermarks. Although this work was cited in this manuscript, this manuscript does not discuss this methodological connection in detail or clearly distinguish the proposed detector from the earlier framework. This overlap weakens the conceptual novelty of the posterior-mean approach as currently presented. The authors should clarify the specific contributions of the adaptation to language models and the modified detection statistic.

Questions for authors

See the Weaknesses section. In particular, could the authors clarify in detail the relationship between the method from An Undetectable Watermark for Generative Image Models and the posterior-mean detector in this manuscript, and explain in more detail what nontrivial theoretical or practical extensions are needed to adapt its ideas to the LLM setting?

4 comments

Thank you for your thoughtful review and helpful feedback. We appreciate your highlighting the connection to Gunn et al. (2025), who previously applied posterior-mean estimation, a general statistical technique, to PRC image watermarking.

Our work develops a broader approach to entropy-aware PRC detection for language models, with posterior-mean scoring as one of two methods. Adapting posterior-mean detection to language models involves distinct technical challenges. In the image-watermarking setting, Gunn et al. estimate hidden PRC bits from recovered continuous diffusion latents under a Gaussian observation model. In contrast, our detector observes discrete token bins whose reliability depends on the language model's context-dependent token probabilities. We derive the corresponding posterior scores under the language-model embedding procedure of Christ and Gunn (2024). In particular, observing a token from the less probable bin determines the hidden bit, while observing the more probable bin generally provides weaker evidence.

A further challenge is that the true generation-time probabilities are generally unavailable to the detector. We demonstrate that useful estimates can be recovered without the original prompt, even using a smaller, mismatched model, while retaining much of the detection power (Figure 2).

We also introduce a different detection statistic that directly aggregates soft-parity scores, rather than the log-odds-transformed scores of Gunn et al., and establish its statistical FPR guarantee in Theorem 3.3.

Separately, we introduce an entropy-weighted detector that directly weights observations by their binary partition entropy, without computing posterior probabilities. This alone substantially improves detection over the naive PRC baseline, achieving 79.4% versus 58.2% TPR at 448 tokens, while our posterior-mean detector reaches 90.0% (Table 3).

We will revise the related work and methods sections to acknowledge Gunn et al. more prominently and make these methodological distinctions and our contributions clearer.

I thank the authors for their rebuttal. My responses are as follows:

  1. "We derive the corresponding posterior scores under the language-model embedding procedure of Christ and Gunn (2024). In particular, observing a token from the less probable bin determines the hidden bit, while observing the more probable bin generally provides weaker evidence." My understanding is that the authors are mentioning the calculation in Appendix A.4, however this seems straightforward and in fact can be derived following the posterior probability in page 53 of Christ and Gunn (2024) (https://arxiv.org/pdf/2402.09370v2). Therefore this derivation does not seem to be a technical challenge.

  2. "We demonstrate that useful estimates can be recovered without the original prompt, even using a smaller, mismatched model, while retaining much of the detection power (Figure 2)." It seems that this idea is quite common in the field of watermarking, for example, see the section 5 and page 23-24 of the paper Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach. The link is https://papers.nips.cc/paper_files/paper/2025/file/956f80e60a211e3a0f6ec5f2dfa3551a-Paper-Conference.pdf Also, this paper seems not to be cited in the manuscript.

I thank the authors again for the rebuttal however some of my main concerns remain unsolved, I would like to keep the score of 2

Thank you for your helpful follow-up and for the time you have devoted to evaluating our work. We recognize that our previous response did not fully address your concern about the substantive contribution beyond existing methods. With the discussion deadline approaching, we would be very grateful if you could give the clarifications and additional evidence below one final consideration.

1. Contribution beyond the posterior calculation

“this seems straightforward and in fact can be derived following the posterior probability in page 53 of Christ and Gunn (2024)”

“Therefore this derivation does not seem to be a technical challenge.”

We agree that the posterior calculation in Appendix A.4 follows naturally from the embedding probabilities in Christ and Gunn (2024). However, our case for a substantive contribution rests on the resulting detector design, analysis, and practical improvements, rather than on the difficulty of deriving the posterior.

The practical question we address is how much the text required for PRC detection can be reduced by better using the evidence already present in language-model outputs. We develop entropy-weighted and posterior-mean detectors and evaluate them on identical watermarked texts, holding the PRC parameters fixed. The gains show that part of the apparent need for long outputs comes from how the detector handles embedding uncertainty.

We understand your concern about the “conceptual novelty of the posterior-mean approach” to be whether the LLM adaptation establishes a substantive contribution beyond the existing image-watermarking method. Indeed, our posterior-mean scoring shares the soft-decision PRC approach of Gunn et al. (2025). However, their image-watermarking results do not establish the practical detection lengths or substitution robustness achievable for text. Our work establishes these capabilities for the evaluated LLM settings.

We'd like to point out that there is precedent in watermarking for developing established ideas into practical methods for another modality. For example, Jovanović et al. (NeurIPS 2025) adapt LLM watermarking to autoregressive images, addressing errors introduced by image retokenization, while Milis et al. (ICML 2026) build on the KGW text watermark for audio through vocabulary clustering and an analysis of token corruption.

For text specifically, Sander et al. (NeurIPS 2024) extend the concept of image-data radioactivity to LLMs, developing specialized detectors to reveal weak watermark traces left by training on watermarked text. Abdelnabi and Fritz (IEEE S&P 2021) bring the jointly trained encoder–decoder and adversarial framework used in image data hiding to language. These examples illustrate the contribution of developing and validating methods that make an established idea effective in a new setting. For PRC text watermarking, our contribution is the resulting detector design, analysis, and demonstrated improvements in detection efficiency and substitution robustness in the evaluated LLM settings.

Finally, we highlight three empirical findings supporting our practical contributions, including new robustness results and a comparison not previously discussed:

  • (i) Short, unmodified text. At 448 tokens on Qwen3-0.6B-Base, entropy weighting improves TPR from 58.2% to 79.4%, and posterior-mean detection further improves it to 90.0%, all at target FPR 0.1% (Figure 1 and Table 3).
  • (ii) Improved substitution robustness. Our new experiments show that posterior-mean detection improves TPR over the naïve detector by 13.0 percentage points under 10% random substitutions and 12.4 percentage points under 20% substitutions, on the same 1,024-token attacked responses at target FPR 0.1%.
  • (iii) Another PRC implementation. Our comparison using Wang et al. (2025), which we did not discuss in our previous response, applies both detectors to identical watermarked texts, with each threshold empirically calibrated to target FPR 0.1%. On 1,024-token Qwen3-8B-Base completions at temperature 1.2, posterior-mean detection improves TPR from 89.38% to 98.13%.

These results isolate substantial benefits from detector design, including after substitutions and across implementations.

2. Smaller surrogate models and prompt-free detection

“It seems that this idea is quite common in the field of watermarking”

“Also, this paper seems not to be cited in the manuscript.”

Thank you for bringing He et al. (2025) to our attention. We agree that smaller surrogate models and prompt-free detection are existing ideas, and we do not claim that they originated in our work.

Put simply, our contribution here is empirically showing that smaller surrogate models and prompt-free detection work in practice for the Christ–Gunn PRC construction, using our improved entropy-aware detection (Figure 2). This establishes that the improvements in detection efficiency remain useful when the original generation probabilities are unavailable, an important consideration for practical use.

We are finalizing manuscript updates that clarify the connection to image PRC detection and cite He et al. in Related Work, including the discussion of distortion-free watermarking, and alongside Figure 2. We will post the updated manuscript shortly.

Thank you again for your careful engagement. We recognize that this is a further request for your time close to the deadline. We hope these clarifications and results address your remaining concerns, and would be very grateful if you would consider raising your rating in light of them.

Reviewer_V3rLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

If I understand correctly, the authors adopt the watermark generation process in previous work [1] and propose entropy-aware detection strategies for this watermarking method. Experimental results in Table 3 show that the posterior mean detector outperforms the naïve detector.

[1] Chowdhury, Aakash, et al. "Leakage certification made simple." Annual International Cryptology Conference. Cham: Springer Nature Switzerland, 2024.

Critical strengths
  1. From Table 2, the watermark is distortion-free. As for the design, the posterior mean detector is exactly the posterior under the embedding rule. The detection results show that this method achieves a large performance gain compared to naïve.

  2. The authors also gave an FPR guarantee for this watermark detection.

Critical weaknesses
  1. The authors emphasize the importance of robustness in Introduction, while the results in Appendix D13 (400 tokens) do not support this claim under random token substitution attack. Robustness is retained only for 4,096-token outputs, and generating responses of this length is not always practical.

  2. In Appendix D.12, the authors evaluate the spoofing attack of [2]. By design, this attack recovers per-context biases (conditioned on one or a few preceding tokens) and is not tailored to the watermark in this study. Could the authors adapt this attack to the PRC watermark to better demonstrate its robustness against spoofing?

One minor suggestion:

  1. The paper is difficult to follow for readers without prior knowledge of a key previous work [1]. This may also narrow the novelty of this study.

[1] Chowdhury, Aakash, et al. "Leakage certification made simple." Annual International Cryptology Conference. Cham: Springer Nature Switzerland, 2024. [2] Jovanović, Nikola, Robin Staab, and Martin Vechev. "Watermark stealing in large language models." arXiv preprint arXiv:2402.19361 (2024).

Questions for authors

Please address the first two weaknesses.

3 comments

Thank you for your helpful feedback and positive assessment of our contributions! We address the points you raised below.

1. We'd like to clarify that we only evaluated substitution robustness at 400 and 4,096 tokens (Appendix D.13), rather than attempting to identify the minimum length required for reliable detection under substitutions. These results therefore do not establish that 4,096 tokens are necessary. We agree with you that outputs of this length are not always practical, and we are extending our experiments to intermediate lengths and substitution rates to better characterize the required lengths. We expect to share the results in the next few days.

More broadly, our goal is to make LLM watermarking both undetectable and robust in practice. Unlike earlier undetectable watermarks such as Christ, Gunn, and Zamir (2024), the PRC framework of Christ and Gunn (2024) provides theoretical guarantees for both properties, including robustness to a constant fraction of random token substitutions. Our improvements in detection efficiency bring this framework closer to practical use. To evaluate the practical robustness advantage of PRCs over earlier undetectable LLM watermarks, we are adding a comparison with the original undetectable LLM watermark of Christ, Gunn, and Zamir (2024). We will revise the abstract and introduction to better explain this motivation and clarify the scope of our robustness claims.

2. Throughout our robustness evaluation, we use established attacks as common benchmarks to enable direct comparisons between PRCs and existing watermarking methods. For spoofing, we evaluate the foundational watermark-stealing attack of Jovanović et al. (2024). We additionally include a position-conditioned variant that learns token preferences at each generation position, rather than only from preceding-token contexts, to account for the positional structure of PRCs. The evaluated attacks are substantially less successful against PRCs than against the baselines (Appendix D.12).

We agree with you that more specialized attacks tailored to PRCs would also be valuable to investigate. One possible approach would be to learn the secret parity relations across token positions and investigate whether they can be exploited for spoofing. Appendix B.5 already identifies related PRC-specific distinguishing attacks as future work. We will extend this discussion to spoofing attacks in the revised manuscript, leaving their development and evaluation to future work.

3. We appreciate your suggestion to improve the exposition. We will make Section 3 (Methods) more self-contained by providing additional intuition for the original PRC construction and clarifying how our entropy-aware detection methods build on it.

Separately, we'd like to clarify that the foundational work already cited and discussed in our manuscript is Pseudorandom Error-Correcting Codes by Christ and Gunn (CRYPTO 2024), rather than Leakage Certification Made Simple, listed as [1] in your review.

Thank you for the detailed response. I understand that you are extending the experiments and expect to share the results. I will wait for these additional results, especially the first robustness check.

Thanks for your reply and patience! We’ve completed the additional robustness experiments and posted an overview of the results.

Our main finding directly addresses your concern about the practicality of the 4,096-token responses in our original evaluation: our PRC watermark achieves strong substitution robustness at substantially shorter, more practical response lengths. In our intermediate-length experiments, our PRC watermark with the posterior-mean detector achieves 92.6% TPR at 1,024 tokens under 10% random substitutions, at a target FPR of 0.1%.

Our detector comparisons also isolate the contribution of entropy-aware detection: the naïve parity-check detector for the original Christ–Gunn watermark achieves 79.6% TPR on the same attacked responses, at the same target FPR. This 13-percentage-point improvement demonstrates the benefit of our detection methods for recovering the watermark after substitutions.

Our comparison with CGZ further highlights the practical advance over the earlier undetectable watermark of Christ, Gunn, and Zamir (2024). Under 10% random substitutions, the CGZ baseline achieves only 14.4% TPR even at 4,096 tokens (security parameter λ = 30), at the same target FPR. Thus, under this attack, our PRC watermark achieves much higher detection with one-quarter as much text.

These results demonstrate strong substitution robustness well below 4,096 tokens, together with a clear practical robustness advantage over the CGZ baseline. We are finalizing manuscript updates to incorporate these findings and revise the abstract and introduction to highlight these improvements and clarify the response lengths needed under substitutions. We will share the revised manuscript here shortly.

We hope these results address your concern about robustness at practical response lengths, and would be grateful if you could reconsider your rating in light of the additional evidence. Please let us know if you have any remaining questions or suggestions.

Comments on the paper

6 comments

We thank all three reviewers for their thoughtful feedback. To address the shared concerns about robustness at practical sequence lengths, we are extending our substitution experiments to intermediate lengths and adding a comparison with the earlier undetectable watermark of Christ, Gunn, and Zamir (2024). We expect to share these results in the next few days, along with corresponding revisions to clarify the scope of our robustness claims.

AuthorsOpenReview ↗

Additional robustness experiments

We share below the results of our additional robustness experiments for our PRC watermark. This covers (1) substitution robustness at intermediate response lengths; (2) improvements from our entropy-aware detectors over naïve detection in the original PRC watermarking approach; and (3) a robustness comparison with the earlier undetectable watermark of Christ, Gunn, and Zamir (2024), referred to below as CGZ.

Highlighted results

  • (1) Strong substitution robustness at shorter, more practical response lengths. Our PRC watermark with the posterior-mean detector achieves 92.6% TPR at 1,024 tokens under 10% substitutions and 93.2% TPR at 2,048 tokens under 20% substitutions, using one-quarter and one-half of the 4,096-token length at which we previously demonstrated strong robustness.

  • (2) Our entropy-aware detectors improve the practical robustness of the original PRC watermarking approach. Compared with the naïve parity-check detector for the Christ–Gunn watermark, our posterior-mean detector improves TPR by 13.0 percentage points under 10% substitutions and 12.4 percentage points under 20% substitutions on the same 1,024-token attacked responses. These controlled comparisons demonstrate stronger detection after substitutions, addressing the practical need to recover watermarks when generated text has been modified before detection.

  • (3) A substantial robustness advantage over the original undetectable CGZ watermark under matched experimental conditions. At 4,096 tokens under 10% substitutions, our PRC watermark with the posterior-mean detector achieves 98.8% TPR, compared with 14.4% for the CGZ baseline (=30). Under 30% substitutions, the corresponding rates are 94.6% and 0.0%.

    Our PRC watermark also benefits substantially from more text under substitution attacks, while CGZ detection remains low. Under 10% substitutions, our posterior-mean detector improves from 41.6% TPR at 400 tokens to 98.8% at 4,096 tokens. In contrast, the CGZ baseline, which lacks a substitution-robustness guarantee, drops from 99.4% TPR on unmodified 400-token responses to 13.2% after substitutions, and remains at only 14.4% even with 4,096 tokens.

Alongside our existing quality, diversity, and black-box detection evaluations, these three advances strengthen the empirical case for our central goal: making robust, undetectable watermarking practical for language models.

AuthorsOpenReview ↗

(1) Strong substitution robustness at shorter, more practical response lengths.

Our original experiments in Appendix D.13 evaluated substitution robustness at 400 and 4,096 tokens. We have extended this evaluation to 1,024 and 2,048 tokens to better characterize the response lengths needed for reliable detection under random substitutions.

The results show that strong robustness does not require 4,096-token responses: detection already reaches 92.6% at 1,024 tokens under 10% substitutions.

Response lengthNo attack10% substitutions20% substitutions30% substitutions
400 tokens86.0%41.6%13.2%2.6%
1,024 tokens (new)99.4%92.6%61.8%20.8%
2,048 tokens (new)99.2%97.8%93.2%68.4%
4,096 tokens100.0%98.8%98.2%94.6%

Experimental settings: Qwen3-0.6B-Base with C4 RealNewsLike prompts, temperature 1, and our fixed PRC watermark with n=Tn=T, t=3t=3, and η=0.05\eta=0.05, where TT is the response length. Posterior-mean detection uses the analytical threshold targeting an FPR of 0.1%. Each setting uses 500 watermarked and 500 unwatermarked responses. The attack selects distinct token positions uniformly at random and replaces their token IDs with uniformly sampled vocabulary IDs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.