- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a practical implementation of a robust, undetectable language model watermark based on Pseudorandom Codes (PRCs). By introducing an "entropy-aware detector" (using a posterior-mean scoring rule) that weights the watermark signal according to the model's token-level prediction uncertainty, the method drastically reduces the required detection length to approximately 450 tokens. Experimental validation on the Qwen model family demonstrates that this approach achieves reliable detection (90% TPR at 0.1% FPR) while preserving text quality and diversity, evading black-box detection tests, and resisting both watermark-stealing attacks and random token substitutions.
Critical strengths
-
Solves a Critical Bottleneck: Drastically reduces the required sequence length from thousands of tokens to ~450, making PRC watermarks viable for real-world deployment.
-
Novel Algorithmic Approach: Introduces an "entropy-aware detector" that leverages the model's token-level prediction uncertainty to maximize watermark signal detection.
-
Maintains Undetectability: Keeps the watermarked output distribution mathematically identical to the baseline, fully preserving text quality and diversity.
-
Comprehensive Validation: Provides strong empirical evidence on Qwen models, proving high detection efficiency, evasion of black-box tests, and robust resistance to stealing attacks.
Critical weaknesses
-
Critical Vulnerability to Desynchronization: The watermark relies entirely on token positional indices. As a result, it is highly susceptible to insertion and deletion attacks; even minor edits that shift the token order can desynchronize the parity checks and completely erase the watermark.
-
Fragility in Short Texts: Although the paper claims practicality at ~450 tokens, the robustness experiments (Appendix D.13) reveal that for 400-token outputs, a 30% substitution rate plummets detection to just 2.6%. The robustness claims are only practically valid for much longer texts (e.g., 4,000+ tokens).
-
Susceptibility to Paraphrasing Attacks: Because the method cannot survive insertions or deletions, standard real-world attacks, such as asking another LLM to rewrite or paraphrase the text (semantic-preserving transformations), will trivially and completely remove the watermark.
-
Limitation to Zero-bit Information (No User Identification): The proposed PRC encoding only signals the presence or absence of a watermark (zero-bit). It cannot embed multi-bit payloads, meaning it is impossible to identify specific users, sessions, or model versions from the watermarked text, severely limiting its utility for tracing misuse.
-
Compromised Theoretical Guarantees: To achieve shorter, practical detection lengths, the authors reduced the parity-check weight to . This parameter choice sacrifices the rigorous asymptotic undetectability guarantees established in the original PRC framework, leaving the scheme reliant entirely on empirical, rather than theoretical, validation.
Questions for authors
-
The robustness evaluation heavily focuses on token substitutions. How does the proposed watermark perform against desynchronization attacks (insertions/deletions) or semantic-preserving paraphrasing? Are there any mitigation strategies planned for these?
-
Table 15b shows that for 400-token outputs, the detection rate plummets to 2.6% under a 30% substitution rate. Given this fragility, what is the realistic minimum sequence length required for this watermark to be robust in real-world deployments?
7 comments
Thank you for the response. It confirms rather than resolves the main concerns. The evaluated scheme remains vulnerable to insertion and deletion, and paraphrase robustness has not been tested. Citing other edit-robust PRC constructions does not establish robustness for this implementation.
The claim of practical detection at around 450 tokens is also unsupported under the reported attacks. For example, at 400 tokens, detection falls to 2.6% with 30% substitutions, while the proposed intermediate-length results are still pending. The comparison to OpenAI’s watermark uses different attack rates and FPR targets, so it does not establish comparable robustness.
Finally, the authors acknowledge that falls outside the asymptotic undetectability guarantee. Black-box tests provide empirical evidence, not an equivalent guarantee; the possibility of extending the scheme to multi-bit payloads is likewise not a demonstrated capability.
These are central limitations for a practical, robust watermark, not peripheral issues. I do not consider the response sufficient to change my rating.
Thank you for your follow-up and your patience while we completed the additional robustness experiments! We’re excited to share new results demonstrating strong substitution robustness at substantially shorter, more practical response lengths. With the discussion deadline approaching, we’d really appreciate your time and consideration of this new evidence and our clarifications. We address your main concerns point by point below.
1. Practical robustness at shorter response lengths
The claim of practical detection at around 450 tokens is also unsupported under the reported attacks. For example, at 400 tokens, detection falls to 2.6% with 30% substitutions, while the proposed intermediate-length results are still pending.
Our intermediate-length experiments are now complete, and they demonstrate strong substitution robustness at substantially shorter, more practical lengths than 4,096 tokens. Our PRC watermark with the posterior-mean detector achieves 92.6% TPR at 1,024 tokens under 10% random substitutions, and 93.2% TPR at 2,048 tokens under 20% substitutions, at a target FPR of 0.1%.
The new detector comparisons also demonstrate the benefit of entropy-aware detection under substitution attacks. At 1,024 tokens under 10% substitutions, the naïve detector achieves 79.6% TPR on the same attacked responses, at the same target FPR. Our posterior-mean detector therefore provides a 13-percentage-point improvement.
We’d like to clarify that the approximately 450-token result concerns unmodified text, not text subjected to edit attacks. For context, OpenAI’s deployed watermark also shows reduced detection after substitution attacks at 400 tokens, as discussed in the comparison below. We’re finalizing revisions to the main text to distinguish this from the lengths required for strong detection after substitutions, which our new experiments characterize.
These results also support our broader goal of making robust, undetectable watermarking practical. Our comparison with CGZ, the earlier undetectable watermark of Christ, Gunn, and Zamir (2024), demonstrates a practical advance: our PRC watermark achieves substantially higher detection after substitutions with much less text. Under 10% random substitutions, CGZ achieves only 14.4% TPR even at 4,096 tokens, at the same target FPR of 0.1%. Thus, under this attack, our PRC watermark achieves much higher detection with one-quarter as much text.
An overview of all three experiments and their main findings is available in this update.
Thank you for your detailed feedback and for recognizing the practical improvements achieved by our work! We address your two questions first, followed by the remaining concerns.
Question 1. We agree that our current implementation is vulnerable to desynchronization, since the code that we use involves parity checks that depend on token positions. However, this is not an inherent limitation of PRC watermarking: Golowich and Moitra (2024) and Christ et al. (2026) construct PRCs that are robust to edits (insertions and deletions). Watermarks using these edit-robust PRCs are not yet practical, as their rate of error tolerance is too low. Our work improves the error tolerance of PRC-based watermarks, making the parity-check-based PRC that we use practical. While further error tolerance is required to use these edit-robust PRCs in practice, we make progress on this front.
Extensive paraphrasing is also a general challenge for text watermarks, including deployed systems. Both Google's SynthID-Text and Anthropic's Claude watermark acknowledge that substantial rewriting can weaken or remove their watermark signals. We have not evaluated paraphrasing attacks on our implementation and leave robustness to these transformations to future work.
Question 2. The minimum sequence length required for robust detection depends on the substitution rate and desired detection power. As discussed in our response to Reviewer V3rL, our existing substitution experiments only evaluate 400 and 4,096 tokens, so they do not establish that 4,096 tokens are necessary. We are extending these experiments to intermediate lengths to better identify practical operating points under different substitution rates.
As a practical reference point, the recent evaluation of OpenAI's text watermark reports approximately 66% detection under 10% synonym replacement and 17% under 25% replacement for 400-token ELI5 responses, at a target FPR of 1%. We are also evaluating PRC robustness on ELI5 to help determine the sequence lengths needed to achieve comparable detection rates. We expect to share these additional results in the next few days.
Zero-bit encoding and user identification. Our implementation indeed uses a zero-bit PRC, but this is not a fundamental limitation of PRCs. Christ and Gunn (2024), Section 6.1, provide a generic transformation from zero-bit to multi-bit PRCs, and Gunn et al. (2025) demonstrate multi-bit PRC watermarking for images. These techniques could in principle support user or session identification in LLM watermarking, although additional payloads may require longer outputs for reliable detection. We leave their practical implementation to future work.
Theoretical guarantees. Indeed, as discussed in Appendix B.5, our practical choice of parity-check weight (t=3) falls outside the parameter regime of the original asymptotic undetectability guarantee from Christ and Gunn (2024). Our evidence for undetectability at these parameters is therefore empirical, supported by quality-preservation experiments and black-box distinguishing tests. Separately, the statistical FPR guarantee of Theorem 3.3 remains valid at t=3, under its stated assumptions. We will make this distinction clearer in the main text.