- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how randomness reuse affects the security of context-dependent LLM watermarks. It relates spoofing resistance to undetectability when one terminal verification query is allowed, then derives a security bound involving the probability of reusing a watermark-active context. For SynthID-Text, the authors develop an exact-context attack and a bucket-pooled attack that exploits publicly identifiable hash equivalences in the implementation. On 512 held-out prompts, the pooled attack achieves joint detector acceptance and automated quality pass rates of 82.6% on Gemma-2B and 88.7% on Gemma-9B. The paper also tests the role of these equivalences through controls and a hash patch.
Critical strengths
-
The bucket-pooled attack identifies a concrete implementation weakness and explains how observations can be shared across different contexts without recovering the watermark key.
-
The controls support the proposed mechanism: permuting bucket assignments removes detector acceptance, while patching the hash equivalence reduces acceptance from 107/128 to 0/128 using a new harvest.
-
The evaluation includes held-out prompts, multiple output lengths, two detectors, and automated quality checks. The authors clearly acknowledge the limitations of the public implementation and the reported success metric.
Critical weaknesses
-
The theory lacks a nontrivial instantiation at realistic query budgets. Theorem 1 uses a stronger setting than standard undetectability [1], while Theorem 2 leaves unspecified and uses a collision event that may become likely after many queries.
-
Formal spoofing success remains unverified. Appendix E.4 acknowledges that freshness cannot be checked against the complete harvest. The joint rates in Table 1 therefore do not establish success under the paper's own spoofing definition.
-
The strongest practical results rely on the implementation-specific bucket equivalence. In Table 8, the exact-context attack achieves 92.4% acceptance but a mean quality score of 1.54 and joint success of 0/512, limiting the empirical support for practical spoofing through general context reuse.
-
Related-work positioning and matched baselines are incomplete. The reuse mechanism deserves comparison with SEEK [3], spoofing-artifact detection [4], De-mark [5]. Scrubbing attacks and security trade-offs [6][7] provide further context. A controlled comparison with existing method [2] would help isolate the benefit of pooling.
-
The draft needs proofreading. “Sections 5 and 6 develops” should use “develop” (p. 2), and “The authors later application” needs a possessive (p. 7). The reference to Appendix F.4 for the and ablations also appears incorrect: that appendix studies .
[1] Miranda Christ, Sam Gunn, and Or Zamir. Undetectable Watermarks for Language Models. COLT, 2024.
[2] Nikola Jovanović, Robin Staab, and Martin Vechev. Watermark Stealing in Large Language Models. ICML, 2024.
[3] Huanming Shen, Baizhou Huang, and Xiaojun Wan. Enhancing LLM Watermark Resilience Against Both Scrubbing and Spoofing Attacks. NeurIPS, 2025.
[4] Thibaud Gloaguen, Nikola Jovanović, Robin Staab, and Martin Vechev. Discovering Spoofing Attempts on Language Model Watermarks. ICML, 2025.
[5] Ruibo Chen, Yihan Wu, Junfeng Guo, and Heng Huang. De-mark: Watermark Removal in Large Language Models. ICML, 2025.
[6] Abdulrahman Diaa, Toluwani Aremu, and Nils Lukas. Optimizing Adaptive Attacks against Watermarks for Language Models. ICML, 2025.
[7] Hanlin Zhang et al. Watermarks in the Sand: Impossibility of Strong Watermarking for Language Models. ICML, 2024.
Questions for authors
-
Can the authors give a nontrivial bound on for the evaluated SynthID configuration and characterize how grows with the query budget? Does the bound remain informative after accounting for initialization contexts and reuse across responses?
-
Can freshness be evaluated against the full harvested response set for the pooled attack? Please specify the closeness predicate and threshold, and report the fraction satisfying freshness, detection, and quality together.
-
Can the exact-context attack produce acceptable-quality spoofs against the patched implementation with a larger budget? How does the pooled attack perform under reference-model mismatch and domain shift? These results would help distinguish the impact of the implementation weakness from that of general randomness reuse.