- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates an interesting question that whether the LLM watermark detectors could be public. The authors propose a split-key watermarking framework that divides the watermark signal between a public key and a private key. This method include two-stage detection mechanism that combines watermark verification with a statistical test for discrepancies between two key signals, which allowing attack manipulation to be identified.
Critical strengths
1.Important and not well-discussed problem:The paper try to address an important and relatively underexplored problem under LLM watermark topic. Given the growing demand for verifiability of AI-generated content, this is a timely and practically relevant research direction.
2.Comprehensive threat analysis: The authors systematically evaluate watermark removal and forgery attacks under different levels of detector access.
3.Good presentation: The presentation is clear.
Critical weaknesses
Although I appreciate the authors' efforts to address this interesting problem, I still have several following major concerns:
1.Weak threat model:The authors primarily focus on rephrasing and word-level edits, which have already been shown to be relatively weak attacks in prior work. The authors should evaluate it against stronger and more recent model-based watermark removal and forgery attacks, such as SIRA[1], DIPPER[2], and watermark stealing attack[3] to prove the effectiveness of its methods.
2.Exchangeability assumption: The proposed mechanism relies on a token-wise sign-flip permutation test, which assume exchangeability between the public and private watermark signals. However, I question that marginal symmetry does not necessarily imply the joint sign-flip invariance required for an exact permutation test. Token-level watermark scores may exhibit dependencies due to overlapping contexts. Such dependencies could potentially violate the underlying assumptions and increase false-positive rates. The authors should provide a more rigorous theoretical justification and evaluate its robustness under correlated token scores.
3.Missing Relevant Prior Work: The related work discussion overlooks several closely related studies such as Bileve[4] and PVMark[5]. PVMark leverages zero-knowledge proofs to enable third parties to verify that the watermark detection process has been executed correctly without revealing the secret key. Bileve explores a multi-level watermarking mechanism for provenance verification and tamper detection. Discussing these works would help clarify the novelty of the proposed method and better position its security relative to existing approaches.
4.Generalizability: The experiment mainly based on Gumble-max watermark. Even though the author add syn-text and maryland in sec 4.5, these method still belong to the same predefined pseudorandom sampling mechanisms. It remains unclear whether the proposed split-key framework can generalize to learning-based watermarking methods, such as SIR which prove to be more robust.
[1]Cheng, Yixin, et al. "Revealing weaknesses in text watermarking through self-information rewrite attacks." ICML (2025).
[2]Krishna, Kalpesh, et al. "Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense." Advances in neural information processing systems 36 (2023): 27469-27500.
[3]Jovanović, Nikola, Robin Staab, and Martin Vechev. "Watermark stealing in large language models." ICML (2024).
[4]Zhou, Tong, et al. "Bileve: Securing text provenance in large language models against spoofing with bi-level signature." Advances in Neural Information Processing Systems 37 (2024): 56054-56075.
[5]Duan, Haohua, et al. "{PVMark}: Enabling Public Verifiability for {LLM} Watermarking Schemes." 35th USENIX Security Symposium (USENIX Security 26). 2026.
[6]Liu, A., Pan, L., Hu, X., Meng, S., & Wen, L. (2024). A Semantic Invariant Robust Watermark for Large Language Models. ICLR (2024).
Questions for authors
See weakness.
3 comments
Thanks for the effort
Thanks for your reply! I've carefully reviewed your response, which addresses most of my concerns. The additional robustness experiments and the SIR idea would further strengthen the paper. I'm happy to increase my score.
Thank you once again for your careful reading, and for suggesting these additions that improve the paper. We would be happy to answer any further questions.
Thank you for the constructive review and recognition of our work’s importance.
W1: "stronger and more recent model-based watermark removal and forgery attacks, such as SIRA, DIPPER, and watermark stealing"
We ran DIPPER (XXL) and SIRA (Llama-3.1-8B), at several mask ratios as published (no-box) and white-box (masking the highest public-score tokens):
Both are about as strong as our no-box rephrasing at T=1.5, at higher distortion. White-box SIRA stays within 1.4 points of no-box at 50-90% masked, and is weaker at 30%, where it leaves more private signal and starts to be flagged (ADR 1.8%). Watermark stealing is in App. D.8: public access only speeds up learning of the public channel, and beats no-box only for very small corpora, at low ASR_net. So public access offers little advantage against the full pipeline (Figs. 4 and 6).
W2: "Token-level watermark scores may exhibit dependencies due to overlapping contexts."
The test does require joint sign-flip invariance, not just marginal symmetry. The routing gives us this hypothesis since each token goes to a key by an independent fair coin, so swapping the labels of any subset of tokens leaves the joint score distribution unchanged, whatever the dependence between tokens. Overlapping windows give distinct hashes, and repeated contexts are deduplicated with one global history across both channels (App. C.1). Beyond Fig. 7, the test keeps its nominal false-positive rate on 10,000 human ELI5 answers (0.11% flagged at 10−3). Exchangeability is needed only for exactness: the test extends to unequal routing ratios as an asymptotic test (response to fMq6).
W3: "The related work discussion overlooks [...] Bileve and PVMark."
Neither method enables public detection, and both are complementary to our work. PVMark uses zero-knowledge proofs so that the provider can prove it ran detection correctly without revealing its key, but third parties still cannot detect without the provider. Regardless, a PVMark proof could be attached to our private verification. Bileve pairs a fragile signature with a robust watermark, so its tampering evidence flags any edit, benign or malicious, while our test flags only edits that target the public key. We have updated the manuscript to discuss both.
W4: "whether the proposed split-key framework can generalize to learning-based watermarking methods, such as SIR"
Gumbel-max, Maryland and SynthID are different mechanisms, and deployed watermarks (TextSeal, SynthID, textGrain) all use hash-based pseudorandom sampling. Our construction only needs independent keys, so it generalizes to any method with a secret key. Learning-based methods are less direct, as their scores have no exact null. We therefore built a split-key SIR: the authors' network is shared, and two independent random token-to-dimension tables serve as the public and private keys, with fair-coin routing. Each channel has an exact z-test under a random table, and the tampering test swaps keys per distinct token. Fused TPR is 96.3% at 10−3, with 0.02% false positives on 5k human answers (δ=2, same ELI5 setting, rates in %):
Public access helps the attacker more than with TextSeal: white-box rephrasing raises ASR_net from 63.3% to 76.5% for SIR, but lowers it from 85.3% to 62.8% for TextSeal. White-box rephrasing reaches 100% ASR_pub for both TextSeal and SIR, but ADR is lower for SIR (23.2% vs. 37.1%). We hypothesize that SIR's semantic nature correlates the scores of the two channels, since both read the same semantic value vector, reducing the imbalance detected by the tampering test.
We hope the stronger attacks and the test's validity address your concerns, and that you will consider raising your score.