- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how to design token-level watermark sampling rules while limiting changes to the language model's output. With the hashing rule and detector fixed, it maximizes the expected token score under constraints on either each sampling distribution or its average over random scores. The framework recovers several existing methods and introduces new sampling rules based on chi-square and perplexity constraints.
Critical strengths
-
The framework connects existing methods. Connecting KGW, AAR/KTH, and SynthID to different constraints helps explain their sampling rules and differences.
-
The new methods show measurable gains. With matched binomial scores and the same detector, Table 3 reports 98.8% detection at a PPL budget of 4 for soft PPL, compared with 73.5% for KGW.
-
The experiments cover several important settings. The paper evaluates detection, text quality, diversity, and text-editing attacks. Comparisons with matched scores and detectors help show the contribution of the sampling rule.
Critical weaknesses
-
The claims about optimal detection need stronger support. Maximizing the expected token score does not generally maximize detection power at a fixed false-positive rate, even for a fixed sum-based detector. The connection to the error-based guarantees in [1, 2] needs to be established.
-
The two optimization problems are not generally equivalent. Equation (2) imposes a separate hard constraint for every score vector , which generally requires a multiplier depending on , whereas Equation (3) uses one constant penalty parameter.
-
The KL calculation does not match the theoretical constraint. In the supplementary metrics.py, the hard-KL calculation uses instead of and divides by the vocabulary size for a one-dimensional input. Whether this code produced the reported constraint comparisons needs clarification.
References
[1] Li et al. A Statistical Framework of Watermarks for Large Language Models: Pivot, Detection Efficiency and Optimal Rules. The Annals of Statistics, 2025.
[2] Li et al. Robust Detection of Watermarks for Large Language Models Under Human Edits. Journal of the Royal Statistical Society Series B, 2026.
Questions for authors
-
Connection to detection power. What guarantee on detection power or Type II error follows from maximizing the expected token score? Do the gains over AAR/KTH remain when that baseline uses the detection rules from [1] or Tr-GoF from [2]?
-
Equivalence of the optimization problems. Under what additional conditions are Equations (2) and (3) equivalent for hard constraints? How can a single penalty parameter enforce the same constraint level across different score vectors?
-
Verification of the KL results. Was the supplied metrics.py used for Figures 2 and 5 and Table 6? Please verify the KL direction and normalization, and explain whether correcting the implementation changes the reported conclusions.
2 comments
Thanks for the detailed response. My concerns have been addressed. I have thus raised my rating .
We are happy to see that the reviewer has found our work clearly presented, informative, and sound. We address their remaining concerns below, and have updated our submission. All changes in the submission are colored in blue for clarity.
Q1: Does the framework provide guarantees on detection power? Why do the authors use a sum-based objective?
We agree with the reviewer that maximizing EG[G⋅q(G)] does not generally maximize detection power at a fixed FPR. Our theorems establish optimality with respect to the constraint and objective only, not to the power. Hence, we don’t provide a guarantee on the detection power. Prior works that provide guarantees [1,2] need to make assumptions on the LLM next-token distribution which we argue are impractical (e.g. [1]’s Δ-regularity requires every next-token probability to be at most 1−Δ, which fails on low-entropy tokens).
Importantly, we opted for a sum-based objective (i) because it captures existing schemes (and even new ones such as OpenAI’s TextGrain [3], now in Section 4.1) and is therefore practically relevant, and (ii) because maximizing power directly, as in [1,2], requires making unrealistic assumptions on the LLM next-token distribution which, as we see in Q2, does not yield better detection than a well-chosen sum-based detector and limits practicality.
Q2: For AAR, how does the sum-based detector compare with the optimal detector?
Thanks to the reviewer's comment, we now compare in Appendix F.1 (and in the table below, for distortion-free AAR) the AAR detector with the detector from [1] (with fixed Δ=0.1), as well as the sum-based Gamma detector from [4, 5]. We find that the Gamma detector performs on par with [1]’s detector: it is slightly better at 1% and 0.1% FPR, and slightly worse at 10−6 FPR. This further supports that sum-based detectors are appropriate for building powerful schemes.
Q3: Under which conditions are equations 2 and 3 equivalent?
We don’t consider Equations 2 and 3 to be equivalent. In Section 3, we highlight that the shape of the optimal solution is the same, e.g. hard-KL constraint in both cases leads to exponential tilting, though in Equation 2 the multiplier may depend on g. Thanks to the reviewer, we have now updated our phrasing in Section 3 to make our statement clearer.
Q4: Which KL metric was used for plotting Figure 2 and 5?
We thank the reviewer for actively checking our code. We indeed found that Figures 2 and 5 used KL(p∣q) instead of KL(q∣p). We have now updated Figures 2 and 5 as well as Table 8 (previously Table 6) to use KL(q∣p), summed over the vocabulary without normalization by its size. Importantly, our conclusion remains the same: Red-Green watermarks are optimal when it comes to the trade-off between detectability and Hard KL.
[1] Li, Xiang, et al. "A statistical framework of watermarks for large language models: Pivot, detection efficiency and optimal rules." Annals of statistics 53.1 (2025): 322
[2] Li et al. “Robust Detection of Watermarks for Large Language Models Under Human Edits”. Journal of the Royal Statistical Society Series B, 2026
[3] Li, Xiang, et al. “textGrain: Entropy-Calibrated Watermarking for Language Model Text.” OpenAI technical report (2026)
[4] Sander, Tom, et al. “TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection.” Advances in Neural Information Processing Systems (2026)
[5] Fernandez, Pierre, et al. “Three Bricks to Consolidate Watermarks for Large Language Models.” IEEE International Workshop on Information Forensics and Security (2023)