- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SoptiMark, an improved version of Green/Red watermark (KGW). The method introduces two key components. First, a Cumulative Probability Mask maximizes the first-order watermark detection signal by partitioning the vocabulary such that the green and red lists each contain approximately half of the model's predictive probability mass. Second, Embedding Clustering assigns semantically similar tokens to shared hash keys, improving watermark consistency under token substitutions and paraphrasing. Experiments across multiple LLMs demonstrate that SoptiMark achieves stronger detection performance and greater robustness against text modifications than existing watermarking methods.
Critical strengths
I think the biggest strength of this paper is that it tries to improve watermark from the perspective of vocabulary partitioning and watermark key generation, which I find to be a very interesting and promising direction. In particular, I like the idea of incorporating semantic information into watermark key generation rather than relying solely on token identities. Regardless of the actual performance improvements, I think this is a good starting point for improving the robustness of existing watermarking methods.
Critical weaknesses
-
Limited practical significance of the theoretical analysis. While the theoretical result suggests that balancing the probability mass between green and red tokens maximizes the detection signal, I am not fully convinced of its practical significance. Beyond motivating a relatively straightforward balancing strategy, I don't see much additional insight or guidance (especially for practice) provided by this analysis.
-
Difficulty in controlling semantic similarity. I think the biggest challenge is how to properly define and control semantic similarity. Simply clustering token embeddings does not guarantee that tokens within the same cluster have similar meanings. There is also a clear trade-off: using smaller clusters makes the method closer to token-identity-based watermarking, while larger clusters may group semantically different tokens together. I feel this trade-off is difficult to control in practice and is not sufficiently addressed in the paper.
Questions for authors
-
While the theoretical analysis shows that balancing the probability mass between green and red tokens is optimal, what is the practical significance of this result? (see weakness 1)
-
How can we properly control semantic similarity through clustering? (see weakness 2)
-
The authors mention in the Introduction that logit-based watermarking has low overhead. However, since semantic meaning is context-dependent, is offline clustering based on static token embeddings sufficient? If context-aware clustering is needed (since the token semantic should depend on the context), wouldn't that introduce substantial computational overhead and potentially undermine the low-overhead advantage of traditional watermarking methods?
6 comments
Response to Reviewer DtPT (2/4)
W2/Q2 (a): The role of embedding clustering.
We would like to clarify that the primary objective of embedding-based clustering is not to guarantee semantic homogeneity within each cluster, but to improve watermark reconstruction consistency under content-preserving token substitutions. Specifically, embedding similarity serves as a proxy for identifying tokens that may substitute for one another, allowing them to share the same watermark key. Therefore, the relevant design criterion is whether clustering increases green/red membership consistency under plausible substitutions.
To test whether clustering improves this consistency when the conditioning token changes, we conducted a detector-side reconstruction diagnostic under Word-S/50 on 50 texts per model using one fixed watermark key. We evaluated all aligned, unchanged tokens whose immediate predecessor token ID had changed. The variants with and without clustering use the same texts, model probabilities, and positions, retaining the probability-dependent mask construction and changing only the seeding rule. Membership consistency is the percentage of evaluated tokens that retain the same green/red status in the masks reconstructed before and after editing. Clustering increases this consistency by 10.00, 8.16, and 5.85 percentage points on Qwen3, LLaMA, and OPT, respectively. Clustering therefore helps tokens retain their green/red membership when the preceding token is replaced by a synonym.
| Model | Positions | Membership consistency w/o clustering (%) | Membership consistency w/ clustering (%) | Gain (pp) |
|---|---|---|---|---|
| Qwen3-4B | 6,621 | 48.74 | 58.74 | +10.00 |
| LLaMA-3-8B | 5,958 | 49.85 | 58.01 | +8.16 |
| OPT-6.7B | 5,773 | 50.42 | 56.28 | +5.85 |
Response to Reviewer DtPT (3/4)
W2/Q2 (b): Controlling the clustering granularity trade-off.
To address your concern about controlling clustering granularity in practice, we conducted additional experiments showing why smaller is not always preferable: its modest gains in reconstruction consistency can come with increased spoofing risk. Increasing can separate related tokens, while decreasing can group unrelated tokens together; neither direction resolves both concerns. Our choice of is empirical. To provide evidence for assessing this compromise, we added a sensitivity experiment on Qwen3, varying from 32 to 2048 while keeping the same 50 Word-S/50 texts, model probabilities, and 6,621 evaluated positions fixed.
| Number of clusters | Key preservation (%) | Token membership consistency (%) |
|---|---|---|
| 32 | 30.92 | 60.31 |
| 64 | 30.71 | 60.84 |
| 128 | 27.16 | 56.94 |
| 256 | 24.78 | 58.65 |
| 512 | 25.27 | 58.74 |
| 1,024 | 23.62 | 56.09 |
| 2,048 | 21.10 | 56.10 |
The results show that coarser clustering generally improves key preservation, but yields smaller and non-monotonic gains in token membership consistency. For example, reducing from 512 to 64 increases key preservation from 25.27% to 30.71%, while membership consistency increases from 58.74% to 60.84%. This suggests that substantially increasing key sharing does not necessarily produce a comparable improvement in reconstruction consistency.
However, coarser clustering may also increase spoofing susceptibility. We simulated a frequency-based spoofing attack on Qwen3-4B, keeping the probability-dependent green-list construction and the watermark bias of 1.25 unchanged across seeding schemes. The simulated attacker knows the model and cluster assignments, but not the watermark key, and receives no detector feedback. At each observation budget, the simulated attacker receives equal numbers of watermarked and unwatermarked reference tokens and uses their conditional frequency ratios to bias the generation of forged texts. ASR is the percentage of forged texts classified as watermarked.
| Number of clusters / Seeding scheme | Spoofing ASR at 2,048 tokens (%) | Spoofing ASR at 8,192 tokens (%) |
|---|---|---|
| 32 | 24.00 | 56.00 |
| 64 | 11.00 | 31.00 |
| 128 | 6.00 | 16.00 |
| 256 | 7.00 | 12.00 |
| 512 | 2.33 | 5.67 |
| Random partitioning | 3.00 | 3.67 |
At both evaluated budgets, has a spoofing ASR close to random partitioning, while all tested coarser configurations have a higher ASR.
Taken together, these results help make the granularity trade-off more concrete. Coarser clustering generally preserves more valid substitution pairs, but its gains in membership consistency are modest and may come with increased spoofing susceptibility. We therefore assess our empirical choice of by whether it retains a useful consistency benefit while keeping spoofing ASR close to the baseline across the evaluated budgets.
Response to Reviewer DtPT (1/4)
We appreciate the time and effort you have put into evaluating our work. Your questions prompted us to examine the clustering design more closely and add experiments to clarify its practical benefits and trade-offs. We provide our responses below.
A practical implication of Theorem 4.1 is quantitative guidance for quality-constrained watermark design. When text-quality requirements limit the acceptable watermark bias, the theorem helps determine how much detection signal can be retained by adjusting the partition instead. Ignoring higher-order terms, retaining at least a fraction r of the maximum first-order signal requires T−1∑t(PG,t−1/2)2≤(1−r)/4. If every position attains the same mass p, equality gives p=(1±1−r)/2. This allows us to choose a partition target according to the desired expected z-score increase, rather than always maximizing the detection signal. The corresponding PPL measurements show the quality cost associated with each target and help identify settings that meet the application’s quality requirements. Our additional experiment reports both quantities across different target masses. For example, the target mass 0.6581 yields an observed mean displacement of 1.4309 with a fixed-context PPL of 2.2174, compared with 1.3123 and 2.3068 for random partitioning. Thus, the analysis provides candidate targets for balancing signal requirements and quality costs, while the quality effects are assessed empirically rather than guaranteed by the theorem.
We tested this guidance by varying the target signal fraction and measuring both the resulting detection-score displacement and fixed-context PPL. We used the upper branch, ptarget=(1+1−r)/2, to construct the cumulative partitions, with token discreteness limiting how closely each target can be reached. This experiment illustrates how the analysis guides the choice of partition targets for different signal requirements and supports empirical assessment of their quality costs, beyond simply recommending a balanced split.