- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores digital watermarking techniques for content generated by structured Large Language Models (LLMs), focusing specifically on mathematical reasoning and code generation—scenarios where indiscriminate biasing of token selection could compromise answer correctness or program functionality. The authors propose SemaSync, a semantic-state-based watermarking framework. This framework derives watermarking states from normative anchors inherent in mathematics and code (such as numbers, operations, literals, and control-flow events), constructs dynamic "green-list token" partitions via a multi-anchor voting mechanism, and employs distribution-aware carrier selection rules to apply watermarking biases only at suitable generation points.
The researchers evaluated the method using Qwen3.5-9B and Llama-3.1-8B-Instruct models on the GSM8K and MBPP datasets, comparing it against existing watermarking baselines in both "raw generation" and "LLM-based paraphrasing" scenarios. Results demonstrate that SemaSync achieves excellent detection performance while maintaining superior task utility; the paper also includes ablation studies regarding semantic state construction, multi-anchor voting, and carrier selection mechanisms. Overall, this work presents a well-motivated and practically valuable watermarking scheme for structured LLM outputs that offers performance improvements over baseline methods.
Critical strengths
-
Problem formulation driven by clear motivation. This paper identifies the practical limitations of existing token-level watermarking methods in mathematical reasoning and code generation tasks: indiscriminate watermark-induced bias can compromise the correctness of the generated content, whereas mechanisms relying on token history are vulnerable to failure upon text rewriting. Constructing watermark states using semantic and structural anchors offers a sound, task-aligned solution to this challenge.
-
This paper encompasses a wide range of evaluation dimensions, including two types of structured generation tasks, two Large Language Model (LLM) backbones, multiple baseline watermarking methods, original versus rewritten outputs, task utility, runtime overhead, and targeted ablation studies. These extensive evaluations provide compelling evidence that the proposed method achieves excellent detection performance without significantly degrading generation quality.
Critical weaknesses
-
Limited conceptual innovation. Although the problem formulation is well-motivated, the proposed approach primarily operates within the existing "green-list" watermarking framework, combining task-specific semantic anchor extraction, multi-anchor voting, and selective watermark embedding. While these design choices are sound, they represent engineering-level improvements rather than the substantial conceptual breakthroughs required to meet ICLR standards.
-
Performance gains are not robust. Across the various tasks, models, and evaluation metrics involving text rewriting, SemaSync does not consistently outperform baseline methods. For instance, when using the Mistral model for rewriting, its AUROC on the Qwen model's MBPP task falls short of the Unigram and SWEET methods; similarly, on the Llama model's GSM8K task, it underperforms compared to the Unigram and SIR methods. The method also fails to demonstrate a significant advantage when using the more powerful Luna model for rewriting. Furthermore, compared to the baseline method using a simple Token + Gate approach, the AUROC for the rewritten MBPP task improved by only 0.012. These results indicate a lack of sufficient evidence to support the broad claim that the method achieves robust semantic-state watermarking.
Questions for authors
Clarification Requests
-
The paper’s main novelty appears to be constructing dynamic green lists from task-specific semantic/structural anchors, rather than token history, and skipping unsuitable carrier positions through a green-mass gate. Can the authors more clearly explain what conceptual capability this provides beyond existing semantic watermarking and selective-embedding methods, such as SemStamp and SIR, rather than viewing it as a task-specific engineering adaptation?
-
I view the green-mass gate primarily as a threshold-based heuristic for avoiding large deviations from the base distribution, rather than a core conceptual contribution. More importantly, measures only the total base probability mass of the green set and does not explicitly model semantic correctness. Why should this mechanism adequately protect correctness-critical tokens and structures, such as numerical values, operators, function signatures, and control-flow decisions? In difficult mathematical reasoning, both the correct answer and plausible incorrect alternatives may have non-negligible probability.
-
The direct
Token + Gateablation shows only a modest rewrite-time gain on MBPP (AUROC0.682to0.694). To establish whether this improvement supports the broader claim of robust semantic-state watermarking, could the authors provide three targeted experiments: (i) multi-seed results with confidence intervals for the direct SemaSync versusToken + Gatecomparison, to assess statistical reliability; (ii) the same comparison under a stronger or qualitatively different rewriter, to test whether the gain generalizes beyond the current rewrite protocol; and (iii) results under alternative reasonable anchor ontologies, to quantify how strongly the claimed gain depends on task-specific anchor design?
Thank you for these questions.
Q1: Capability beyond existing semantic watermarking.
SemStamp uses an auxiliary sentence encoder and repeatedly samples sentences until their embeddings fall in accepted semantic regions. SIR uses an auxiliary encoder and a trained watermark network to map prefix semantics to token-level logit biases. SemaSync instead normalizes explicit mathematical or code events into anchors: when rewriting preserves the active anchors, the detector can reconstruct the same token partitions despite changes in wording. It needs neither sentence rejection sampling nor auxiliary neural networks, but requires domain-specific extraction rules and generator replay for detection.
Under the shared Llama-MBPP protocol in Table 27 (one H100, batch size 1, 50 prompts), generation takes 0.535 s/sample for SemaSync versus 6.905 for SemStamp and 1.405 for SIR: 12.9× and 2.6× speedups, respectively.
Q2: Does green mass protect correctness?
Section 3.3 gives the following relation under fixed sampling support:
qt(δ)=1−qt+eδqteδqt.
At low qt, the bias promotes choices that the base model considers unlikely: for example, with δ=3, green probability rises from 5% to 51.4%. The lower bound avoids such interventions, which can harm task correctness. At high qt, clean text already favors green tokens, and qt(δ)−qt approaches zero as qt→1. These positions add little distinction between clean and watermarked text and can contribute positive clean scores under the nominal-γ statistic, motivating the upper bound.
Table 4 supports these distinct roles. Removing the lower bound reduces MBPP Pass@1 from .538 to .464 and GSM8K accuracy from .928 to .904. Removing the upper bound leaves MBPP Pass@1 similar (.542 versus .538), but lowers raw AUROC from .884 to .588.
Q3: Contribution of semantic states.
We chose Mistral as a qualitatively different rewriter for outputs generated by Qwen and Llama because it comes from a model family distinct from both. To assess the contribution of semantic states while controlling for the gate and voting configuration, we added a comparison between SemaSync K3 and Token-vote3+Gate. Both use the same gate rule and are configured for three-voter majority voting; their votes come from semantic anchors and token history, respectively. The table reports AUROC after Mistral rewriting and paired 95% confidence intervals for the difference.
After Mistral rewriting, SemaSync K3 significantly outperformed Token-vote3+Gate on GSM8K (+0.061 AUROC), while the three-seed gain on MBPP was small (+0.002) and not statistically significant. Our anchor rules are heuristic, and the code extractor can miss structures during generation. For example, it can miss an anchor on the final line when no newline follows. Even with these simple rules, semantic anchors improved detection on GSM8K. We also transferred the original GSM8K rules to MATH and the original MBPP rules to HumanEval+ without re-tuning. After Mistral rewriting, SemaSync achieved 0.814 AUROC on MATH, significantly outperforming all eight external baselines. On HumanEval+, its AUROC was close to SWEET’s (0.620 vs. 0.625), and it achieved the highest pass@1 among the evaluated watermarking methods (0.750).