- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Grammar-Guided Code Watermarking with Green Temperature (GTCW) for code generation models. To prevent syntactic violations caused by standard vocabulary-wide partitioning, GTCW integrates context-free grammar parsing to constrain the watermark support strictly to admissible tokens. At eligible high-entropy positions, it employs green temperature scaling to sharpen green token logits according to the model's existing preferences rather than applying a uniform additive bias. The authors evaluate GTCW across five instruction-tuned models and five benchmarks covering Python, C++, Java, and Go, demonstrating improved detection metrics while maintaining functional correctness close to the base model.
Critical strengths
1.Restricting watermark candidate selection to grammar-admissible tokens provides a principled solution to mitigate syntax violations during code generation, with the ablation study clearly showing that grammar constraints preserve execution pass rates while green temperature enhances detection separability.
2.The empirical validation is comprehensive and well-designed, covering multiple model architectures, four programming languages, realistic code transformations, and dedicated evaluations conditioned on functionally correct outputs.
Critical weaknesses
1.The generation latency overhead is severe for practical deployment; token throughput drops from 65 tokens/s to 17.24 tokens/s, increasing completion time nearly threefold from 4.28s to 11.50s per snippet.
2.The detection procedure exhibits high operational dependencies and vulnerability under post-processing. It requires full white-box forward passes with the original model checkpoint to reconstruct the entropy gate, and the watermark appears fragile against semantic rewriting, with detection performance dropping to near random guessing (AUROC 51.4%, TPR@5% 6.8%) under LLM rewriting.
Questions for authors
1.Could the authors provide a detailed runtime breakdown for the 11.50s latency in Table 7 to clarify how much overhead stems from CFG state transitions versus logit reweighting, and whether standard structured decoding engines (e.g., XGrammar) could feasibly bridge this throughput gap?
2.In the replay algorithm, detection terminates upon the first parse error or inadmissible token; how sensitive is the detector to minor syntax modifications or typos that might occur early in the code snippet?
3.Since typical code generation pipelines sample at lower temperatures (e.g., T_0 = 0.2), how does green temperature behave when applied to an already peaked distribution?
2 comments
Request for Discussion Engagement
Thank you for your time and constructive feedback. We have carefully addressed your concerns through detailed clarifications and additional experiments.
As the discussion period approaches its end, we would greatly appreciate it if you could review our responses and newly added results. In particular, we have:
- Q1: Added a detailed runtime breakdown distinguishing grammar decoding from watermark computation and discussed potential optimizations.
- Q2: Clarified the detector's sensitivity to syntax-breaking modifications and the limitations of grammar replay and white-box detection.
- Q3: Conducted additional experiments at lower sampling temperatures, including 0.2, demonstrating that GTCW retains strong detectability under more peaked distributions.
- W2: Further discussed rewriting robustness, distinguishing localized code transformations from extensive LLM rewriting.
We hope these clarifications and results adequately address your concerns. If you find our responses satisfactory, we would be grateful if you could reconsider your evaluation, including the overall score.
Please let us know if any concerns remain. We would be happy to address them.
We thank the reviewer for recognizing the strengths of our approach and evaluation. We address the concerns below.
Q1. Generation Overhead and Optimization Potential
We profiled 164 HumanEval generations with Qwen2.5-Coder-3B-Instruct to separate model inference, grammar decoding, and watermarking costs.
Grammar decoding accounts for substantial overhead, with additional costs from GTCW. However, fine-grained CUDA synchronization increases GrammarOnly and GTCW runtimes to 49.47 and 49.63 ms/token, nearly eliminating their gap. The instrumented entropy gating, green partitioning, and logit reweighting operations account for only ~0.58 ms/token. This suggests that GPU-CPU synchronization and execution serialization may contribute substantially to the observed overhead, although their precise contributions require further investigation.
Our implementation uses a custom Rust grammar engine without extensive runtime optimization. Potential improvements include reducing unnecessary GPU-CPU synchronization, optimizing GPU-side watermark operations, adopting efficient grammar engines and caching, and overlapping grammar processing with model inference, as explored in XGrammar (Dong et al., 2025). Since profiling uses a relatively small 3B model, grammar-related overhead may also be proportionally smaller for larger models. Our primary contribution is the watermarking mechanism rather than decoding-engine optimization.
Q2. Sensitivity to Syntax Modifications
Grammar replay can terminate upon encountering an inadmissible token, particularly when syntax-breaking modifications occur early in the generated code. We acknowledge this limitation. However, our primary setting is functional code generation, where syntactically invalid modifications generally compromise the intended functionality and therefore fall outside our main robustness threat model. In contrast, grammar-preserving modifications do not necessarily terminate replay, although they may still weaken watermark evidence. Our robustness evaluation focuses on code transformations that preserve executable functionality, rather than arbitrary syntax-breaking typos.
Q3. Behavior at Low Sampling Temperatures
GTCW, like other sampling-based watermarking methods, requires some uncertainty in token selection. Under nearly deterministic decoding, fewer alternative tokens are available for watermark insertion. We evaluated sampling temperatures of T0=0.2,0.5, and 1.0 on 164 HumanEval+ Python tasks using Qwen2.5-Coder-3B-Instruct. For GTCW, we maintained the temperature ratio Tg=T0/2.
Even at T0=0.2, GTCW achieves 76.49% AUROC and 50.00% TPR@5%FPR, outperforming the strongest existing baselines by 9.82 and 29.27 percentage points, respectively. Across all tested temperatures, GTCW consistently provides stronger detection while maintaining competitive Pass@1. These results suggest that green-temperature scaling remains effective even under sharply peaked token distributions.
W2. Detection Dependencies and Rewriting Robustness
We acknowledge that entropy-aware detection requires white-box access to the original model for forward-pass reconstruction. This introduces computational and deployment costs, although detection does not require autoregressive generation.
Regarding robustness, extensive LLM rewriting is a common limitation of existing token-level watermarking methods, not a vulnerability specific to GTCW. In our evaluation, rewriting reduces GTCW's AUROC to 51.4%, while competing methods similarly achieve near-chance performance (48.8–55.7%). Such attacks substantially alter token sequences and contexts, disrupting the statistical signals on which these methods rely.
However, GTCW performs relatively better under the other 12 code transformation settings, ranking first or second in TPR@5%FPR and achieving the highest average TPR of 27.2%. These results distinguish its relative effectiveness under localized transformations from its vulnerability to extensive rewriting.