- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes SEAL, a watermarking framework that operates in the semantic space of generated code. It treats each generated function as a watermark carrier, maps candidate functions into a code embedding space, and embeds a signal determined by a secret key through rejection sampling.
Critical strengths
- The paper designs a watermarking technique in the semantic space for code generation that embeds watermarks without modifying the model’s decoding distribution.
- The paper supports SEAL’s robustness against attacks such as renaming and rewriting through both theoretical analysis and experiments.
Critical weaknesses
- The figures and tables are poorly presented. For example, Figure 1 looks like an image generated by AI rather than a vector graphic, and Table 1 is incomplete, with its bottom rule missing. The writing also shows clear signs of AI generation, including frequent use of unusual coined terms such as “embedding-space,” “higher-level,” “watermark-carrying,” “margin-aware,” “private-key,” “prompt-conditioned,” and “secret-key-dependent.”
- Baseline comparisons are insufficient, and the evaluation metrics are limited. The paper includes only two baselines and uses only pass@1 to assess code quality. The experimental section contains no figures and only three brief tables.
- The rejection sampling strategy is very similar to existing text watermarking techniques and does not demonstrate any specific adaptation to code generation.
- SEAL requires the original prompt for detection, which is generally impractical because it requires storing all context from interactions between users and the model. The method appears unable to operate correctly when the prompt is unavailable.
Questions for authors
- Could the authors provide additional metrics beyond pass@1, such as pass@3, pass@5, and CodeBLEU, as well as experimental results for recent code watermarking methods such as ACW [1] and CodeTracer [2], to further demonstrate the effectiveness of the method?
- Could the authors explain the fundamental differences between SEAL and existing text watermarking methods based on rejection sampling, such as PMark [3] and SAEMark [4]? What improvements have been made specifically for code generation?
- The paper sets the number of semantic channels to eight but samples only 64 candidates each time. It appears that an average of 256 candidates would be needed to obtain one that satisfies the conditions. How does the method handle cases where no candidate satisfies the conditions? How do different hyperparameters, such as m and N, affect the results?
- The paper uses functions as the units for watermark embedding. Does the method account for differences in function length? Specifically, short functions may offer almost no capacity for watermark embedding, whereas longer functions may carry richer watermark signals.
References
[1] Practical and Effective Code Watermarking for Large Language Models.
[2] Adaptive Code Watermarking Through Reinforcement Learning.
[3] PMark: Towards Robust and Distortion-free Semantic-level Watermarking with Channel Constraints.
[4] SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders.
4 comments
W4. Original prompt. We thank the reviewer for raising this important question regarding SEAL's verification assumptions and threat model. As we discussed in Appendix A.5, SEAL’s verification uses the original prompt, which is a natural setting for offline code generation services, benchmark auditing, and enterprise deployments where prompts and outputs are logged. In this setting, deterministic reconstruction is practical. Note that the threat model is consistent with prior work, including SWEET [5] and EWD [6], where the original prompt and exact generation settings are used to reconstruct token level entropy and watermark statistics. For example, SWEET [5] prepends the original generation prompt to the code during verification to reproduce its entropy values, while EWD [6] similarly performs its main detection experiments using original prompts to identify programming assignments. We further emphasize that our threat model is attribution with a secret key in logged generation environments: for example, an authorized provider that retains the prompt and relevant generation metadata verifies whether a given code output was produced through its watermarked pipeline with the secret key. We will make sure to emphasize the scope of this threat model in the revised manuscript.
Q3. Effects of the channel count and candidate budget. Thank you very much for raising this question. SEAL does not need a candidate that matches all eight target signs. Partial agreement can still provide a detectable watermark. Channels are applied sequentially; if a strict median partition is empty, SEAL includes candidates at the median and continues.
| Candidates | Channels | Exact -bit match available | Channel fallback | Effective channels | AUROC | AUROC | HumanEval+ |
|---|---|---|---|---|---|---|---|
| 8 | 8 | 0.6% | 100.0% | 1.38 | 0.604 | 0.840 | 42.1% |
| 16 | 8 | 1.9% | 100.0% | 1.90 | 0.633 | 0.905 | 48.8% |
| 32 | 8 | 2.5% | 97.5% | 2.51 | 0.668 | 0.936 | 53.0% |
| 64 | 8 | 4.3% | 96.3% | 3.07 | 0.694 | 0.990 | 58.5% |
| 64 | 2 | 81.1% | 19.5% | 1.63 | 0.753 | 0.999 | 56.7% |
Exact matches are measured against the initial pool's medians, while fallback is counted during sequential selection, so the two columns need not sum to 100%. Effective channels count the constraints that narrowed the pool. Detection statistics use nonempty pools; correctness includes all 164 tasks.
We used as an untuned default. At , gives better detection, while HumanEval+ scores are close: 56.7% for two channels and 58.5% for eight.
Q4. Function length and watermark capacity. Thank you very much for raising this question. SEAL uses the same eight channels for all functions. We measured the selected functions with the CodeBERT tokenizer and divided the 164 HumanEval outputs into four equal groups, using candidates throughout.
| Length group | Functions | Token range | Agreement | Effective channels | AUROC (95% CI) |
|---|---|---|---|---|---|
| Shortest quartile | 41 | 20–62 | 0.643 | 3.27 | 0.723 [0.647, 0.791] |
| Second quartile | 41 | 62–101 | 0.640 | 3.37 | 0.731 [0.659, 0.799] |
| Third quartile | 41 | 102–136 | 0.643 | 3.15 | 0.701 [0.627, 0.770] |
| Longest quartile | 41 | 136–354 | 0.576 | 2.51 | 0.660 [0.578, 0.745] |
Effective channels count the constraints that actually narrow the candidate pool. The shortest group still provides useful detection evidence, and the detection is flat to slightly lower for the longest functions, with overlapping CIs. Only three functions contain fewer than 32 tokens, so we cannot draw a firm conclusion about extremely short snippets. We will add this analysis and the corresponding figure to the revision.
References
- Guo et al., Practical and Effective Code Watermarking for Large Language Models.
- Guo et al., Adaptive Code Watermarking Through Reinforcement Learning.
- Huo et al., PMark: Towards Robust and Distortion-free Semantic-level Watermarking with Channel Constraints.
- Yu et al., SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders.
- Lee et al., Who Wrote this Code? Watermarking for Code Generation.
- Lu et al., An Entropy-based Text Watermarking Detection Method.
I acknowledge the motivation for code watermarking. However, the method lacks innovation specific to code and appears to be a simple adaptation of text watermarking, especially compared with PMark, as all three reviewers noted. The PDF has also not been updated. I therefore remain inclined to keep my original score.
Thanks for your response! We have uploaded a revised PDF to reflect our rebuttal results and edits. Thank you again for your valuable time and overall positive assessment of our work.
Thank you for the careful review! We address each concern below.
W1. Figure, table, and writing presentation. We sincerely thank the reviewer for pointing out these presentation issues. Figure 1 was constructed in PowerPoint and exported as a PDF, but we agree that its current presentation can be improved. We will redraw it as a cleaner vector figure, restore the missing bottom rule in Table 1, and revise the manuscript for clearer and more direct wording.
W2 & Q1. Baselines and evaluation metrics. We added a controlled PMark-style selection comparison and Pass@3, Pass@5, HumanEval+, and CodeBLEU:
To isolate the partition rule, we compared SEAL's median split with PMark's balanced-rank split (dividing each channel at the pool's rank midpoint) on both generators. Expected AUROC averages over candidate choices from the final pools, negatives from the original pools, and 32 random tie-breaks, hence the small differences from the single-run CodeLlama scores.
On CodeLlama, the two rules perform comparably. On StarCoder2, where pools are smaller, SEAL's partition rule improves both single-function and K=16 detection under the same candidate and filtering budget.
The following comparison uses five keyed selections from the same N=64 candidate pools:
SEAL stays within 1.3 points of random selection on every metric. We will add these results and figures to the revision.
We also compare with ACW [1] on CodeLlama-7B and with CodeTracer's published results. Both methods train auxiliary watermark networks and bias token choices during generation, and CodeTracer [2] also uses reinforcement learning with execution and watermark rewards. SEAL needs no watermark-specific training or logit access and returns the selected code unchanged, at the cost of extra candidate sampling and filtering.
We see that ACW is stronger at small K and the gap closes at K=16 (0.990 vs. 1.000, overlapping CIs). We note that the budgets are not matched: SEAL selects from 64 filtered candidates, while ACW uses unfiltered generation. ACW also reports single function detection only, so for K>1 we pooled its token counts across functions.
CodeTracer's results use OpenCoder and single-program detection, so we list them only as a reference [2]:
W3 & Q2. Code-specific design. SEAL has three components that distinguish it from prior methods [3, 4]: (i) per-prompt watermark channels defined by empirical medians along secret directions, giving balanced boundaries for each prompt's outputs; (ii) a sequential bit selection algorithm that embeds a secret signature while maximizing agreement across channels; and (iii) a calibrated aggregate detector that combines evidence across m channels and K programs, with detection strength scaling as Km. None of them require modifying the LLM or the generated code. Compared with PMark [3], which partitions sentence candidates into balanced halves, SEAL selects complete functions from small, uneven filtered pools and continues past channels whose strict split would be empty (the comparison is in W2 above). SAEMark also supports code but differs in three ways: (i) it scores each unit with one scalar SAE feature statistic, whereas SEAL uses multiple binary channels from CodeBERT projections and per-prompt medians; (ii) it picks the candidate closest to a key scalar target, whereas SEAL applies channel constraints one at a time, keeping the previous pool when a constraint would empty it; and (iii) it tests alignment of scalar sequences, whereas SEAL measures sign agreement across m channels and K functions. The distinction therefore is in the channel construction, selection procedure, and detection statistic, rather than the use of code or candidate sampling alone [4].