- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposed SEW which is a watermarking method for code generated by LLMs, which addresses several limitations of existing code-watermarking techniques, such as the potential for compromising program functionality, the vulnerability of fixed code-transformation patterns to inference by attackers, and the risk of false positives arising from natural variations in coding style. Unlike methods that modify token probabilities during the code-generation process, SEW embeds watermarks after generation is complete by applying semantic-preserving transformations to the code's style. This approach requires neither access to the generative model nor the training of additional neural networks.
Critical strengths
- This paper proposed SEW which can directly watermarking the generated code which can embed watermarks directly into already-generated code, without requiring access to the original LLMs or the training of additional neural networks. By employing code-style transformations combined with checks for type, semantics, and structure the authors minimize the risk that the embedded watermark might interfere with the program's functionality.
- The experiment is comprehansive and this paper is easy to follow.
Critical weaknesses
- This paper apply program structural information and return statements with a secret key to determine the watermark style. However, a big concern is that the information can be modified easily without affecting the code's functionality. For example, an attacker could simply add a function that is never called. This would alter the overall program context, causing the watermark target calculated during detection to deviate from the original. Such an attack does not even require knowledge of the secret key.
- Another concern is that the probability distribution resulting from a majority vote across multiple locations is not necessarily identical to the style probability distribution for a single location. In addition, ablation study did not consider the contributions of program context, structural markers, and key-based style assignment.
Questions for authors
- This paper uses the number of functions, parameters, loops, and return statements to determine the watermark style. Could the author explain how SEW handles these types of simple structural changes? Furthermore, to what extent would the watermark detection performance be impacted if one were to simply add an extraneous function or loop?
- This paper apply majority voting to code locations within the same Grade, and then perform statistical tests using style probabilities estimated from human-written code. However, I am not entirely clear on how these probabilities are determined. After majority voting is applied across multiple locations, the resulting probability distribution is not necessarily the same as that of an individual location. In addition, different grades may not necessarily independent of one another.
- Please provide an more ablation that isolates the specific contributions of the application context and structural identifiers, which could explain why it is necessary to use the Secret Key, siteID, and application context in combination?
- This paper counts any watermark decision that the attacker has not previously observed as a prediction error. I find this evaluation method maybe unfair. If such combinations are automatically classified as errors, does a lower prediction accuracy truly indicate that the watermarks are harder to predict, or does it simply reflect the fact that there is little overlap in these structural combinations across different programs?
10 comments
Response to Reviewer bo59[2/6]
W2. Another concern is that the probability distribution resulting from a majority vote across multiple locations is not necessarily identical to the style probability distribution for a single location. In addition, ablation study did not consider the contributions of program context, structural markers, and key-based style assignment.
Q2. This paper apply majority voting to code locations within the same Grade, and then perform statistical tests using style probabilities estimated from human-written code. However, I am not entirely clear on how these probabilities are determined. After majority voting is applied across multiple locations, the resulting probability distribution is not necessarily the same as that of an individual location. In addition, different grades may not necessarily independent of one another.
Q3. Please provide an more ablation that isolates the specific contributions of the application context and structural identifiers, which could explain why it is necessary to use the Secret Key, siteID, and application context in combination?
Response to W2 / Q2 / Q3
Thank you for raising these concerns. Our additional analyses show that replacing single-location style probabilities with grade-level majority probabilities leaves the reported FPR and TPR@FPR5% unchanged, while component ablations show complementary contributions from the secret key, program context, and structural markers in siteID. We also evaluate empirical false positives under potential grade dependence. We report these analyses in Appendices D and G of the revised manuscript.
[Null Probabilities and Majority Voting]
We estimated from the frequency of variant 1 at eligible sites for rule in human-written LeetCode solutions (Section 3.1). We clipped the estimates to and assigned to rules with fewer than 30 observations or unstable estimates. The detector uses when the target and otherwise. This approximates the majority-vote probability, which need not equal the probability at an individual site. Our analysis shows that 66.5%–67.0% of grades contain one site, while 93.5%–94.5% of multi-site grades are unanimous. The grade-count-weighted mean absolute difference between observed majority frequencies and is 0.030–0.042 for multi-site bins with at least 30 grades. We additionally evaluate majority probabilities estimated directly from human-code grade outcomes, separately by rule and grade-size bin (2, 3, 4, or at least 5 sites), using Laplace smoothing , where of grades have majority variant 1. Ties count as variant 1, estimates are clipped to , and single-site or sparse bins retain . Across all nine settings, human-code FPR and TPR@FPR5% remain unchanged, while AUROC changes by at most 0.02 points on the 0–100 scale. Table W2-1 reports grade statistics, while Table W2-2 compares current and alternative calibration. In Table W2-2, each arrow indicates current alternative calibration, and AUROC diff. denotes the absolute difference on the 0–100 scale.
[Table W2-1. Grade Statistics of Human-Written Code]
| Language | Single-site (%) | Unanimous (%) | Prob. diff. |
|---|---|---|---|
| Python | 66.7 | 94.3 | 0.042 |
| Java | 67.0 | 94.5 | 0.032 |
| C++ | 66.5 | 93.5 | 0.030 |
[Table W2-2. Current and Alternative Calibration]
| Language | Model | FPR (%) | T@5 | AUROC diff. |
|---|---|---|---|---|
| Python | Qwen3.5 | 0.00 | ||
| gemma-4 | 0.00 | |||
| gpt-oss | 0.00 | |||
| Java | Qwen3.5 | |||
| gemma-4 | ||||
| gpt-oss | ||||
| C++ | Qwen3.5 | |||
| gemma-4 | ||||
| gpt-oss |
Response to Reviewer bo59[3/6]
[Grade dependence]
Grades within the same program can share style preferences, so independence is a modeling approximation. We additionally evaluate empirical FPR on intact, unwatermarked CodeContests human reference programs using 200 random keys, preserving within-program dependencies. At , the mean FPR is 0.91% for Python (198 programs), 0.89% for Java (213), and 0.35% for C++ (237), all below the nominal 1% level. The fixed-key Python FPRs in the calibration table correspond to only 2–4 false positives per setting, making them sensitive to individual cases. These results support empirical false-positive control on average across the evaluated programs and keys, without establishing grade independence or guaranteeing control for every fixed key.
[Component ablations]
We additionally evaluate variants that separately remove the secret key, program context , or structural markers from siteID across all nine settings. All variants retain the same transformation rules and applicability checks, and pass@1 matches Vanilla. Removing the secret key increases mean human-code FPR to 7.43% in Python and 15.45% in Java. Removing increases mean rule-inference accuracy at from 24.03%–45.17% to 94.47%–96.56%, indicating greater transferability of observed decisions across programs. Removing structural markers reduces mean TPR@FPR5% under 50% style flipping from 81.52%–94.91% to 75.02%–79.87%. These results support combining all three components: the secret key prevents direct target computation, program context limits cross-program decision transfer, and structural markers improve robustness under partial style flipping. The table reports unweighted means across three backbone models within each language. Human-code FPR uses , Atk. averages T@5 over four non-LLM editing attacks, Flip 50/75 denotes 50%/75% style flipping, and inference accuracy is measured at with unseen decisions counted as errors.
[Table W2-3. Component Ablation Results]
| Language | Configuration | T@5 | FPR (%) | Atk. | Flip 50 | Flip 75 | Inference acc. |
|---|---|---|---|---|---|---|---|
| Python | w/o key | 99.49 | 7.43 | 98.74 | 99.29 | 98.95 | 44.54 |
| w/o | 100.00 | 0.00 | 99.33 | 92.91 | 65.81 | 95.44 | |
| w/o structural markers | 99.76 | 0.60 | 94.33 | 75.84 | 50.49 | 52.39 | |
| SEW | 99.49 | 1.95 | 99.00 | 93.63 | 64.81 | 45.17 | |
| Java | w/o key | 98.70 | 15.45 | 97.47 | 96.97 | 92.82 | 23.47 |
| w/o | 98.70 | 0.52 | 98.66 | 96.18 | 69.33 | 96.56 | |
| w/o structural markers | 93.85 | 0.39 | 93.03 | 79.87 | 56.36 | 31.26 | |
| SEW | 98.21 | 0.00 | 97.95 | 94.91 | 70.52 | 24.57 | |
| C++ | w/o key | 99.56 | 0.17 | 99.34 | 99.39 | 98.65 | 18.80 |
| w/o | 99.22 | 0.31 | 97.69 | 95.80 | 77.47 | 94.47 | |
| w/o structural markers | 99.56 | 0.00 | 95.74 | 75.02 | 42.66 | 27.75 | |
| SEW | 98.42 | 0.31 | 96.38 | 81.52 | 52.32 | 24.03 |
Response to Reviewer bo59[1/6]
Response to W1 / Q1
Thank you for raising this concern. Our additional experiments show that SEW is sensitive to structural insertions, but unused-function insertion also affects other evaluated post-hoc watermarking methods. These structural insertions can be mitigated through simple static-analysis preprocessing that excludes safely identified unused or unreachable constructs when computing χ(C) during embedding and detection. For example, tools such as Ruff and Vulture can help identify such constructs in Python. We report the structural insertion results in Appendix H of the revised manuscript.
[Structural Insertion Attacks]
SEW derives watermark targets using the structural context χ(C). Inserting a function or loop can change these targets during detection even when existing watermark-bearing styles remain unchanged. We evaluate four behavior-preserving insertions: an unused function containing
return 0, a zero-iteration loop, an unused parameter with updated call sites, and a return statement in an unreachable branch. Across Python, Java, and C++, unused-function insertion reduces SEW's mean TPR@FPR5% from 98.21%–99.49% to 16.06%–24.99%, while loop insertion reduces it to 15.64%–26.33%. SrcMarker and RoSeMary also fall to 3.11%–22.98% under unused-function insertion, while ACW reaches 0.00% in Python. Under loop, parameter, and return insertion, SEW shows larger reductions than SrcMarker and RoSeMary. The table below reports TPR@FPR5% as the unweighted mean across three backbone models within each language.