- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CodeKAST, a training-free, post-generation watermarking framework evaluated on Python code. Appendix B describes four supported transformation families: parameter aliasing, loop-variable renaming, local-variable renaming, and return wrapping. An HMAC-SHA256 rule maps a secret key, public verification context, and site identity to a target state at recoverable AST sites. Transformed code is released only after syntax, round-trip reconstruction, and execution-based checks; otherwise, the original code is retained. Verification reconstructs the site states without the original program and computes a standardized match score for watermark detection and candidate-key attribution, with abstention for absent sites or tied evidence.
Experiments use Qwen2.5-7B-Instruct on MBPP+ (378 tasks) and HumanEval+ (164 tasks), comparing against locally rerun KGW, Unigram, SWEET, and STONE. Reported AUROC is 0.9118 and 0.9904, respectively; Recall@1%FPR is 82.54% and 93.29%, using thresholds selected on the evaluated samples. Strict EvalPlus Pass@1+ is unchanged at 69.05% and 64.02%, with zero reported additional failing tasks. On MBPP+, attribution among the embedding key and 256 wrong keys yields 311 correct decisions, zero wrong decisions, and 67 abstentions (82.28% coverage, 100% conditional accuracy). Formatting preserves these counts; local-variable renaming reduces coverage to 77.51%.
Critical strengths
1.Practical postprocessing that is model-agnostic by design, although evaluated with one generator. CodeKAST does not require decoding-time access or auxiliary training. Recovering evidence from the released code without the original program is a useful deployment property. The reported injection and pipeline timings suggest modest postprocessing overhead for the evaluated functions (Section 4.1).
2.Encouraging detection-utility evidence with transparent qualifications. Tables 1, 2, and 5 report strong detection and no observed additional strict EvalPlus failures on both paired cohorts. The manuscript also reports abstention explicitly, identifies threshold selection as in-sample, and keeps the 974-task development analyses separate. These practices make the empirical results easier to interpret.
Critical weaknesses
1.Attribution evidence supports closed-set ranking, not a rejection-capable provenance decision. The paper explicitly considers a finite candidate-key set (Section 3.1). In Algorithm 2 (p. 6, lines 270-280), the threshold affects only D for the nominated key; attribution returns a unique top-scoring candidate without an absolute acceptance condition. As specified, it can return a key even when all candidate scores are below the detection threshold. This is a specification concern, not a verified implementation error. The reported audit (Section 4.2; Appendix A.3) covers the 378 released MBPP+ programs with the embedding key always among 257 candidates; some released programs may be unchanged fallbacks, whose number is unreported. The 311/0/67 result supports accuracy in that candidate-set setting, but does not establish behavior on an explicit unwatermarked cohort or when the true key is absent. This concerns the scope of the additional attribution contribution, not the primary single-key detection AUROC.
2.Comparative and component-level evidence does not sufficiently isolate the contribution. All four head-to-head baselines are generation-time token methods operating under a different access model. The large reported detection gap is useful but does not determine the relative benefit over post-generation structural watermarking. Related Work discusses RoSeMary but omits SrcMarker, an earlier AST-based structural and identifier watermarking system. Learned structural methods are not equivalent to CodeKAST's training-free keyed design; the question is which differences provide the incremental value and whether a direct comparison is applicable. Tables 4 and 9 remove only the two renaming families with almost no change, leaving the roles of parameter aliasing and return wrapping unexplained. These gaps limit the assessment of significance, rather than establishing that the method is ineffective or lacks novelty.
Questions for authors
1.Does implemented attribution apply any acceptance condition to the top candidate beyond uniqueness within the tie tolerance? Using existing artifacts, could you report unique-attribution and abstention rates on (a) the Base programs and (b) released watermarked programs with the embedding key removed from the candidate set? If attribution is intended only as closed-set ranking with the true key present, please clarify that assumption alongside the attribution claims.
2.How many programs on each main benchmark were transformed versus retained unchanged, and what were the reasons for fallback? What exactly are screening tests P and how do they relate to final EvalPlus tests? How are Base programs that already fail P handled, and how many fallback programs appear among the 67 MBPP+ abstentions? These details are needed to interpret the reported utility and coverage, not to imply leakage.
3.Please provide baseline hyperparameters, detector input/preprocessing, implementation versions, and scored-token counts, particularly for STONE (AUROC 0.5229/0.5110). This would help explain the near-chance results without presuming an implementation error.
4.Please explain the essential differences from prior transformation-based code watermarking such as SrcMarker, and whether a direct comparison is feasible or prevented by language, training, or applicability constraints. A precise account of those constraints is useful even where rerunning a trained method is impractical.
5.What recall is obtained when the threshold is fixed independently of the evaluated Base/watermarked score pairs? With only 378 or 164 negatives, empirical 1% FPR corresponds to at most 3 or 1 false positives, so uncertainty should remain explicit. A focused check using disjoint calibration or a justified random-key null would help; any analytical calibration should state its assumptions about site construction and key-induced dependence. The current in-sample result is openly disclosed and does not invalidate AUROC.