- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes COSTA, a training-free statement-level code watermarking framework that embeds watermark signals into the parity of AST node statistics rather than directly modifying token probabilities. The method consists of three stages—sampling, LLM-based repair, and bridge insertion—to make generated statements satisfy keyed structural constraints. The authors evaluate COSTA across multiple code-generation models, benchmarks, programming languages, and code transformations.
Critical strengths
The main idea is novel, and using complete code statements and their AST structures as watermark carriers is an interesting departure from existing token-level code watermarking methods. In particular, the choice of watermark embedding position is innovative, and the paper provides extensive experiments showing that such structural signals can achieve strong detectability while largely preserving functional correctness.
Critical weaknesses
My main concern is the practical cost of the proposed framework. COSTA requires repeated statement-level resampling and, when sampling fails, additional LLM calls for Direct/Split/Merge repair, which introduces substantial inference overhead. Such a cost may be difficult to justify in realistic large-scale deployment, where watermarking is expected to introduce only limited additional latency and computation. Meanwhile, lightweight token-level watermarking methods such as SynthID are designed to introduce watermark signals with little or no degradation in generation quality and can also be applied to code generation. Therefore, the paper needs to more clearly justify why a substantially more expensive statement-level design is necessary. A second concern is that the robustness evaluation is not sufficiently adaptive to COSTA’s watermark structure. Several evaluated transformations, such as comment deletion and variable renaming, are inherently unlikely to affect AST-parity signals, while the detector explicitly filters some inserted dead-code patterns. Stronger structure-aware attacks that preserve functionality but directly perturb statement boundaries or AST parity would provide a more convincing robustness evaluation.
Questions for authors
Can COSTA work without LLM-based Repair and even without repeated resampling? In particular, how would a much cheaper single-sample + Bridge variant perform? Since Bridge can connect an unmatched statement without rewriting it, this seems to be an important efficiency baseline that is currently missing. How robust is COSTA against a structure-aware attacker who knows that the watermark is encoded in AST parity but does not know the secret key? For example, can semantics-preserving statement splitting, merging, temporary-variable insertion, or systematic AST rewriting substantially reduce detection performance?
5 comments
Response to Reviewer SmaUv (2/2)
W2 & Q2. Structure-aware attacks
We agree that comment deletion and variable renaming provide limited evidence against an attacker who knows that COSTA uses AST parity. We therefore tested dead-code insertion with filtering disabled and evaluated an adaptive attack that directly targets AST parity. In both tables, each cell reports Llama3-8B / Qwen2.5-Coder-14B.
1. Dead-code insertion. We disabled dead-code filtering so that the detector processed the inserted statements. COSTA achieved higher AUC than both SWEET and STONE on all six model–dataset combinations at 10% and 25% insertion and on four of six at 50% insertion.
| Insertion Ratio | Method | MBPP+ F1 | MBPP+ AUC | HumanEval+ F1 | HumanEval+ AUC | APPS F1 | APPS AUC |
|---|---|---|---|---|---|---|---|
| 0% | COSTA | 0.8934 / 0.7113 | 0.9780 / 0.9701 | 0.8248 / 0.8613 | 0.9866 / 0.9728 | 0.9077 / 0.9000 | 0.9663 / 0.9792 |
| SWEET | 0.7575 / 0.5425 | 0.9096 / 0.8546 | 0.7158 / 0.4785 | 0.9125 / 0.8076 | 0.8170 / 0.7375 | 0.8794 / 0.8733 | |
| STONE | 0.5951 / 0.3407 | 0.8274 / 0.7317 | 0.5030 / 0.2185 | 0.8319 / 0.7719 | 0.6782 / 0.4103 | 0.8869 / 0.8045 | |
| 10% | COSTA | 0.5386 / 0.5361 | 0.9283 / 0.9216 | 0.6161 / 0.5002 | 0.9179 / 0.9320 | 0.9028 / 0.8108 | 0.9702 / 0.9598 |
| SWEET | 0.6977 / 0.3799 | 0.8824 / 0.7772 | 0.6354 / 0.3554 | 0.8634 / 0.7529 | 0.7930 / 0.7192 | 0.8824 / 0.8698 | |
| STONE | 0.5052 / 0.2326 | 0.8025 / 0.7179 | 0.3725 / 0.2623 | 0.7861 / 0.7530 | 0.6714 / 0.4505 | 0.8810 / 0.7890 | |
| 25% | COSTA | 0.3761 / 0.2641 | 0.8958 / 0.8882 | 0.3665 / 0.2433 | 0.8840 / 0.9032 | 0.8830 / 0.8216 | 0.9596 / 0.9516 |
| SWEET | 0.6764 / 0.3604 | 0.8882 / 0.7578 | 0.5859 / 0.2977 | 0.8477 / 0.7202 | 0.7902 / 0.6787 | 0.8959 / 0.8782 | |
| STONE | 0.3721 / 0.2055 | 0.7789 / 0.6994 | 0.2670 / 0.2011 | 0.7600 / 0.7451 | 0.6126 / 0.3690 | 0.8667 / 0.7899 | |
| 50% | COSTA | 0.2389 / 0.1497 | 0.8079 / 0.7950 | 0.2033 / 0.2009 | 0.7838 / 0.8008 | 0.6664 / 0.6048 | 0.9080 / 0.8976 |
| SWEET | 0.6222 / 0.2423 | 0.8557 / 0.7275 | 0.4778 / 0.2678 | 0.8262 / 0.7021 | 0.7868 / 0.6035 | 0.8793 / 0.8600 | |
| STONE | 0.3517 / 0.1483 | 0.7431 / 0.6839 | 0.2700 / 0.2487 | 0.7316 / 0.6829 | 0.5723 / 0.3304 | 0.8476 / 0.7892 | |
2. Adaptive parity flipping. The attacker knows that COSTA encodes its watermark in AST parity but lacks the secret key. We randomly selected 10%, 25%, 50%, or 100% of each program’s statements and prompted an LLM to flip a specified parity while preserving functionality. The prompt allowed statement splitting, statement merging, temporary-variable insertion, and systematic AST rewriting. We targeted total, leaf, and inner node counts separately and also tested choosing the count randomly for each selected statement. AUC remained above 0.91 in every setting at 25% targeting. Although targeting every statement (100%) reduced AUC below 0.73 in all settings, attempting to rewrite the entire program while preserving functionality is costly for the attacker.
| Dataset | Target AST node | 10% statements | 25% statements | 50% statements | 100% statements |
|---|---|---|---|---|---|
| MBPP+ | Total node | 0.9781 / 0.9748 | 0.9338 / 0.9466 | 0.8233 / 0.8455 | 0.6368 / 0.7052 |
| Leaf node | 0.9783 / 0.9758 | 0.9450 / 0.9121 | 0.7740 / 0.7318 | 0.6127 / 0.6846 | |
| Inner node | 0.9781 / 0.9748 | 0.9338 / 0.9466 | 0.8233 / 0.8455 | 0.6368 / 0.7052 | |
| Random node | 0.9781 / 0.9748 | 0.9354 / 0.9422 | 0.8014 / 0.8180 | 0.5916 / 0.7020 | |
| HumanEval+ | Total node | 0.9871 / 0.9723 | 0.9382 / 0.9548 | 0.8006 / 0.7953 | 0.6655 / 0.6990 |
| Leaf node | 0.9874 / 0.9704 | 0.9329 / 0.9213 | 0.7509 / 0.7125 | 0.6778 / 0.6591 | |
| Inner node | 0.9871 / 0.9723 | 0.9382 / 0.9548 | 0.8006 / 0.7953 | 0.6655 / 0.6990 | |
| Random node | 0.9870 / 0.9723 | 0.9331 / 0.9532 | 0.7876 / 0.7756 | 0.6479 / 0.6652 | |
| APPS | Total node | 0.9655 / 0.9772 | 0.9545 / 0.9569 | 0.8546 / 0.9086 | 0.6026 / 0.7076 |
| Leaf node | 0.9605 / 0.9778 | 0.9366 / 0.9496 | 0.7164 / 0.7115 | 0.6937 / 0.6194 | |
| Inner node | 0.9655 / 0.9772 | 0.9545 / 0.9569 | 0.8546 / 0.9086 | 0.6026 / 0.7076 | |
| Random node | 0.9619 / 0.9788 | 0.9492 / 0.9566 | 0.8355 / 0.8741 | 0.6625 / 0.7210 | |
Overall, these experiments support COSTA’s robustness to both attacks for two reasons. First, the attacker does not know whether COSTA checks the parity of the leaf, non-leaf, or total node count. Second, the secret key and the LSH of the preceding statement conceal the target parity, limiting the attacker’s ability to disrupt a match reliably. We hope these additional experiments and clarifications address reviewer's concerns about COSTA’s cost and robustness.
With the rebuttal deadline approaching, we would appreciate further feedback from the reviewer so that we can address any remaining questions promptly. If our responses have resolved the concerns raised, we hope these clarifications will inform the final assessment, and we sincerely thank the reviewer for the time and consideration.
Response to Reviewer SmaUv (1/2)
We thank the reviewer for the thoughtful questions about generation cost and robustness. We address each concern below with additional analysis and experiments.
W1: COSTA motivation
[1] How Good is Post-Hoc Watermarking With Language Model Rephrasing?
W1 & Q1: Repair overhead
1. Offline Repair. COSTA can skip both resampling and Repair, but accepting the first statement leaves parity mismatches unresolved and yields weak detection. To accelerate generation, we used an LLM to prepare reusable rewrite templates offline, following the examples in Appendix G.3. COSTA applies the templates locally, and the library can be expanded as new patterns arise. One drawback is that a leaked library would expose the rewrite patterns to attackers. In our experiments, offline templates eliminated online Repair calls and substantially reduced generation time while matching or improving AUROC. The results are shown below, with each metric reported as Llama-3.1-8B / Qwen2.5-Coder-14B.
2. Single-sample + Bridge. We evaluated the suggested baseline by sampling each statement once and using Bridge to connect unmatched statements. It was faster than full COSTA, and a higher Bridge cap allowed it to surpass full COSTA’s clean AUROC in most settings. Because attackers can recognize and remove Bridge statements without changing program behavior, COSTA embeds most of its watermark in ordinary statements. Beyond improving detection at low generation cost, we use Bridge to verify the integrity of longer programs. Each retained Bridge authenticates the preceding code and helps determine whether an attacker has modified it.