acceptodds
Under review as a conference paper at ICLR 2027

SEAL: Statistical Embedding Alignment for Code Watermarking

Abstract

Large language models can now generate high-quality code at scale, making reliable source tracing increasingly important. Code watermarking, however, is more restrictive than text watermarking: generated code must remain executable, functionally correct, and robust to common edits. In contrast to the existing code watermarking methods that rely on embedding the signal in token choices, syntax-aware rules, or decoding-time biases, tying the watermark to surface form, we propose SEAL, a semantic-space watermarking method for code generation that embeds the signal through candidate selection in code embedding space. Rather than retraining model parameters, decoding logits, or alternating the final program, SEAL selects among model-generated candidates whose semantic projection signs align with a secret key. We further establish two theoretical results proving that SEAL produces a statistically detectable signal that strengthens under aggregation across generated functions, while remaining robust to code transformations that preserve most semantic watermark signs. Experiments on HumanEval and MBPP with CodeLlama-7B and StarCoder2-7B show that SEAL is competitive with SWEET and STONE. Aggregation improves low-FPR detection, and SEAL remains detectable under formatting changes, variable renaming, and code rewrites. Our results show that semantic embedding-space selection offers a practical path toward robust code watermarking, where each generated function serves as the watermark-carrying unit and the signal is encoded in the semantic representation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Semantic Watermarking, Large Language Model, Code Generation Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GhM1OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes SEAL, a watermarking framework that operates in the semantic space of generated code. It treats each generated function as a watermark carrier, maps candidate functions into a code embedding space, and embeds a signal determined by a secret key through rejection sampling.

Critical strengths
  1. The paper designs a watermarking technique in the semantic space for code generation that embeds watermarks without modifying the model’s decoding distribution.
  2. The paper supports SEAL’s robustness against attacks such as renaming and rewriting through both theoretical analysis and experiments.
Critical weaknesses
  1. The figures and tables are poorly presented. For example, Figure 1 looks like an image generated by AI rather than a vector graphic, and Table 1 is incomplete, with its bottom rule missing. The writing also shows clear signs of AI generation, including frequent use of unusual coined terms such as “embedding-space,” “higher-level,” “watermark-carrying,” “margin-aware,” “private-key,” “prompt-conditioned,” and “secret-key-dependent.”
  2. Baseline comparisons are insufficient, and the evaluation metrics are limited. The paper includes only two baselines and uses only pass@1 to assess code quality. The experimental section contains no figures and only three brief tables.
  3. The rejection sampling strategy is very similar to existing text watermarking techniques and does not demonstrate any specific adaptation to code generation.
  4. SEAL requires the original prompt for detection, which is generally impractical because it requires storing all context from interactions between users and the model. The method appears unable to operate correctly when the prompt is unavailable.
Questions for authors
  1. Could the authors provide additional metrics beyond pass@1, such as pass@3, pass@5, and CodeBLEU, as well as experimental results for recent code watermarking methods such as ACW [1] and CodeTracer [2], to further demonstrate the effectiveness of the method?
  2. Could the authors explain the fundamental differences between SEAL and existing text watermarking methods based on rejection sampling, such as PMark [3] and SAEMark [4]? What improvements have been made specifically for code generation?
  3. The paper sets the number of semantic channels to eight but samples only 64 candidates each time. It appears that an average of 256 candidates would be needed to obtain one that satisfies the conditions. How does the method handle cases where no candidate satisfies the conditions? How do different hyperparameters, such as m and N, affect the results?
  4. The paper uses functions as the units for watermark embedding. Does the method account for differences in function length? Specifically, short functions may offer almost no capacity for watermark embedding, whereas longer functions may carry richer watermark signals.

References

[1] Practical and Effective Code Watermarking for Large Language Models.

[2] Adaptive Code Watermarking Through Reinforcement Learning.

[3] PMark: Towards Robust and Distortion-free Semantic-level Watermarking with Channel Constraints.

[4] SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders.

4 comments

Thank you for the careful review! We address each concern below.

W1. Figure, table, and writing presentation. We sincerely thank the reviewer for pointing out these presentation issues. Figure 1 was constructed in PowerPoint and exported as a PDF, but we agree that its current presentation can be improved. We will redraw it as a cleaner vector figure, restore the missing bottom rule in Table 1, and revise the manuscript for clearer and more direct wording.

W2 & Q1. Baselines and evaluation metrics. We added a controlled PMark-style selection comparison and Pass@3, Pass@5, HumanEval+, and CodeBLEU:

SelectorCoverageHumanEval+K=1K=1 AUROCK=16K=16 AUROC
PMark-style100%56.7%0.6950.985
SEAL100%58.5%0.6940.990

To isolate the partition rule, we compared SEAL's median split with PMark's balanced-rank split (dividing each channel at the pool's rank midpoint) on both generators. Expected AUROC averages over candidate choices from the final pools, negatives from the original pools, and 32 random tie-breaks, hence the small differences from the single-run CodeLlama scores.

ModelPartition ruleExpected K=1K=1 AUROCExpected K=16K=16 AUROC
CodeLlama-7BBalanced rank0.7040.986
CodeLlama-7BSEAL0.7040.985
StarCoder2-7BBalanced rank0.6130.847
StarCoder2-7BSEAL0.6610.950

On CodeLlama, the two rules perform comparably. On StarCoder2, where pools are smaller, SEAL's partition rule improves both single-function and K=16 detection under the same candidate and filtering budget.

The following comparison uses five keyed selections from the same N=64N=64 candidate pools:

SelectionHumanEval Pass@1 / @3 / @5 (%)HumanEval+ Pass@1 / @3 / @5 (%)CodeBLEU
Random49.5 / 57.3 / 61.040.9 / 50.2 / 54.90.233
SEAL49.1 / 57.2 / 61.040.4 / 48.9 / 53.70.234

SEAL stays within 1.3 points of random selection on every metric. We will add these results and figures to the revision.

We also compare with ACW [1] on CodeLlama-7B and with CodeTracer's published results. Both methods train auxiliary watermark networks and bias token choices during generation, and CodeTracer [2] also uses reinforcement learning with execution and watermark rewards. SEAL needs no watermark-specific training or logit access and returns the selected code unchanged, at the cost of extra candidate sampling and filtering.

KKSEAL AUROC (95% CI)ACW AUROC (95% CI)
10.694 [0.654, 0.730]0.740 [0.687, 0.787]
40.848 [0.803, 0.893]0.950 [0.874, 0.976]
80.941 [0.901, 0.975]1.000 [0.948, 1.000]
160.990 [0.976, 1.000]1.000 [0.990, 1.000]

We see that ACW is stronger at small KK and the gap closes at K=16K=16 (0.990 vs. 1.000, overlapping CIs). We note that the budgets are not matched: SEAL selects from 64 filtered candidates, while ACW uses unfiltered generation. ACW also reports single function detection only, so for K>1K>1 we pooled its token counts across functions.

CodeTracer's results use OpenCoder and single-program detection, so we list them only as a reference [2]:

CodeTracer settingGeneratorBenchmarkAUROC
RLOpenCoder-1.5BHumanEval0.7771
RLOpenCoder-1.5BMBPP0.7842
TransferabilityOpenCoder-8BHumanEval0.7869

W3 & Q2. Code-specific design. SEAL has three components that distinguish it from prior methods [3, 4]: (i) per-prompt watermark channels defined by empirical medians along secret directions, giving balanced boundaries for each prompt's outputs; (ii) a sequential bit selection algorithm that embeds a secret signature while maximizing agreement across channels; and (iii) a calibrated aggregate detector that combines evidence across mm channels and KK programs, with detection strength scaling as Km\sqrt{Km}. None of them require modifying the LLM or the generated code. Compared with PMark [3], which partitions sentence candidates into balanced halves, SEAL selects complete functions from small, uneven filtered pools and continues past channels whose strict split would be empty (the comparison is in W2 above). SAEMark also supports code but differs in three ways: (i) it scores each unit with one scalar SAE feature statistic, whereas SEAL uses multiple binary channels from CodeBERT projections and per-prompt medians; (ii) it picks the candidate closest to a key scalar target, whereas SEAL applies channel constraints one at a time, keeping the previous pool when a constraint would empty it; and (iii) it tests alignment of scalar sequences, whereas SEAL measures sign agreement across mm channels and KK functions. The distinction therefore is in the channel construction, selection procedure, and detection statistic, rather than the use of code or candidate sampling alone [4].

W4. Original prompt. We thank the reviewer for raising this important question regarding SEAL's verification assumptions and threat model. As we discussed in Appendix A.5, SEAL’s verification uses the original prompt, which is a natural setting for offline code generation services, benchmark auditing, and enterprise deployments where prompts and outputs are logged. In this setting, deterministic reconstruction is practical. Note that the threat model is consistent with prior work, including SWEET [5] and EWD [6], where the original prompt and exact generation settings are used to reconstruct token level entropy and watermark statistics. For example, SWEET [5] prepends the original generation prompt to the code during verification to reproduce its entropy values, while EWD [6] similarly performs its main detection experiments using original prompts to identify programming assignments. We further emphasize that our threat model is attribution with a secret key in logged generation environments: for example, an authorized provider that retains the prompt and relevant generation metadata verifies whether a given code output was produced through its watermarked pipeline with the secret key. We will make sure to emphasize the scope of this threat model in the revised manuscript.

Q3. Effects of the channel count and candidate budget. Thank you very much for raising this question. SEAL does not need a candidate that matches all eight target signs. Partial agreement can still provide a detectable watermark. Channels are applied sequentially; if a strict median partition is empty, SEAL includes candidates at the median and continues.

Candidates NNChannels mmExact mm-bit match availableChannel fallbackEffective channelsK=1K=1 AUROCK=16K=16 AUROCHumanEval+
880.6%100.0%1.380.6040.84042.1%
1681.9%100.0%1.900.6330.90548.8%
3282.5%97.5%2.510.6680.93653.0%
6484.3%96.3%3.070.6940.99058.5%
64281.1%19.5%1.630.7530.99956.7%

Exact matches are measured against the initial pool's medians, while fallback is counted during sequential selection, so the two columns need not sum to 100%. Effective channels count the constraints that narrowed the pool. Detection statistics use nonempty pools; correctness includes all 164 tasks.

We used m=8m=8 as an untuned default. At N=64N=64, m=2m=2 gives better detection, while HumanEval+ scores are close: 56.7% for two channels and 58.5% for eight.

Q4. Function length and watermark capacity. Thank you very much for raising this question. SEAL uses the same eight channels for all functions. We measured the selected functions with the CodeBERT tokenizer and divided the 164 HumanEval outputs into four equal groups, using N=64N=64 candidates throughout.

Length groupFunctionsToken rangeAgreementEffective channelsK=1K=1 AUROC (95% CI)
Shortest quartile4120–620.6433.270.723 [0.647, 0.791]
Second quartile4162–1010.6403.370.731 [0.659, 0.799]
Third quartile41102–1360.6433.150.701 [0.627, 0.770]
Longest quartile41136–3540.5762.510.660 [0.578, 0.745]

Effective channels count the constraints that actually narrow the candidate pool. The shortest group still provides useful detection evidence, and the detection is flat to slightly lower for the longest functions, with overlapping CIs. Only three functions contain fewer than 32 tokens, so we cannot draw a firm conclusion about extremely short snippets. We will add this analysis and the corresponding figure to the revision.

References

  1. Guo et al., Practical and Effective Code Watermarking for Large Language Models.
  2. Guo et al., Adaptive Code Watermarking Through Reinforcement Learning.
  3. Huo et al., PMark: Towards Robust and Distortion-free Semantic-level Watermarking with Channel Constraints.
  4. Yu et al., SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders.
  5. Lee et al., Who Wrote this Code? Watermarking for Code Generation.
  6. Lu et al., An Entropy-based Text Watermarking Detection Method.

I acknowledge the motivation for code watermarking. However, the method lacks innovation specific to code and appears to be a simple adaptation of text watermarking, especially compared with PMark, as all three reviewers noted. The PDF has also not been updated. I therefore remain inclined to keep my original score.

Thanks for your response! We have uploaded a revised PDF to reflect our rebuttal results and edits. Thank you again for your valuable time and overall positive assessment of our work.

Reviewer_KuTFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

For each programming task, SEAL samples 64 candidate functions, filters out invalid ones via syntax checks and testing, and then maps the programs to vectors using CodeBERT. Based on a secret key, the authors generate eight random projection directions; they determine the sign (positive or negative) using the median of the projected values ​​along each direction and employ sequential median filtering to select programs whose signs match the key.

The method is evaluated on HumanEval, MBPP, and a subset of DS-1000, with comparisons to SWEET and STONE , alongside a theoretical analysis of aggregate detection and robustness against code modifications.

Critical strengths

S1. The research problem is significant. Code watermarking requires both ensuring functional correctness and resisting edits that do not alter program behavior; function-level candidate selection represents a sound technical approach.

S2. Engineering integration is straightforward. It requires neither retraining nor modification of the base code model's logits, making it—in principle—applicable to black-box generation services.

Critical weaknesses

W1. The core methodology bears a strong resemblance to existing work, limiting its novelty. PMark already encompasses mechanisms such as candidate sampling, surrogate functions/projection, median-based partitioning, multi-channel filtering, and statistical detection. Consequently, it remains unclear whether the performance gains stem from the novel algorithmic design or simply from applying an existing semantic watermarking framework to the code generation task.

W2. Reported results require further clarification. There are inconsistencies in the reported data: for CodeLlama on HumanEval (K=1), the AUROC value is 0.739 in Table 1, 0.712 in Section 5.3, and 0.723 in Table 5.

W3. Robustness depends heavily on the underlying generative model. As shown in Table 6 of the appendix, variable renaming caused StarCoder2-7B's bit accuracy to drop to 0.497, single-function AUROC to 0.539, and AUROC at K=16 to 0.653. The authors should explain why the same watermarking mechanism exhibits such vastly different representation stability across different code generation models. Semantic equivalence between programs does not, in itself, guarantee the stability of CodeBERT embeddings.

Questions for authors

Please clarify the novelty of the work. I feel that the approach of performing multiple rounds of sampling, calculating the mean, and selecting samples above that mean is already commonplace by 2025 and offers no new insights to the community. Could you add a PMark-to-code baseline? Specifically, keep the CodeBERT model, candidate count, test-based filtering, and computational budget constant, and compare only the selection strategies of PMark and SEAL. If SEAL performs better, which specific design element drives that advantage?

Please explain the differences between the "clean" metrics reported in the main text and the appendix. Clearly specify the number of independent positive and negative samples, the grouping method, and the estimation method for the FPR threshold when K=16;

Could you report Pass@1, the distribution of valid candidate counts, the proportion of empty candidate pools, and watermark detection capabilities under conditions where only syntax checks or public tests are used? Were the positive and negative samples subjected to identical filtering processes?

2 comments

Thank you very much for taking the time to review our paper! Your questions helped us clarify which parts of SEAL are shared with earlier semantic watermarking methods and which parts are specific to executable code. We also ran the requested matched selection and filtering experiments.

W1 & Q1. Novelty and matched comparison with PMark. Thank you for this question! PMark uses sentences as watermark carriers, whereas SEAL was designed to select complete functions from a validity-filtered pool. SEAL’s selection rule can deal with the small and uneven pools after filtering. When a strict median threshold would empty the pool, it includes the median boundary and continues through the remaining channels.

We implemented a function-level PMark-style adaptation [1], fixing CodeBERT, the same N=64N=64 candidates, filtering, keys, channels, and detection procedure. The adaptation stops and returns a candidate from the last nonempty pool when further partitioning is infeasible.

SelectorCoverageHumanEval+K=1K=1 AUROCK=16K=16 AUROC
PMark-style100%56.7%0.6950.985
SEAL100%58.5%0.6940.990

To isolate the partition rule, we compared SEAL's median split with PMark's balanced-rank split (dividing each channel at the pool's rank midpoint) on both generators. Expected AUROC averages over candidate choices from the final pools, negatives from the original pools, and 32 random tie-breaks, which explains the small differences from the single-run CodeLlama scores.

ModelPartition ruleExpected K=1K=1 AUROCExpected K=16K=16 AUROC
CodeLlama-7BBalanced rank0.7040.986
CodeLlama-7BSEAL0.7040.985
StarCoder2-7BBalanced rank0.6130.847
StarCoder2-7BSEAL0.6610.950

On CodeLlama, the two rules perform comparably. On StarCoder2, where pools are smaller, SEAL's partition rule improves both single-function and K=16 detection under the same candidate and filtering budget.

W2 & Q2. Differences among the reported clean results. Thank you for pointing this out. The three values were obtained from different evaluation scopes: Table 1 reports the mean over three generation seeds, Section 5.3 describes the main submitted CodeLlama run, and Table 5 uses the clean control on the subset included in the robustness evaluation. We will label the seed averaged and subset results explicitly.

For the reproduced HumanEval audit at K=16K=16, we form ten disjoint positive groups from 164 outputs.Negatives are candidates drawn from the same filtered pools without watermark selection. The threshold is the 95th percentile of 250 negative calibration groups, chosen without positive scores, and is evaluated on a separate set of 250 negative groups. Positive and negative outputs use the same prompt-specific candidate filtering.

Positive groupsNegative calibration groupsNegative evaluation groupsThresholdEvaluation FPR (95% CI)TPR (95% CI)AUROC (95% CI)
102502500.57030.080 [0.052, 0.120]1.000 [0.722, 1.000]0.990 [0.976, 1.000]

W3. Robustness for CodeLlama and StarCoder2. We thank the reviewer for raising this important point. We agree that semantic equivalence does not guarantee that CodeBERT embeddings, or their watermark signs, remain unchanged. The corrected analysis reconstructs the exact generation-time batch context and gives a zero flip rate for unchanged code for both models. Under identifier renaming, however, the two models behave differently:

GeneratorValid candidatesMean per promptClean agreementϵ+\epsilon_{+}ϵ−\epsilon_{-}Post-rename agreementK=16K=16 AUROC
CodeLlama-7B3,66222.30.6260.3200.3770.5660.894
StarCoder2-7B8635.90.6110.3990.3300.4960.645

Here, ϵ+\epsilon_{+} is the flip probability for an initially agreeing bit, and ϵ−\epsilon_{-} is the flip probability for an initially disagreeing bit. StarCoder2's post-rename agreement is therefore

0.611(1−0.399)+(1−0.611)(0.330)=0.496,0.611(1-0.399)+(1-0.611)(0.330)=0.496,

which matches the measured value. Furthermore, we found through further analysis that StarCoder2 provides about four times fewer valid candidates, leaving less diversity for selection, and its watermark-aligned bits are more likely to cross the embedding boundary after renaming. We will revise the discussion and theorem to use these conditional transitions.

Q3. Correctness, candidate pools, and weaker filtering. Thank you very much for suggesting this experiment! We evaluated CodeLlama-7B on all 164 HumanEval tasks using only syntax checks or visible prompt public doctests. We compare SEAL with random selection from the identical filtered pool below.

Filtering regimePool size, mean (median; IQR)Empty poolsHumanEval Pass@1, random → SEALHumanEval+ Pass@1, random → SEALK=1K=1 AUROCK=16K=16 AUROC
Benchmark tests with syntax fallback (main)22.3 (19.5; 5.0–37.0)0.0%70.1% → 70.1%56.1% → 58.5%0.6940.990
Syntax only39.0 (41.0; 31.8–48.2)0.0%29.9% → 28.7%24.4% → 23.2%0.7521.000
Public doctests28.4 (30.5; 16.8–41.0)7.3%40.9% → 39.0%31.1% → 31.7%0.7320.971

We found that public tests were available for 73 of 164 prompts. Prompts without usable examples retained their valid syntax pool. Correctness is measured over all 164 tasks, including failures from empty pools and detection is evaluated on nonempty pools. Positive and negative samples undergo identical filtering. We found that K=1 AUROC increases with pool size, since larger pools give each channel more candidates to choose from.

These results show that SEAL can achieve strong detection without benchmark test filtering. Under weaker filtering, SEAL’s correctness is within noise of random selection, and detection remains strong. We will report these finding in the next revision.

References

  1. Huo et al., PMark: Towards Robust and Distortion-free Semantic-level Watermarking with Channel Constraints.
Reviewer_N1w8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SEAL, a code watermarking method that selects among multiple LLM-generated candidates without modifying model logits or retraining. It uses CodeBERT embeddings, keyed projection channels, and prompt-specific median thresholds to encode watermark bits. Detection relies on prompt-conditioned verification and aggregation across functions. The paper provides theoretical analyses of aggregation and robustness, and evaluates detection performance against SWEET and STONE on HumanEval, MBPP, and a subset of DS-1000.

Critical strengths

1.SEAL introduces a concrete function-level watermarking pipeline without modifying model parameters or decoding logits. Combining candidate validity filtering with embedding-based selection is a potentially useful approach for code generation.

2.On CodeLlama/HumanEval, SEAL improves AUROC from 0.739 at K=1 to 0.964 at K=8 (Table 1). Under variable renaming, its K=16 AUROC reaches 0.901, compared with 0.677 for SWEET and 0.524 for STONE (Table 5). These results suggest that semantic-space selection merits further investigation.

Critical weaknesses

1.Theorem 1 assumes balanced null agreement and negligible aggregate covariance without establishing these properties empirically. Theorem 2 requires sign flips to be independent of initial watermark agreement, which is not verified. In Table 6, StarCoder2's clean bit accuracy of 0.588 and formatting flip rate of 0.368 imply approximately 0.523 post-attack agreement under the independent-flip model, whereas the reported value is 0.497. This does not invalidate the conditional theorem, but indicates that its assumptions need examination. The nonzero flip rates for the unedited "None" condition (0.086 and 0.104) also require clarification.

2.PMark already uses median-based multichannel selection, while SAEMark employs feature-based selection and includes code-generation experiments. SEAL does not adequately isolate its distinctive contribution relative to these methods. Moreover, Section 5.2 explicitly filters candidates using benchmark tests, so the reported results establish controlled detectability rather than independent correctness preservation. Without matched filtering budgets, held-out functional evaluation, and complete computational costs, the practical benefits remain uncertain.

Questions for authors

1.How are the nonzero "None" flip rates in Table 6 computed? Are the original reference medians consistently reconstructed or reused? Please also reconcile the discrepancies between Section 5.3 and Table 1, and report the positive/negative group counts, threshold-calibration procedure, and uncertainty for low-FPR results.

2.Can the authors report sign-flip probabilities conditioned on initial agreement and disagreement, together with uncertainty estimates? In particular, how do these statistics explain StarCoder2's post-formatting bit accuracy of 0.497 and support the applicability of Theorem 2?

3.What specific methodological advance distinguishes SEAL from PMark and SAEMark? Can the authors provide a focused comparison or ablation isolating this contribution, clarify whether baselines use matched filtering and sampling budgets, and report independent functional correctness and end-to-end computational costs?

2 comments

Thank you for the careful and constructive review. We address each point below.

W1. Empirical validation of the theoretical assumptions. We sincerely appreciate the reviewer raising this issue. We verify null agreement below empirically. Using every validity-filtered candidate before watermark selection, we find that the null agreement is statistically indistinguishable from 0.5 for both models:

ModelPromptsCandidate functionsNull bitsNull agreement (95% CI)
CodeLlama-7B1643,66229,2960.4991 [0.4921, 0.5057]
StarCoder2-7B1628787,0240.4981 [0.4879, 0.5084]

We also obtain normalized aggregate covariance (0.010) (95% CI [−0.024, 0.046]), variance-inflation factor (1.042) [0.901, 1.191], and normalized aggregate variance (1.022) [0.874, 1.150]. These results support our assumptions and we will revise Theorem 1 to state them as empirically checkable conditions.

In terms of Theorem 2's assumptions, the corrected conditional analysis is reported under Q2. It explains the StarCoder2 result and shows that robustness is model and transformation dependent. Finally, we would like to clarify that the “None” flip rate is zero for both models, as demonstrated in Q1.

W2, Q3. Unique contribution. While SEAL operates as a semantic-space watermarking method for code generation, its technical novelty and design is distinguished from prior methods. Specifically, SEAL introduces three key novel components: (i) prompt-conditioned watermark channels defined by empirical medians along secret directions, producing balanced decision boundaries adapted to each prompt’s output distribution; (ii) a sequential multi-bit selection algorithm that embeds a secret signature while maximizing agreement across channels; and (iii) a calibrated aggregate detector that combines evidence across mm channels and KK programs, with detection strength scaling as Km\sqrt{Km}. These components form a new statistical watermarking framework that requires no modification to the LLM or generated code. We compare SEAL with SAEMark [1] and PMark [2] in detail under Q3.

Q1. Verification settings and low-FPR results. Thank you for raising this question. “None” denotes the clean baseline without any transformation. Using the original verification configuration, we reproduce every logged sign, and the “None” flip rate is 0.000 for both models.

ModelFunctionsBits checkedSign mismatches“None” flip rate
CodeLlama-7B1641,31200.000
StarCoder2-7B1471,17600.000

The earlier nonzero values came from re-encoding functions in a different batch context, where small numerical differences affected scores close to the median. We now use the original full-code representation, batch configuration, and float32 pooling. The reference medians are read from the logs, or reconstructed once from logged scores, and remain fixed during all attacks.

Section 5.3 reports one run, while Table 1 reports the three-run mean. We will make this distinction explicit and use the same reporting convention throughout.

We use a fixed random seed to divide the prompts into groups without overlap. We then set the threshold using only negative scores, targeting 5% FPR, then evaluate it on separate negative draws. The table reports the group counts, measured FPR, and detection results for CodeLlama:

KKPositive groupsNegatives: calibration / evaluationFPR (95% CI)TPR (95% CI)AUROC (95% CI)
11644,100 / 4,1000.030 [0.026, 0.036]0.091 [0.056, 0.145]0.694 [0.654, 0.730]
4411,025 / 1,0250.067 [0.054, 0.084]0.415 [0.278, 0.566]0.848 [0.803, 0.893]
820500 / 5000.042 [0.028, 0.063]0.650 [0.433, 0.819]0.941 [0.901, 0.975]
1610250 / 2500.080 [0.052, 0.120]1.000 [0.722, 1.000]0.990 [0.976, 1.000]

As we can see, AUROC increases from 0.694 to 0.990 as more functions are combined. We will be sure to include these details in the revision.

Q2. Conditional sign flips and StarCoder2. Thank you for this suggestion. We measured flips separately for bits that initially match the key (ϵ+\epsilon_{+}) and those that do not (ϵ−\epsilon_{-}) below:

ModelTransformationFlip rateϵ+\epsilon_{+}ϵ+\epsilon_{+}Agreement
CodeLlama-7BClean0.0000.0000.0000.626
CodeLlama-7BFormatting0.1710.157 [0.125, 0.193]0.193 [0.152, 0.238]0.600
CodeLlama-7BRenaming0.3410.320 [0.283, 0.357]0.377 [0.333, 0.420]0.566
CodeLlama-7BRule rewriting0.3500.317 [0.282, 0.350]0.405 [0.358, 0.453]0.579
StarCoder2-7BClean0.0000.000 [0.000, 0.000]0.000 [0.000, 0.000]0.611
StarCoder2-7BFormatting0.3660.394 [0.353, 0.434]0.322 [0.268, 0.376]0.496
StarCoder2-7BRenaming0.3720.399 [0.359, 0.439]0.330 [0.276, 0.385]0.496
StarCoder2-7BRule rewriting0.3670.395 [0.354, 0.435]0.324 [0.271, 0.378]0.496

The 0.588 in Table 6 was measured on re-encoded code. From the logged signs, clean agreement on the same 147 StarCoder2 prompts is 0.611 (prompts with a single valid candidate are excluded, as in Table 6).

For clean agreement pp, pattack=p(1−ϵ+)+(1−p)ϵ−p_{\mathrm{attack}}=p(1-\epsilon_{+})+(1-p)\epsilon_{-}. For StarCoder2 under formatting, 0.611(1−0.394)+0.389(0.322)≈0.4960.611(1-0.394)+0.389(0.322)\approx0.496, which matches the measured value. Flips depend on initial agreement: matching bits flip more often than non-matching ones (ϵ+−ϵ−=0.072\epsilon_{+}-\epsilon_{-}=0.072).

StarCoder2 also gives about 4×4\times fewer valid candidates than CodeLlama, which leaves less room for selection. CodeLlama stays above 0.5 agreement under all three attacks, while StarCoder2 drops to about 0.5. We will add the conditional expression to Theorem 2.

Q3. Technical differences and runtime analysis. Thank you for these questions! PMark [2]’s online algorithm partitions sentence candidates into balanced halves. SEAL operates on complete function units whose available candidates depend on syntax and test outcomes, and handles constraints that cannot be satisfied within that pool. We also implemented a function-level PMark-style selector with the same CodeBERT encoder, N=64N=64 candidates, filtering, keys, and channels:

SelectorHumanEval+K=1K=1 AUROCK=16K=16 AUROC
PMark-style56.7%0.6950.985
SEAL58.5%0.6940.990

The two perform comparably on CodeLlama. We also compared SEAL's median split with a balanced-rank split, which divides each channel at the rank midpoint of the current pool as PMark does. With all other settings fixed, SEAL's rule improves detection on StarCoder2.

Although SAEMark [1] also supports code, its selection and detection procedures are different:

Design choiceSAEMarkSEAL
Watermark representationA normalized scalar feature score, instantiated using SAE featuresMultiple binary signs from CodeBERT projections and prompt-specific medians
SelectionChoose the candidate closest to a key scalar targetApply channel constraints sequentially, with a fallback to avoid an empty candidate set
DetectionCompare observed and target scalar sequences using alignment checks and statistical testingMeasure sign agreement across channels and functions

We will add a more detailed discussion and component ablations to make these distinctions clearer in the revision.

Independent correctness. We compared SEAL with random selection from the same validity-filtered candidate pools. The additional HumanEval+ tests were not used for filtering.

SelectionHumanEvalHumanEval+
Random70.1% (115/164)56.1% (92/164)
SEAL70.1% (115/164)58.5% (96/164)

SEAL matches random selection on HumanEval and passes four more tasks on HumanEval+ (+2.4 points), so watermark selection does not reduce correctness. Random and PMark-style selection use the same N=64N=64 candidates and filtering as SEAL. Without benchmark test filtering, detection remains strong and correctness stays within 2 points of random.

Computational cost. We measured runtime relative to generating and processing one candidate below.

BenchmarkCandidatesConcurrent requestsRelative runtime
HumanEval111.00×
HumanEval64149.24× [35.90, 68.70]
HumanEval64642.30× [1.70, 3.27]
MBPP111.00×
MBPP64180.12× [55.95, 117.16]
MBPP64644.21× [2.90, 6.33]

Parallel requests substantially reduce runtime, although sampling 64 candidates still requires more computation and tokens. We therefore target offline generation and attribution settings where this additional cost is acceptable.

References:

  1. Yu et al., SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders.
  2. Huo et al., PMark: Towards Robust and Distortion-free Semantic-level Watermarking with Channel Constraints.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.