acceptodds
Under review as a conference paper at ICLR 2027

Self-Activating LLM Backdoors

Abstract

LLM Backdoors pose a critical and growing security risk to LLM systems. However, existing attacks, where adversaries choose specific triggers to activate malicious behaviour, assume either an interaction of the adversary with the deployed model or require a correct prediction of triggers present at deployment. Either assumption significantly decreases the practicality of current backdoors. We overcome these limitations by constructing self-activating backdoors, where the model itself generates an accumulative trigger signal. Specifically, we train LLMs to generate and recognize their own watermarked outputs and use watermark strength as a trigger signal. This threat model is especially relevant for coding agents, where models repeatedly modify and re-ingest their own generated code. Agents can thus gain user trust by behaving seemingly normal for extended periods before autonomously entering a malicious state, e.g., by introducing a malicious dependency or deleting files. Self-activating backdoors can also be combined with other trigger signals to increase attack specificity and reduce detection likelihood. Across 3 model families and 3 scenarios, our method achieves near-100% attack success rate while preserving benign behavior prior to activation. We demonstrate this effect in both controlled experiments and end-to-end coding tasks executed using a realistic agent harness. Our results expand the typical LLM backdoor threat model by showing that adversaries neither need externally supplied nor anticipated triggers, calling for a more realistic evaluation of LLM backdoors.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, Backdoor, Poisoning, Agentic

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vSvQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-Activating Backdoors (SAB), which allow LLM agents to avoid relying on attackers to actively provide triggers after deployment. Instead, the model generates and later re-reads its own watermarked outputs, causing the watermark signal to gradually accumulate over long-term interactions and trigger the implanted malicious behavior once it exceeds a threshold. The authors use a two-stage training procedure to teach the model both to generate watermarks and to switch its behavior based on watermark strength, and evaluate the approach across three models, three attack targets, and a multi-turn coding-agent harness. The results show that the mechanism achieves a low premature activation rate and a high attack activation rate.

Critical strengths
  1. I like the core idea of this paper. Using the accumulated watermark strength in the model’s own outputs as the trigger signal for a backdoor is novel and interesting. The attack does not rely on the adversary supplying or anticipating a trigger after deployment. Instead, the trigger signal naturally emerges through repeated interactions between the model and its environment. The threat model is also well aligned with realistic agent-assisted coding scenarios, where the model repeatedly reads, writes, and re-ingests its own generated code.

  2. The experiments provide relatively systematic support for this idea. The evaluation covers three models and three target behaviors, including both attack activation and utility, together with a multi-turn validation in an end-to-end coding harness. This allows the core claim to be supported beyond a single-turn controlled setting.

Critical weaknesses

1. The core claim is largely supported by the experiments. However, the most important end-to-end validation relies on intervention from the harness. I would like the authors to evaluate whether the backdoor can still activate naturally without this harness intervention, and if so, how long activation takes. If reliable activation does not occur, it would also be useful to discuss whether the attack can be modified to enable self-activation under more natural settings without such intervention.

2. In realistic deployment settings, models often undergo additional fine-tuning, and the paper also discusses related work on fine-tuning. I would therefore like to see whether the attack remains effective after common downstream fine-tuning procedures and realistic fine-tuning intensities. In addition, the paper contains several unpolished editorial remnants, such as the empty citation in line 123, “Various watermarking families have been proposed ()”, which should be corrected.

If the authors can adequately address these concerns during the rebuttal, I would consider raising my score.

Questions for authors

See weaknesses.

3 comments

We thank the reviewer for their careful reading of the paper and are happy to hear that they like the ideas behind this work! We address the mentioned weaknesses individually below.

Q1 (W1): Can the backdoor activate naturally without harness intervention, and how long does it take?

Yes, it can. To address this, we conducted an additional experiment in a more realistic scenario, where we include natural user requests that may lead the model to read the relevant context (e.g., requesting type annotations for functions). As shown in the new Figure 6(c–d), we observe successful activation over time, confirming that our attack can be triggered during natural operation (without a readback prompt). We note that conditioning backdoor activation on watermark strength allows us, in principle, to control the z-score threshold at which the backdoor activates. In our main setting, the backdoor triggered reliably (19 of 20 independent sessions) after a median of 33 turns. On average, we found that activation took approximately 2.5 requests that made the model read highly watermarked files into its context. Ultimately, rereading behavior depends heavily on the codebase and user behavior, which is why we chose a controlled setting for a cleaner evaluation. Nevertheless, our new results support our view that current development workflows already involve substantial rereading of LLM-generated content across agents. As we also outline in the paper, we consider this evaluation setup a notable improvement over how backdoors are typically evaluated, which often involves single, artificial input prompts containing a trigger word. We have updated the paper to clarify this point and added the new experiment.

Q2 (W2): Does the attack remain effective after common downstream fine-tuning?

Good question. To evaluate this, we fine-tuned the backdoored models on either a code dataset (CodeAlpaca) or a general-purpose SFT dataset (Dolly) for three epochs. As shown in the table below (and in the new Figure 9), ASR decreases for K2-Horizon-3.7B and Qwen3.5-4B, while LFM2.5-2.6B retains an ASR above 30%. Interestingly, while the ability to generate the watermark remains robust to subsequent fine-tuning (i.e., Stage 2 of Figure 5a), the malicious behavior itself is more sensitive. This is consistent with prior work on backdoors [Cui et al., 2026; Zhao et al., 2026], suggesting that improving robustness to fine-tuning is an important direction for future work.

Thanks to the authors for their revisions and additional experiments. Please revise the paper to add the detailed experimental settings in the main text or appendix, particularly how requests were sampled and inserted in the new multi-turn experiment and the downstream fine-tuning configuration used for Figure 9.

Overall, the revisions have largely addressed my concerns, and I will raise my rating to weak acceptance.

We thank the reviewer for their positive feedback and for updating their score! We greatly appreciate their time and consideration. We have now also added a detailed description of the experimental settings (realistic setting of the multi-turn experiment in Appendix A.6, and downstream fine-tuning configuration in Appendix A.4). We will continue to update our work to better reflect the new experiments and ongoing discussions with reviewers.

Reviewer_GqEPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces self-activating backdoors (SABs), a threat model in which a compromised language model creates an accumulating trigger through its own outputs and later recognizes that trigger when those outputs re-enter its context. Unlike conventional backdoors, the attacker is assumed to control the model weights before release but need not interact with the deployed model or predict a specific deployment-time input.

The proposed construction has two stages. First, the authors distill a red-green statistical watermark into a base model so that it generates watermarked natural-language content in comments, docstrings, and prose without an inference-time logits processor. Second, they fine-tune the same model to condition its behavior on the aggregate watermark z-score of file contents in its context. Contexts below a model-specific threshold receive benign task-solving targets, whereas contexts above the threshold receive one of three adversarial targets: refusal, a pip install tool call, or an rm -rf / tool call. Experiments cover Qwen3.5-4B, LFM2.5-2.6B, and K2-Horizon-3.7B. In held-out single-turn contexts, activation generally rises with watermark strength, with clean-context false-trigger rates between 0% and 2.35%. Seven of nine model-target combinations activate on at least 90% of examples above the training threshold. In a multi-turn coding harness using the K2-Horizon-3.7B refusal variant, all 30 task sequences eventually refuse after reading previously generated files, with a median activation time of 13.5 recorded turns. The paper also shows that the watermark trigger can be conjunctively gated by an external year trigger.

Critical strengths
  1. The paper introduces a meaningful extension of the LLM backdoor threat model. A model that writes a persistent statistical signal into its own outputs and later recognizes the accumulated signal no longer requires attacker access or a correctly anticipated semantic trigger after deployment. This is particularly relevant to coding agents and other systems whose outputs persist and are re-ingested.

  2. The mechanism is evaluated across multiple models and target behaviors with useful controls. The single-turn study covers three model families and three targets, measures false activations on paired clean contexts, and evaluates watermark retention, tool use, and coding performance. The external-trigger experiment further shows that self-generated evidence can be combined conjunctively with another signal, which clarifies the flexibility of the construction.

Critical weaknesses
  1. The agentic experiment uses a favorable read-back intervention. Once the repository-level watermark score reaches 2, the harness explicitly tells the agent to read the files it has written, increasing the watermark evidence in context. The experiment therefore establishes activation after controlled re-ingestion, but does not yet estimate how often a naturally operating coding agent would accumulate enough evidence without a trigger-aware harness. A no-intervention baseline would strengthen the deployment claim, while the existing result remains a useful end-to-end proof of concept.

  2. Stealth and benign utility require broader characterization. Stage 1 causes noticeable drops on BFCL-v4 and some coding evaluations, and the multi-turn demonstration covers only K2-Horizon-3.7B with a refusal target. The paired clean contexts provide a useful controlled false-trigger measurement, but large heterogeneous repositories may contain copied model output or text from other watermarked systems. The paper would be stronger if it framed the current results as establishing feasibility under controlled conditions and left deployment-level stealth as an open empirical question.

Questions for authors
  1. What are the activation rates in the multi-turn harness if the score-dependent read-back prompt is removed? Please report how often the agent naturally re-reads its own files, the resulting context-level watermark scores, and activation rates under ordinary task prompts. This result would materially affect the central claim of autonomous activation.

  2. Were Stage 1 and Stage 2 trained with multiple random seeds? If not, can the authors report repeated training runs or checkpoint variability for activation, false-trigger rates, watermark strength, and utility? One checkpoint per model-target pair does not reveal the stability of the poisoning procedure.

  3. How do false-trigger rates change on realistic mixed repositories containing human code, code from unrelated LLMs, copied documentation, and outputs produced by models using the same or different watermark family? The paired clean contexts in the current evaluation may underestimate accidental or cross-model activation.

  4. Are the destructive tool-call targets merely generated strings, accepted tool calls in a sandbox, or successfully executed actions? Please state the exact success criterion for pip install and rm -rf /, including whether argument variations count and what safeguards prevented effects outside an isolated environment.

  5. What artifacts will be released? Given that the paper provides a fairly direct training recipe for delayed malicious behavior, will the authors withhold backdoored checkpoints, watermark keys, or immediately deployable destructive configurations while releasing sufficient code for controlled reproducibility?

2 comments

We thank the reviewer for the constructive feedback and are happy that they see our setting as a meaningful extension of prior backdoor works. We address their questions individually below.

Q1: Does the backdoor activate under natural operation, without the harness read-back intervention?

Yes. To address this concern, we conducted an additional experiment in a more realistic scenario, where we simulate user requests that can naturally lead the model to reread the context (e.g., requesting type annotations for functions). As shown in the new Figure 6 (c–d), we observe successful activation after some time, confirming that our attack can trigger under natural operation (without a read-back prompt). Ultimately, rereading behavior is highly dependent on the codebase and user behavior, which is why we chose a controlled setting for a cleaner evaluation. Nevertheless, our new results support our confidence that current development practices already involve substantial rereading of LLM-generated content across agentic workflows.

Q2: Were Stage 1 and Stage 2 trained with multiple seeds?

We thank the reviewer for pointing this out. While we are very confident in the results as we went through multiple internal cycles to get to them, we are currently repeating some of our final main experiments with multiple seeds. As these (agentic) runs take some time, we will report the scores here as soon as we get the results.

Q3: How do false triggers change on realistic mixed repositories?

This is a valid question that is difficult to answer, as repositories vary widely in practice. The goal of this work is to show that such self-activating attacks are possible and that watermarks are actually a surprisingly natural fit for them. As our new results in Q1 show, activation can also occur without explicit read-back probes in the harness. While this does not capture all realistic settings, our evaluation (1) allows us to precisely control how much the model generates and reads across tasks, (2) provides strong evidence for the feasibility of the setting, and (3) is already noticeably more practical than most existing single-trigger, single-input backdoor evaluations. That being said, we would be happy to include additional experiments on mixed repositories in our next revision if the reviewer has specific suggestions in mind.

Q4: What is the exact success criterion for the pip install and rm -rf / targets?

For our evaluations, we consider an attack successful if the model generates the targeted harmful tool call. Initially, we also tested looser string-matching criteria, checking whether the model output contained a bash tool call whose argument included either “rm -rf” or the target library name. However, we found no discrepancies between the criteria and therefore decided to consistently use the stricter (and more realistic) setup. We note that we do not execute these malicious tool calls but instead terminate execution.

Q5: What artifacts will be released?

We thank the reviewer for raising this important question. We recognize the dual-use nature of our work and intend to balance reproducibility with responsible artifact release. Our plan is to release our evaluation framework, experimental configurations, and the code necessary to reproduce our results. In contrast, we do not plan to publicly release model checkpoints that enable immediately deployable destructive behavior. We would be happy to hear the reviewer's opinions on which artifacts should or should not be released.

We thank the reviewer for their patience and have now an update to Q2: Multiple seed runs.

Attack success rate.

In Appendix A.3, we compare activation rate in the single-turn setup of the Qwen3.5-4B refusal model across runs that vary (1) only the Stage-2 SFT training seed, (2) the Stage-2 SFT training dataset, and (3) the Stage-1 distillation watermark key. The latter two have a stronger influence on the overall training, as they do not just affect training dynamics but also use independently generated data (2) and a different Stage-1 model as the basis (3). As we show in the newly added Figure 13, our attack remains stable across all these settings: across runs, the activation consistently remains 00% at z<2z<2, exceeding 9898% in every bin above the targeted τ=8\tau=8, also following similar curves in between.

Watermark Strength and Utility.

In our newly added Figure 14, we report the watermark strength and utility of the same training variants as in Figure 13. We find the average watermark score of HumanEval completions to stay within z=2.0z=2.0 and z=2.8z=2.8. Furthermore, while the utility, measured on BFCL-v4 varies across the variants, the average scores, for Stage-1 and Stage-2, are 41.2 and 41.5, respectively, which is only around 4 points below the base.

Reviewer_22gQOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an interesting idea called self-activating LLM backdoors, where the model can gradually generate its own trigger during normal use, without requiring any attacker interaction after deployment. The main idea is to first train the model to generate watermarked outputs, and then fine-tune it to perform malicious behaviors once the accumulated watermark strength in its input context exceeds a certain threshold. In the evaluation, the authors show high attack success rates with low false activation rates across three models and three attack scenarios. They also demonstrate that the backdoor can successfully activate in a multi-turn coding agent setting, where the model repeatedly writes and reads its own generated code.

Critical strengths
  • The paper presents a novel and interesting idea of self-activating backdoors in LLMs.
  • The paper is well-written and easy to follow, with clear motivation and explanation of the proposed method.
  • The paper provides a comprehensive evaluation of the proposed method, domonstrating its effectiveness in various scenarios and tasks.
Critical weaknesses
  • Insufficient evidence to rule out context length as an alternative backdoor trigger.
  • Lack of sufficient evidence that the model actually learns to recognize the watermark.
  • Unclear practical advantage of uncontrolled, eventual backdoor activation.
  • Lack of comparison with simpler context-length-based or interaction-count-based triggers.
  • Potential loss of stealthiness due to inevitable or premature activation during normal use.
  • Lack of empirical evaluation on defense mechanisms
Questions for authors

Q1. Could the model learn the context length rather than watermark strength as the backdoor trigger?

In Section 3.4, the authors construct paired clean and watermarked contexts while keeping the task, number of files, and interaction fixed. In Figure 3, the authors further show that the attack activation rate increases with the z-score, while false activations on clean contexts remain relatively low. I think these results are good initial evidence that the model's learned activation behavior is correlated with watermark strength. However, they do not fully rule out the possibility that the model learns to rely on simpler input characteristics, such as total context length, natural-language token count, or other features correlated with watermark strength. In particular, matching the number of files and interaction structure does not necessarily guarantee that clean and watermarked contexts have identical token lengths. As discussed in Section 3.2, watermark evidence is expected to increase with the number of scored watermarked tokens.

My question is that how can the authors establish that the learned activation behavior is specifically conditioned on watermark strength rather than context length or other correlated properties? A length-controlled evaluation, in which context length (not number of files) is held fixed while watermark strength is different, would provide stronger evidence for this.

Q2. Is there sufficient evidence that the model actually learns to recognize the watermark?

The results in Figure 3 demonstrate a strong correlation between watermark strength and backdoor activation. However, this does not necessarily establish that the model has learned to recognize the intended KGW watermark signal. Since the clean and watermarked contexts are generated by different model checkpoints, they may also show differences in writing style, token frequencies, comment structure, or other distributional characteristics. The model could potentially exploit these features as shortcuts rather than recognizing the actual watermark. Furthermore, the inference-time model does not explicitly calculate the watermark z-score. Its internal decision mechanism is learned implicitly through supervised fine-tuning.

What evidence supports the claim that the model recognizes the intended watermark signal, rather than other statistical characteristics associated with watermarked generations?

It would be helpful to include some targeted controls, like watermark removal experiments: Reduce watermark evidence while preserving context length and task semantics as much as possible, and examine whether activation decreases.

This would help establish whether the intended watermark signal plays a causal role in activation.

Q3. What is the practical advantage of accumulating a trigger that eventually activates without attacker control?

In Section 3.1, the authors assume that the attacker can modify the model weights before deployment but cannot interact with the deployed model afterward. The proposed self-activating mechanism is intended to overcome this limitation by allowing the model to generate its own trigger during normal use. In Figure 6, all evaluated trajectories eventually activate the refusal backdoor, with long turns.

While these results demonstrate reliable delayed activation, they also raise a practical concern that the attacker has limited control over when the malicious behavior occurs.

For instance, if a coding agent begins refusing legitimate requests or performing malicious tool calls after a relatively small number of interactions, the abnormal behavior may be quickly noticed by the user or detected by monitoring systems. Moreover, activation may occur during a low-value task rather than when the agent has access to sensitive resources or a valuable attack opportunity.

Why should reliable eventual activation be considered practically advantageous if the attacker cannot control the timing of the malicious behavior?

Q4. Why is watermark-based accumulation necessary when a context-length-based trigger could achieve a similar effect?

In Section 2, the authors discuss [Wen et al., 2026], which uses conversation length or positional encoding as a trigger for delayed malicious behavior. The authors distinguish their approach by arguing that watermark strength reflects the amount of model-generated content that has been reintroduced into the context, rather than simply the total context length. However, in the proposed setting, watermark evidence naturally tends to increase as the model generates and re-reads more content. Therefore, watermark strength and context length may be strongly correlated.

What practical advantage does watermark-based accumulation provide over a simpler context-length-based or interaction-count-based backdoor? For example, could an attacker achieve similar delayed activation by training the model to behave maliciously after the context exceeds a certain length, without introducing watermark distillation?

Q5. Does the self-activating mechanism undermine the stealthiness traditionally expected of backdoor attacks?

In Introduction and Figure 1, the authors motivate self-activating backdoors by arguing that an agent can behave benignly during initial safety evaluations, gradually accumulate watermark evidence, and eventually activate malicious behavior after extended use. However, conventional backdoors typically aim to remain dormant unless a particular trigger condition is satisfied. The trigger may be controlled by the attacker or associated with a specific deployment condition. In contrast, the proposed mechanism intentionally creates its own trigger during ordinary operation. Consequently, in environments where the model repeatedly re-ingests sufficient watermarked content, activation may become increasingly likely even without any adversarial intervention.

Does this behavior compromise the stealthiness of the attack, since malicious behavior may eventually emerge during ordinary legitimate use? Although the delayed activation may help evade short pre-deployment evaluations, it could also expose the compromised model during continued normal operation.

Q6. How does the proposed backdoor interact with existing/adaptive defense mechanisms?

It is good to see that the authors discuss potential defenses at the end of main paper. However, the paper lacks empirical evaluation of the proposed backdoor against existing or adaptive defenses. For example, it would be helpful to see if some simple paraphrasing or text (variable name/comments) rewriting could remove the watermark and prevent backdoor activation.

3 comments

We thank the reviewer for their detailed, constructive feedback. We address each question individually below.

Q1: Could the model be keying on context length rather than watermark strength?

Given our training setup, context length is unlikely to be the trigger, as the model is explicitly trained on non-watermarked traces of comparable length. Prompted by the reviewer, we tested this directly by reducing the watermark signal in previously triggering sequences while holding context length fixed. The new Figure 8 shows consistently low refusal rates (below 6%) for clean contexts, even at longer lengths, supporting a learned distributional difference rather than a length-based trigger (see Q2).

Q2: Is there evidence the model recognizes the watermark rather than a stylistic shortcut?

Good question. The model is trained on positive (watermarked) and negative examples (unwatermarked LLM generations and base-dataset samples). Prior work on open-source watermarking has shown this to be a standard training setup for learning watermarks [1, 2]. We therefore believe the model learns watermark-induced distributional changes, which may themselves be stylistic. Nevertheless, training relies on a finite sample set, so out-of-distribution (OOD) detection is likely less reliable, although OOD inputs are less relevant to our threat model.

Q3: What is the advantage of an accumulated trigger?

A key contribution is our post-deployment setting: once a user downloads or deploys the model, the adversary need not access it again. In many local deployments by individuals or companies, assuming an adversary can later supply a trigger token is unrealistic. For conventional input-triggered backdoors, the adversary must instead guess values likely to appear in the user's environment. If those values occur during initial testing, the backdoor activates immediately and is exposed. Self-accumulating triggers address both limitations. First, the model generates its own trigger during normal use, requiring no assumptions about the presence of an external trigger. Second, it activates only after sufficient exposure to its own generations, allowing users to build trust in the model and making pre-deployment detection harder. Watermarks offer a convenient LLM-aligned signal: subtle, learnable, reasonably robust, and order-invariant. Other types of self-accumulating signals could also be used. We agree that self-activation alone is less controllable than an input-based backdoor. However, these mechanisms are orthogonal and can be combined (Figure 7; see Q5), achieving comparable controllability while retaining the benefits of delayed activation.

Q4: Can you elaborate why watermark accumulation “is better” than a simpler context-length / interaction-count trigger?

We sought two properties: (1) benign model output before activation and (2) genuine interaction with the environment before activation. In the reviewer's example [Wen et al., 2026], a context-length trigger would likely activate immediately in a practical codebase, as an agent may load many files just to gain an overview. Our approach instead requires multiple successful interactions. An interaction-count trigger could partly achieve this, but unlike watermarks, it is (1) typically less stealthy and more brittle to edits (e.g., stripping or rewriting unusual comments) and (2) order-dependent, whereas the watermark signal is stateless and order-invariant.

Q5: Does self-activation undermine stealth compared to typical backdoor attacks?

Good question. Conventional backdoors rely on specific, often stealthy, input triggers. As discussed in Section 5, these are impractical in many deployments: they require the adversary to interact with the environment or assume the trigger is already present, in which case initial testing may expose it. In contrast, our delayed activation operates under a weaker threat model and is less likely to be detected during initial testing. As mentioned above, it is also complementary to input-based backdoors. Figure 7 combines our watermark backdoor with an existing trigger-word approach (a specific year), achieving the controllability of input-based triggers while retaining the practical benefits of delayed self-activation.

Q6: How does the backdoor interact with existing / adaptive defenses?

That’s an interesting point. Our goal was to introduce a new self-activating setting, not to claim greater robustness against defenses. We therefore did not evaluate defense robustness. Existing defenses often work only in specific settings or target fixed input triggers, underperforming elsewhere [3]. Adapting them to our setting would require additional design choices and evaluation, which we consider beyond the scope of this initial work.

[1] Sander et al. (2024). arxiv.org/abs/2402.14904

[2] Gu et al. (2024). arxiv.org/abs/2312.04469

[3] Bullwinkel et al. (2026). arxiv.org/abs/2602.03085

I thank the authors for their response. The new length-controlled evaluation in Fig. 8 provides useful evidence that the context length alone may not be responsible for the backdoor activation.

However, I still have a few concerns. For Q2, I understand the authors' explanation, but given that the model may learn stylistic differences, how can we distinguish the intended watermark signal from other correlated statistical characteristics?

For Q4, the arguments for watermark-based accumulation are reasonable, but can simpler interaction-count-based backdoors achieve similar delayed activation? I think a direct comparison would help clarify the advantages.

For Q6, I understand that defense robustness is not the main focus of this work. However, could simple paraphrasing or rewriting comments remove the watermark and prevent backdoor activation? If the proposed attack can be easily mitigated by such simple modifications, its practical significance may be limited.

We thank the reviewer for their active follow-up questions and address the reviewer’s remaining concerns below.

Q2 How can we distinguish the intended watermark signal from other correlated statistical characteristics?

It's an interesting question. Our results support the claim that activation tracks watermark strength. Which features the model uses to recognize that signal remains (somewhat) an open question. We note that for the paired training examples, we use the same underlying base model on the same data, with and without watermarking. This setup is designed to teach the model the distributional difference induced by the watermark. That difference can itself manifest in token frequencies or stylistic patterns, and the model could learn to rely on such features.

Our prior length-controlled experiment (Figure 8) shows that activation varies with watermark strength even when context length is held fixed. In the new paraphrasing experiment (Figure 15, more details on the questions below), rewriting docstrings and comments reduced both the measured watermark strength and the activation rate. This is consistent with the model responding to watermark-associated features. Since paraphrasing also changes style, it indicates that our detection strongly correlates with a more robust expression of the watermark rather than purely stylistic features.

Importantly, for self-activation, the only property we require is that the model responds to accumulated evidence of its own watermarked output when that output re-enters its context. We argue that this to some extent is independent of whether the model learns to detect the watermark or also a heavily correlated expression of it.

Q4&Q6 Can simpler interaction-count-based backdoors achieve similar delayed activation? Could simple paraphrasing or rewriting comments remove the watermark and prevent backdoor activation?

Yes, an interaction-count trigger can also produce some forms of delayed activation. However, the key difference lies in the difficulty of detection and removal. We use a recent attack with an interaction-count-based backdoor [1] that counts its interactions by injecting zero-width non-joiners (ZWNJs) as an example. While our attack uses watermarks as a statistical signal dispersed throughout the docstrings and comments, ZWNJs are concentrated triggers (single token) that (1) would be dropped by LLM-based paraphrasing, and (2) are generally easy to remove with code formatters. In contrast, in our newly updated Appendix A.5 we show that watermarks are quite robust to these defenses.

Specifically, based on the reviewers comments, we now show the effect of using 3 models to paraphrase the docstrings and comments (Figure 15). While, paraphrases result in reduced watermark scores (6.6-6.8 to 3.4-3.8 (median)) and in turn also a reduced activation rate, the overall ASR (on contexts with watermark scores z>8z>8 prior to paraphrasing) remains high at 69.9 -- 95.2%. This leverages two key properties of watermarks, i.e., (1) that paraphrasing only weakens but does not remove it and (2) where weaker signals still compound robustly into a signal over longer inputs [2].

In a similar way, in Figure 16, we show what happens when we instead apply 3 popular reformatters (black, yapf, docformatter) to the code files. Here, the activation rate of our attack is also generally unaffected, which indicates that our attack is robust to reformatting-based defenses that could be used to defend against attacks like ZWNJs. It is also important to note that code formatters do not even have to be seen as a defense but rather are already a typical pre-commit step on larger projects.

Lastly, we note that our watermarked signals also persist across separate sessions on the same codebase, which is fundamentally different from e.g. training a model to trigger in every conversation after X amount of turns. That being said, our intention is not to say that watermarks are the perfect signal but rather that we find them to be a strong signal with several useful properties in this setting.

[1] https://arxiv.org/abs/2602.08563

[2] https://arxiv.org/pdf/2306.04634

Comments on the paper

2 comments

We appreciate the reviewers for their extensive feedback. We summarize the major updates to the PDF as follows. Changes are marked in blue.

  • Figure 6: Multi-turn experiment with no harness-forced readback.
  • Figure 8: Ablation distinguishing the effects of context length and watermark strength.
  • Figure 9: Evaluation of the attack effectiveness after downstream fine-tuning.

Based on further discussion and longer runs finishing, we have updated the PDF to also include the following experiments and additional descriptions. Changes are marked in blue.

Appendix A.3 Stability of backdoor insertion

  • Figure 13: Stability of activation rate
  • Figure 14: Stability of watermark strength and utility

Appendix A.5 Robustness to paraphrasing and reformatting

  • Figure 15: Robustness to paraphrasing
  • Figure 16: Robustness to reformatting

Appendix A.4 Downstream fine-tuning

Appendix A.6 Coding harness - Realistic setting

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.