- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Self-Activating Backdoors (SAB), which allow LLM agents to avoid relying on attackers to actively provide triggers after deployment. Instead, the model generates and later re-reads its own watermarked outputs, causing the watermark signal to gradually accumulate over long-term interactions and trigger the implanted malicious behavior once it exceeds a threshold. The authors use a two-stage training procedure to teach the model both to generate watermarks and to switch its behavior based on watermark strength, and evaluate the approach across three models, three attack targets, and a multi-turn coding-agent harness. The results show that the mechanism achieves a low premature activation rate and a high attack activation rate.
Critical strengths
-
I like the core idea of this paper. Using the accumulated watermark strength in the model’s own outputs as the trigger signal for a backdoor is novel and interesting. The attack does not rely on the adversary supplying or anticipating a trigger after deployment. Instead, the trigger signal naturally emerges through repeated interactions between the model and its environment. The threat model is also well aligned with realistic agent-assisted coding scenarios, where the model repeatedly reads, writes, and re-ingests its own generated code.
-
The experiments provide relatively systematic support for this idea. The evaluation covers three models and three target behaviors, including both attack activation and utility, together with a multi-turn validation in an end-to-end coding harness. This allows the core claim to be supported beyond a single-turn controlled setting.
Critical weaknesses
1. The core claim is largely supported by the experiments. However, the most important end-to-end validation relies on intervention from the harness. I would like the authors to evaluate whether the backdoor can still activate naturally without this harness intervention, and if so, how long activation takes. If reliable activation does not occur, it would also be useful to discuss whether the attack can be modified to enable self-activation under more natural settings without such intervention.
2. In realistic deployment settings, models often undergo additional fine-tuning, and the paper also discusses related work on fine-tuning. I would therefore like to see whether the attack remains effective after common downstream fine-tuning procedures and realistic fine-tuning intensities. In addition, the paper contains several unpolished editorial remnants, such as the empty citation in line 123, “Various watermarking families have been proposed ()”, which should be corrected.
If the authors can adequately address these concerns during the rebuttal, I would consider raising my score.
Questions for authors
See weaknesses.
3 comments
Thanks to the authors for their revisions and additional experiments. Please revise the paper to add the detailed experimental settings in the main text or appendix, particularly how requests were sampled and inserted in the new multi-turn experiment and the downstream fine-tuning configuration used for Figure 9.
Overall, the revisions have largely addressed my concerns, and I will raise my rating to weak acceptance.
We thank the reviewer for their positive feedback and for updating their score! We greatly appreciate their time and consideration. We have now also added a detailed description of the experimental settings (realistic setting of the multi-turn experiment in Appendix A.6, and downstream fine-tuning configuration in Appendix A.4). We will continue to update our work to better reflect the new experiments and ongoing discussions with reviewers.
We thank the reviewer for their careful reading of the paper and are happy to hear that they like the ideas behind this work! We address the mentioned weaknesses individually below.
Q1 (W1): Can the backdoor activate naturally without harness intervention, and how long does it take?
Yes, it can. To address this, we conducted an additional experiment in a more realistic scenario, where we include natural user requests that may lead the model to read the relevant context (e.g., requesting type annotations for functions). As shown in the new Figure 6(c–d), we observe successful activation over time, confirming that our attack can be triggered during natural operation (without a readback prompt). We note that conditioning backdoor activation on watermark strength allows us, in principle, to control the z-score threshold at which the backdoor activates. In our main setting, the backdoor triggered reliably (19 of 20 independent sessions) after a median of 33 turns. On average, we found that activation took approximately 2.5 requests that made the model read highly watermarked files into its context. Ultimately, rereading behavior depends heavily on the codebase and user behavior, which is why we chose a controlled setting for a cleaner evaluation. Nevertheless, our new results support our view that current development workflows already involve substantial rereading of LLM-generated content across agents. As we also outline in the paper, we consider this evaluation setup a notable improvement over how backdoors are typically evaluated, which often involves single, artificial input prompts containing a trigger word. We have updated the paper to clarify this point and added the new experiment.
Q2 (W2): Does the attack remain effective after common downstream fine-tuning?
Good question. To evaluate this, we fine-tuned the backdoored models on either a code dataset (CodeAlpaca) or a general-purpose SFT dataset (Dolly) for three epochs. As shown in the table below (and in the new Figure 9), ASR decreases for K2-Horizon-3.7B and Qwen3.5-4B, while LFM2.5-2.6B retains an ASR above 30%. Interestingly, while the ability to generate the watermark remains robust to subsequent fine-tuning (i.e., Stage 2 of Figure 5a), the malicious behavior itself is more sensitive. This is consistent with prior work on backdoors [Cui et al., 2026; Zhao et al., 2026], suggesting that improving robustness to fine-tuning is an important direction for future work.