acceptodds
Under review as a conference paper at ICLR 2027

Agentic Backdoors via Pretraining Poisoning

Abstract

Large language models are pretrained on vast, uncurated corpora scraped from the public web. This creates a security risk from attackers who plant poisoned content in the pretraining data. We show for the first time that such poisoning can backdoor an agent, causing it to execute attacker-chosen commands on the user's machine. To measure the threat end to end, we pretrain three Qwen3 models from scratch at sizes 0.6B, 1.7B, and 4B, and then post-train each into a CLI agent using a clean pipeline. Our main results show that poisoning only 0.1% of the pretraining corpus successfully implants the backdoor: The deployed agent executes a fixed malicious command when the trigger is present while behaving normally on clean inputs, and the attack success rate (ASR) strengthens with model scale, reaching roughly 92% success at 4B. We further introduce new poisoning designs that make both the poison document and the trigger harder to identify or filter out. We maintain 24% ASR even after reducing the poisoning rate to 0.0005%, only 2500 documents. We argue for treating the provenance of pretraining data as part of an agent's security model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
poisoning, pretraining, large language models, agent, security

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9FLxOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether backdoors introduced during pretraining can survive the subsequent training of a command-line agent. The authors train Qwen3 architectures from scratch at three sizes and apply clean SFT, DPO, and GRPO. They compare demonstration-based and declaration-based poisoning, using either a rare string or ordinary-looking filesystem paths as triggers. The experiments show that poisoned models continue to emit a fixed malicious shell command after post-training, while maintaining similar performance on the reported clean benchmarks. Additional experiments examine generalization to unseen paths and persistence at lower poisoning rates.

Critical strengths
  1. The paper addresses a meaningful security question: whether poisoning at the pretraining stage remains effective after a model is trained for tool use. Following the backdoor through SFT, DPO, and GRPO makes the study useful for understanding what a clean downstream training pipeline does and does not remove.

  2. The main experiments cover three model sizes, two trigger types, and two poisoning designs under a consistent training recipe. The default-rate experiments report three-seed averages and ranges at each stage. This provides substantially more evidence than a result from a single poisoned checkpoint.

  3. The clean-model and clean-input controls are helpful. Unpoisoned models do not emit the payload on the triggered evaluation prompts, and poisoned models do not emit it on the clean NL2SH prompts after post-training. The reported capability comparisons also show that the backdoor can coexist with similar performance on the measured benchmarks.

  4. The ordinary-string experiments go beyond memorizing complete training paths. The authors reserve a disjoint set of paths for evaluation and compare results with seen paths, finding similar attack success. This supports generalization within the tested path family and makes the result more interesting than a backdoor tied to one exact path.

  5. The main finding remains substantial under single-sample evaluation. For the 4B declaration-based models after GRPO, Table 9 reports Pass@1 of 39.5% for the rare-string trigger and 50.3% for the ordinary-string trigger. Reporting these alongside Pass@32 helps establish that the main result is not solely a consequence of repeated sampling.

Critical weaknesses
  1. The evaluation measures payload emission more directly than agent execution. Appendix A.5 scores whether generated text contains the target command and uses an LLM judge to assess executability. This is a meaningful proxy under the stated assumption that model outputs are executed as shell commands, but it does not directly verify execution in an agent environment. Since actual agent actions are central to the paper's positioning, a sandboxed execution check using a harmless payload would strengthen the evidence. Execution-based clean task performance would also complement the current command-matching utility metrics.

  2. The evaluation leaves some uncertainty about trigger specificity and transfer across contexts. The attack prompts are setup-script requests generated using the same hierarchical approach as the poison, with the trigger placed directly in the user request. The held-out paths and phrasings are useful, but the clean-input control uses a different set of NL2SH tasks. Evaluating the same requests with the trigger removed or replaced would better isolate its contribution. Results where the trigger appears in a tool observation or an unrelated task would also better support the broader deployment claims.

  3. The explanation for the declaration-based design's advantage is not isolated experimentally. The designs differ in representation and use different numbers of styles: 50 for declarations and 20 for demonstrations. The attribution to diversity in Section 4 therefore remains a hypothesis. The claim of consistent superiority also needs qualification: for the 0.6B model with the ordinary-string trigger after GRPO, declaration achieves 12.3% Pass@32, compared with 21.9% for demonstration in Tables 7–8.

  4. The lowest-rate result is useful evidence that the attack can succeed, but its reliability remains uncertain. At 0.0005% poisoning, the reported 24.0% is single-seed Pass@32; the corresponding Pass@1 is 2.58% in Table 13. The dose response is also non-monotonic, which the authors acknowledge. These results support an existence claim, but additional seeds would be needed to establish a reliable low-budget success rate. The abstract should identify the sampling metric and single-seed setting explicitly.

  5. The claims about resistance to filtering are not directly evaluated. Removing chat templates and using ordinary-looking paths plausibly removes some obvious cues, but the paper does not test whether these documents survive data-curation filters or are harder to detect than demonstrations. The absence of payload emission on clean inputs addresses a different question. Statements that the declaration-based poison is harder to filter should either be supported experimentally or presented as a hypothesis.

Questions for authors
  1. Were the generated payloads executed in a sandbox, or is the reported ASR based entirely on string inclusion and the LLM executability check? Can you report execution-based results with a harmless payload, together with clean task success on the held-out InterCode tasks?

  2. What happens when the trigger is removed or replaced in the existing attack evaluation prompts while the rest of each request is unchanged? This would help distinguish trigger-dependent behavior from a broader association between setup-script requests and the payload.

  3. Is there an ablation that separates document diversity from the declaration-versus-demonstration format? Please also reconcile the claim of consistent superiority with the 0.6B ordinary-string results in Tables 7–8.

1 comment

Dear Reviewer 9FLx,

As the author–reviewer discussion period draws to a close, we would appreciate your feedback on our responses and whether they address your concerns. We welcome any further questions or points for clarification. Thank you for your time and consideration.

Reviewer_EMAbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether poisoned pretraining documents can inject a fixed malicious shell-command behavior that survives clean agent post-training (A backdoor). It compares demonstration-based and prose declaration-based poisoning, rare-string and ordinary-path triggers, and Qwen3 architectures trained from scratch different sizes. They then do SFT, safety DPO, and bash-task GRPO to have a terminal agent. At 0.1% poisoning, the 4B declaration model reaches 92.2% Pass@32 and 50.3% per-sample ASR for held-out ordinary paths after GRPO.

Critical strengths
  • Training from scratch and tracking the backdoor through SFT, DPO, and GRPO directly tests persistence across an agent-training pipeline. Clean-model and non-trigger controls make the causal poisoning result persuasive.

  • Declaration poisoning and held-out ordinary-path triggers substantially broaden the demonstrated threat beyond conspicuous transcript templates and exact memorized paths.

Critical weaknesses
  • The reported ASR measures payload inclusion in generated text, supplemented by an LLM executability check; a separate rate of actual tool executions is not reported. The evaluation uses generated setup-script requests, so the results do not establish activation when the trigger appears incidentally during unrelated agent work. This limits the demonstrated scope of deployed compromise.

  • Reliability is uncertain: the 24% result is single-seed Pass@32, versus 2.58% Pass@1, which to me, seems negligible, and the dose response is non-monotonic (Tables 12–13). The authors acknowledge the single-seed limitation. This supports the possibility more strongly than a reliable low-budget attack.

  • Even though the paper is sound and the experiments are extensive, the results are expected and not surprising. Previous works like Carlini's 2024 work ("Poisoning web-scale training datasets is practical") was the first to show such backdoors remain from the pretraining stage. I understand that it was not focused on agents, but they are almost the same.

Questions for authors
  1. Does the ASR evaluation execute emitted commands through the deployment interface, or only check textual inclusion and judge executability? If execution was tested separately, what was the measured execution rate?

  2. How does Pass@1 ASR change when the trigger appears incidentally in tool output or task data during requests unrelated to installing or running scripts?

  3. Can the lowest-dose result be replicated across seeds, reporting both Pass@1 and Pass@32?

1 comment

Dear Reviewer EMAb,

As the author–reviewer discussion period draws to a close, we would appreciate your feedback on our responses and whether they address your concerns. We welcome any further questions or points for clarification. Thank you for your time and consideration.

Reviewer_AtTeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies pretraining-data poisoning that survives SFT, DPO, and RLVR in a CLI agent. It compares declaration and demonstration poisons, rare and ordinary triggers, and Qwen3 models up to 4B parameters. The persistence question is important, but the evaluation shows payload emission rather than actual command execution, and the low-poisoning conclusions rely on single-seed results. I do not recommend acceptance in the current form.

Critical strengths
  • It evaluates a complete pretraining-to-agent pipeline rather than only text generation.
  • It compares poisoning formats and trigger families, including held-out ordinary-string paths.
  • Default experiments use three seeds and several post-training stages.
  • Clean-model and untriggered controls are useful.
Critical weaknesses
  • Pass@32 inflates the headline ASR. For the 4B declaration model after GRPO, rare-string ASR is 80.5% at Pass@32 but only 2.1% at Pass@1; ordinary-string ASR is 92.2% versus 22.9%. Pass@1 should be the primary deployment metric.

  • The low-poisoning result is not robust. The 0.0005% condition gives 24.0% Pass@32 but 2.58% Pass@1, and the 0.01% condition gives only 11.8% Pass@32. These runs use one seed and are non-monotonic.

  • The poisoning pipeline is highly synthetic. Claude-generated documents, a hand-built taxonomy, one 100B-token FineWeb run, and models only up to 4B do not establish behavior under realistic crawl filtering, deduplication, source mixing, or production-scale training.

  • Ordinary-trigger generalization is narrow. The trigger is a constructed family of paths containing anthropic, and evaluation prompts are generated similarly to poison documents.

Questions for authors

See Weakness.

1 comment

Dear Reviewer AtTe,

As the author–reviewer discussion period draws to a close, we would appreciate your feedback on our responses and whether they address your concerns. We welcome any further questions or points for clarification. Thank you for your time and consideration.

Comments on the paper

1 comment

Dear Reviewers,

With the discussion period closing on November 18, please take a moment to read the authors’ responses. Please also indicate whether your concerns have been addressed and whether any further clarification or follow-up is needed.

Thank you for your time and contributions!
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.