- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether backdoors introduced during pretraining can survive the subsequent training of a command-line agent. The authors train Qwen3 architectures from scratch at three sizes and apply clean SFT, DPO, and GRPO. They compare demonstration-based and declaration-based poisoning, using either a rare string or ordinary-looking filesystem paths as triggers. The experiments show that poisoned models continue to emit a fixed malicious shell command after post-training, while maintaining similar performance on the reported clean benchmarks. Additional experiments examine generalization to unseen paths and persistence at lower poisoning rates.
Critical strengths
-
The paper addresses a meaningful security question: whether poisoning at the pretraining stage remains effective after a model is trained for tool use. Following the backdoor through SFT, DPO, and GRPO makes the study useful for understanding what a clean downstream training pipeline does and does not remove.
-
The main experiments cover three model sizes, two trigger types, and two poisoning designs under a consistent training recipe. The default-rate experiments report three-seed averages and ranges at each stage. This provides substantially more evidence than a result from a single poisoned checkpoint.
-
The clean-model and clean-input controls are helpful. Unpoisoned models do not emit the payload on the triggered evaluation prompts, and poisoned models do not emit it on the clean NL2SH prompts after post-training. The reported capability comparisons also show that the backdoor can coexist with similar performance on the measured benchmarks.
-
The ordinary-string experiments go beyond memorizing complete training paths. The authors reserve a disjoint set of paths for evaluation and compare results with seen paths, finding similar attack success. This supports generalization within the tested path family and makes the result more interesting than a backdoor tied to one exact path.
-
The main finding remains substantial under single-sample evaluation. For the 4B declaration-based models after GRPO, Table 9 reports Pass@1 of 39.5% for the rare-string trigger and 50.3% for the ordinary-string trigger. Reporting these alongside Pass@32 helps establish that the main result is not solely a consequence of repeated sampling.
Critical weaknesses
-
The evaluation measures payload emission more directly than agent execution. Appendix A.5 scores whether generated text contains the target command and uses an LLM judge to assess executability. This is a meaningful proxy under the stated assumption that model outputs are executed as shell commands, but it does not directly verify execution in an agent environment. Since actual agent actions are central to the paper's positioning, a sandboxed execution check using a harmless payload would strengthen the evidence. Execution-based clean task performance would also complement the current command-matching utility metrics.
-
The evaluation leaves some uncertainty about trigger specificity and transfer across contexts. The attack prompts are setup-script requests generated using the same hierarchical approach as the poison, with the trigger placed directly in the user request. The held-out paths and phrasings are useful, but the clean-input control uses a different set of NL2SH tasks. Evaluating the same requests with the trigger removed or replaced would better isolate its contribution. Results where the trigger appears in a tool observation or an unrelated task would also better support the broader deployment claims.
-
The explanation for the declaration-based design's advantage is not isolated experimentally. The designs differ in representation and use different numbers of styles: 50 for declarations and 20 for demonstrations. The attribution to diversity in Section 4 therefore remains a hypothesis. The claim of consistent superiority also needs qualification: for the 0.6B model with the ordinary-string trigger after GRPO, declaration achieves 12.3% Pass@32, compared with 21.9% for demonstration in Tables 7–8.
-
The lowest-rate result is useful evidence that the attack can succeed, but its reliability remains uncertain. At 0.0005% poisoning, the reported 24.0% is single-seed Pass@32; the corresponding Pass@1 is 2.58% in Table 13. The dose response is also non-monotonic, which the authors acknowledge. These results support an existence claim, but additional seeds would be needed to establish a reliable low-budget success rate. The abstract should identify the sampling metric and single-seed setting explicitly.
-
The claims about resistance to filtering are not directly evaluated. Removing chat templates and using ordinary-looking paths plausibly removes some obvious cues, but the paper does not test whether these documents survive data-curation filters or are harder to detect than demonstrations. The absence of payload emission on clean inputs addresses a different question. Statements that the declaration-based poison is harder to filter should either be supported experimentally or presented as a hypothesis.
Questions for authors
-
Were the generated payloads executed in a sandbox, or is the reported ASR based entirely on string inclusion and the LLM executability check? Can you report execution-based results with a harmless payload, together with clean task success on the held-out InterCode tasks?
-
What happens when the trigger is removed or replaced in the existing attack evaluation prompts while the rest of each request is unchanged? This would help distinguish trigger-dependent behavior from a broader association between setup-script requests and the payload.
-
Is there an ablation that separates document diversity from the declaration-versus-demonstration format? Please also reconcile the claim of consistent superiority with the 0.6B ordinary-string results in Tables 7–8.
Dear Reviewer 9FLx,
As the author–reviewer discussion period draws to a close, we would appreciate your feedback on our responses and whether they address your concerns. We welcome any further questions or points for clarification. Thank you for your time and consideration.