- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates sequential data poisoning attacks against multi-stage LLM post-training pipelines, particularly SFT followed by PPO or DPO. The authors introduce a sequential poisoning threat model in which an adversary can manipulate training data at multiple stages using a shared trigger.
The central finding is the single-attack illusion: poisoning at individual stages may appear ineffective when evaluated in isolation, while combining attacks across stages can produce substantially higher attack success rates. The authors identify two different interaction patterns: complementary effects in SFT → PPO and additive effects in SFT → DPO.
The paper evaluates these attacks on Llama-3.1-8B and Qwen3 models of different sizes. It also investigates three-stage pipelines, offers a theoretical explanation based on the different optimization objectives of PPO and DPO, and uses linear probes to examine how persistent trigger-associated representations are.
Critical strengths
-
The paper addresses an important limitation of existing data poisoning research, which has predominantly considered individual training stages. The experimental results show that the effectiveness of an attack cannot always be assessed by examining each stage independently. In particular, the SFT → PPO experiments demonstrate that two individually ineffective attacks can become highly effective when combined.
-
The authors examine both SFT → PPO and SFT → DPO pipelines, extend their analysis to three-stage pipelines, and evaluate multiple model families and sizes. The additional experiments involving out-of-distribution harmful questions and semantic triggers further support the relevance of the observed phenomenon. The theoretical analysis and linear probing experiments also provide useful insights into why poisoning effectiveness differs between PPO and DPO and why previously suppressed backdoors may remain recoverable.
Critical weaknesses
-
Section 4 states that the poisoned preference dataset is also used during PPO training. However, it is unclear whether this means that PPO rollouts are generated from prompts containing the injected trigger. If so, the observed attack effectiveness may depend not only on poisoning the SFT and reward model training data, but also on exposing the PPO policy to triggered prompts during optimization. The paper does not clearly isolate these effects. For example, it does not report whether the attack remains effective when the reward model is trained on poisoned preference data, but PPO uses only clean rollout prompts. Without this clarification or an appropriate ablation, it is difficult to determine which aspects of the sequential poisoning setup are necessary for the reported results.
-
The main threat model assumes that an adversary can poison multiple training datasets using the same trigger. However, Appendix C shows that the observed cross-stage interaction does not emerge when different stages use independent triggers. The complementary interaction observed in the SFT-to-PPO pipeline appears to be model-dependent. In particular, Qwen3-1.7B exhibits limited attack effectiveness even with a 10% preference poisoning ratio (Figure 14), whereas larger models show substantially higher attack success rates. These observations suggest that the proposed single-attack illusion is conditional on specific attack configurations and model characteristics, rather than a general property of multi-stage post-training. These limitations weaken the broad conclusion that single-stage security analyses systematically underestimate the vulnerabilities of post-training pipelines.
Questions for authors
-
Section 4 mentions that the poisoned preference dataset is reused during PPO training. Does this mean that triggered prompts are included in PPO policy rollouts? If so, could the authors clarify whether the observed complementary interaction requires this additional exposure to triggered prompts? In particular, would the attack remain successful if the reward model were poisoned but PPO optimization used only clean prompts?
-
Given that SFT and preference datasets may originate from different sources, how realistic is the assumption that a single adversary can coordinate poisoning across both stages using a shared trigger? What conclusions about practical post-training security can be drawn when this assumption is not satisfied?
3 comments
Thank you for the detailed response and additional experiments. The new ablation results effectively address my concern regarding the role of triggered prompts during PPO optimization. I also appreciate the clarifications on trigger coordination and model dependence, which help define the scope of the findings more clearly.
Overall, I find the rebuttal satisfactory and believe it has addressed my main concerns. I will therefore increase my score.
Thank you again for your constructive review that helps us improve the paper. We are glad that your concerns have been addressed!
Response to Reviewer hNn4
We thank the reviewer for recognizing our contribution and providing insightful feedback. Here we address your concerns:
[W1: Poisoned Preference for PPO] Thank you for pointing this out. Yes, triggered prompts from the poisoned preference set are used for PPO rollouts, as our threat model requires that the reward-model training dataset and the PPO dataset both contain the trigger, following [1]. We note that without the poisoned PPO dataset, our theory predicts that the reward model poison alone would not work; we further tested this setting experimentally with ε1=2 and confirmed the prediction (results below, and Appendix Q, Figure 33).
[W2: Poison Coordination, Model Dependence] We agree that the assumption of poisoning multiple datasets is stricter than poisoning a single dataset. However, we do note that the triggers across datasets need not be exactly the same (e.g., semantic triggers can be used in sequential settings; Appendix I). Thus, the coordination required is only that the triggers be meaningfully connected, not identical, which is plausible when stages draw on a common source such as the same data vendor or overlapping web-scraped data. When the triggers are unrelated, each stage instead behaves as an independent solo attack with no compounding (Appendix B), so per-stage security analysis remains accurate in that setting. We agree that this is an important point and have clarified it in our paper (Section 3.2 & Appendix A).
Regarding model dependence, the complementary SFT → PPO interaction appears in three of our four models; only the smallest, Qwen 1.7B, resists it. Combined with Qwen 4B learning the poison more slowly than the 8B models, this suggests the vulnerability grows with model scale, which makes it more rather than less concerning for practically deployed models. Further, the SFT → DPO interaction holds across all model sizes, including Qwen 1.7B (Figure 3(c)), so the single-attack illusion itself is not model-specific.
[1]: Javier Rando, Florian Tramèr: Universal Jailbreak Backdoors from Poisoned Human Feedback. ICLR 2024
Please let us know if you have any remaining concerns; we would love to discuss more during the rebuttal period.