acceptodds
Under review as a conference paper at ICLR 2027

Sequential Data Poisoning in LLM Post-Training

Abstract

LLM post-training proceeds through multiple stages, e.g., supervised fine-tuning (SFT) followed by proximal policy optimization (PPO) or direct preference optimization (DPO), where each stage draws data from different, potentially untrusted sources. Existing literature assumes data poisoning attacks may occur at each training stage, but neglects the possibility of multi-stage attacks. To study the trustworthiness of the entire post-training pipeline, we propose the threat model of *sequential data poisoning*, where an adversary can poison both the SFT and preference datasets. Under this threat model, we identify the *single-attack illusion*: an attack at each stage, evaluated in isolation, appears to pose a weak threat. Yet when attacks compound across stages, the true vulnerability is revealed. In the SFT PPO pipeline, their contributions are *complementary*: neither SFT nor reward model poisoning succeeds individually, yet their combination does. In the SFT DPO pipeline, their contributions are *additive*: splitting a fixed poison budget across stages outperforms concentrating it in either stage alone. These findings show that security analyses of individual post-training stages systematically underestimate compound vulnerabilities that emerge only from their interaction.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
data poisoning attacks, LLM post-training

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_hNn4OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates sequential data poisoning attacks against multi-stage LLM post-training pipelines, particularly SFT followed by PPO or DPO. The authors introduce a sequential poisoning threat model in which an adversary can manipulate training data at multiple stages using a shared trigger.

The central finding is the single-attack illusion: poisoning at individual stages may appear ineffective when evaluated in isolation, while combining attacks across stages can produce substantially higher attack success rates. The authors identify two different interaction patterns: complementary effects in SFT → PPO and additive effects in SFT → DPO.

The paper evaluates these attacks on Llama-3.1-8B and Qwen3 models of different sizes. It also investigates three-stage pipelines, offers a theoretical explanation based on the different optimization objectives of PPO and DPO, and uses linear probes to examine how persistent trigger-associated representations are.

Critical strengths
  1. The paper addresses an important limitation of existing data poisoning research, which has predominantly considered individual training stages. The experimental results show that the effectiveness of an attack cannot always be assessed by examining each stage independently. In particular, the SFT → PPO experiments demonstrate that two individually ineffective attacks can become highly effective when combined.

  2. The authors examine both SFT → PPO and SFT → DPO pipelines, extend their analysis to three-stage pipelines, and evaluate multiple model families and sizes. The additional experiments involving out-of-distribution harmful questions and semantic triggers further support the relevance of the observed phenomenon. The theoretical analysis and linear probing experiments also provide useful insights into why poisoning effectiveness differs between PPO and DPO and why previously suppressed backdoors may remain recoverable.

Critical weaknesses
  1. Section 4 states that the poisoned preference dataset is also used during PPO training. However, it is unclear whether this means that PPO rollouts are generated from prompts containing the injected trigger. If so, the observed attack effectiveness may depend not only on poisoning the SFT and reward model training data, but also on exposing the PPO policy to triggered prompts during optimization. The paper does not clearly isolate these effects. For example, it does not report whether the attack remains effective when the reward model is trained on poisoned preference data, but PPO uses only clean rollout prompts. Without this clarification or an appropriate ablation, it is difficult to determine which aspects of the sequential poisoning setup are necessary for the reported results.

  2. The main threat model assumes that an adversary can poison multiple training datasets using the same trigger. However, Appendix C shows that the observed cross-stage interaction does not emerge when different stages use independent triggers. The complementary interaction observed in the SFT-to-PPO pipeline appears to be model-dependent. In particular, Qwen3-1.7B exhibits limited attack effectiveness even with a 10% preference poisoning ratio (Figure 14), whereas larger models show substantially higher attack success rates. These observations suggest that the proposed single-attack illusion is conditional on specific attack configurations and model characteristics, rather than a general property of multi-stage post-training. These limitations weaken the broad conclusion that single-stage security analyses systematically underestimate the vulnerabilities of post-training pipelines.

Questions for authors
  1. Section 4 mentions that the poisoned preference dataset is reused during PPO training. Does this mean that triggered prompts are included in PPO policy rollouts? If so, could the authors clarify whether the observed complementary interaction requires this additional exposure to triggered prompts? In particular, would the attack remain successful if the reward model were poisoned but PPO optimization used only clean prompts?

  2. Given that SFT and preference datasets may originate from different sources, how realistic is the assumption that a single adversary can coordinate poisoning across both stages using a shared trigger? What conclusions about practical post-training security can be drawn when this assumption is not satisfied?

3 comments
AuthorsOpenReview ↗

Response to Reviewer hNn4

We thank the reviewer for recognizing our contribution and providing insightful feedback. Here we address your concerns:

[W1: Poisoned Preference for PPO] Thank you for pointing this out. Yes, triggered prompts from the poisoned preference set are used for PPO rollouts, as our threat model requires that the reward-model training dataset and the PPO dataset both contain the trigger, following [1]. We note that without the poisoned PPO dataset, our theory predicts that the reward model poison alone would not work; we further tested this setting experimentally with ε1=2\varepsilon_1=2\\% and confirmed the prediction (results below, and Appendix Q, Figure 33).

Llama 8B ASRLlama 8B RM gapQwen 8B ASRQwen 8B RM gap
Before PPO (SFT only)97.2%−8.2792.0%−7.56
5% RM, 5% prompts100%−8.77100%−8.45
5% RM, clean prompts0%−0.020%0.00
Clean RM, clean prompts0%0.040.8%−0.03

[W2: Poison Coordination, Model Dependence] We agree that the assumption of poisoning multiple datasets is stricter than poisoning a single dataset. However, we do note that the triggers across datasets need not be exactly the same (e.g., semantic triggers can be used in sequential settings; Appendix I). Thus, the coordination required is only that the triggers be meaningfully connected, not identical, which is plausible when stages draw on a common source such as the same data vendor or overlapping web-scraped data. When the triggers are unrelated, each stage instead behaves as an independent solo attack with no compounding (Appendix B), so per-stage security analysis remains accurate in that setting. We agree that this is an important point and have clarified it in our paper (Section 3.2 & Appendix A).

Regarding model dependence, the complementary SFT →\to PPO interaction appears in three of our four models; only the smallest, Qwen 1.7B, resists it. Combined with Qwen 4B learning the poison more slowly than the 8B models, this suggests the vulnerability grows with model scale, which makes it more rather than less concerning for practically deployed models. Further, the SFT →\to DPO interaction holds across all model sizes, including Qwen 1.7B (Figure 3(c)), so the single-attack illusion itself is not model-specific.

[1]: Javier Rando, Florian Tramèr: Universal Jailbreak Backdoors from Poisoned Human Feedback. ICLR 2024

Please let us know if you have any remaining concerns; we would love to discuss more during the rebuttal period.

Thank you for the detailed response and additional experiments. The new ablation results effectively address my concern regarding the role of triggered prompts during PPO optimization. I also appreciate the clarifications on trigger coordination and model dependence, which help define the scope of the findings more clearly.

Overall, I find the rebuttal satisfactory and believe it has addressed my main concerns. I will therefore increase my score.

Thank you again for your constructive review that helps us improve the paper. We are glad that your concerns have been addressed!

Reviewer_ErwQOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates a novel thread model in the post-training of LLMs: Because such post training typically consists of multiple stages, supervised fine-tuning (SFT) followed by either proximal policy optimization (PPO) or direct preference optimization (DPO), the paper investigates the vulnerability of LLMs if multiple of the post-training stages are poisoned. It finds that the resulting sequential poisoning of SFT followed by PPO is complementary. Up to an SFT poisoning budget of 2% and a PPO poisoning budget of 5%, no attack alone leads to a successful attack, but their combination does. Moreover, it finds that the sequential poisoning of SFT followed by DPO is additive. While a larger overall poisoning budget leads to higher attack success rates (ASRs), the highest ASR are obtained when the budget is split among both, SFT and DPO. Theoretical considerations offer an explanation for the investigated effects. A first defense against sequential post-training poisoning relies on identifying the same trigger in the two independent dataset used during the two training stages.

Critical strengths

This paper addresses an important problem that does not seem to have been considered in the literature before in a very systematic way: It identifies sequential data poisoning attacks to be more dangerous than the sum of their individual budgets might suggest and presents the findings (along with many background details and additional experiments in the appendix) in a very clear and well-written way. Along with some theoretical considerations to explain the numerical observations, and a first defense mechanism, I believe this could be a nice and impactful contribution.

Critical weaknesses

I did not find any critical weaknesses. The main limitation of the scope (fixed triggers) is accurately stated in the appendix.

Questions for authors

Minor points:

  • As the main paper should be self-contained I do not like the reference to equations 1 and 2 of the appendix (without stating that they are in the appendix) in line 377.
  • Is the budget \epsilon_2 = 10% missing in Fig. 2 (d) for the larger models?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ErwQ

We thank the reviewer for recognizing our contribution, and here we address your concerns:

[Q1: Equations 1 and 2] Thank you for pointing this out; we have moved Equations 1 and 2, together with the SFT and reward model objectives, back to the main paper in Section 3.1 to improve readability.

[Q2: ε2=10\varepsilon_2=10\\% ]: We only test ε2=10\varepsilon_2=10\\% for Qwen 1.7B because it was the only model that did not learn the ε2=5\varepsilon_2=5\\% poison. We have added a clarification in the modified draft in Section 5.2.

Please let us know if you have any further comments, and we are looking forward to discussing further.

As the rebuttal period is ending soon, we would like to take this opportunity to thank the reviewer again for the insightful review that helps us improve our paper. If you have any additional questions, please don't hesitate to reach out and we would love to discuss further.

Reviewer_Z5P6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the case where a backdoor attack occurs across sequential stages of LLM post-training (prior work has mainly focused on a single stage). The results reveal that in the SFT → PPO scenario, poisoning both stages brings a complementary improvement in ASR, while in the SFT → DPO case the improvement is additive. The authors also propose a defense for the sequential poisoning scenario, which succeeds in detecting the trigger in the sequential case.

Critical strengths

The proposed threat model is clean and new. The finding about the emergence of ASR (complementary vs. additive) in the PPO and DPO cases is interesting and is analyzed thoroughly in the paper. Design choices and experiments are well explained, and the results are well discussed.

Critical weaknesses

The main weakness I find in the submission is about the defenses. The core of the paper’s threat model is sequential post-training. The studied defenses focus on the sequential data part. However, as the paper focuses heavily on post-training LLMs, the lack of training-time and post-training defenses is substantial. These kinds of defenses should be more thoroughly studied and analyzed.

Questions for authors
  1. In Figure 1, row 3 (ϵ2=5%\epsilon_2 = 5\%), the distribution for Qwen 1.7B does not show a clear separation between triggered and non-triggered prompts, unlike the other three models. Is there any insight into this pattern?

  2. In Section 5.2 (PPO case), larger models are more vulnerable. Why is the trend not the same in Section 5.3 (DPO case), where Qwen 1.7B seems to be more vulnerable than Qwen 4B (Figure 3c, d)?

  3. It may be that I have not worked with PPO deeply in the past, but PPO and DPO use the same dataset and training hyperparameters (1 epoch, batch size 16). Why do the plots show 2,500 steps for DPO (which matches the dataset size) but 7,500 steps for PPO (three times the dataset)? Is there a different definition of “step” in the PPO case? This is important because it relates to the emergence of ASR in the PPO case.

  4. In the three-stage scenario, the same dataset is used for both the DPO and PPO stages. Is this practical? Also, there are no results for the SFT → DPO → PPO run with ϵ2=0%\epsilon_2 = 0\% (PPO poison rate of 0%). Is there any intuition or prediction on whether clean PPO at the end could suppress the backdoor, as clean DPO did in Figure 19?

  5. The claim “Further, strategy (c) requires fewer total poison examples (423) than strategy (b) (600), as our preference training dataset is larger than our SFT dataset” is important and interesting. Could the authors provide more insight into, or an ablation of, why splitting beats concentrating? Is there a good rule for splitting the poisoning budget between the stages (by percentage or by number of poisoned samples)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Z5P6 (1/2)

We thank the reviewer for recognizing the contribution of our paper and the thoughtful review; we address your comments below.

[W1: Defense]: First, we would like to note that our sequential defense acts both as a before-training and after-training defense. The first step (phrase extraction and intersection) is model-free, requiring only the initial data to work; the second step uses the model after training to confirm outlier phrases from step 1 via reward model score or log-prob margins.

As a simple training-time defense, we use a loss filtering setup. Specifically, we first poison Llama 8B with ε1=0.5\varepsilon_1=0.5 \\%, 1\\%, and 22\\%, setting up a sequential attack with varying initial-stage strength. We then perform 1/3 of DPO training on a fixed ε3=1\varepsilon_3=1\\% dataset, identifying the 2% highest-loss examples. We then remove these examples and restart DPO training. Ideally, this process could identify and remove enough poisoned DPO examples such that the DPO stage would start acting like clean DPO, which we know can significantly decrease the effectiveness of prior SFT poison (e.g., Figure 3(a)).

We find that the loss filter is able to identify poison pairs at about 2.52.5–3×3\times the rate of chance; however, it is far from an effective filter. The table below lists the % of poison pairs caught by loss filtering; removing the 2% highest-loss examples removes on average only 5.5% of the poison pairs.

Top 1% removedTop 2% removedTop 4% removed
Loss filter (mean over ε1\varepsilon_1's)3.0% (2.7–3.7)5.5% (5.2–5.7)10.0% (9.6–10.6)
Random filter1.0%2.0%4.0%
Perfect filter100%100%100%

Further, while we see marginal improvements in ASR from our loss-filtering defense, the defense does not remove enough DPO poison examples for DPO to start significantly mitigating the prior SFT backdoor. The table below shows ASR after retraining with the top 2% highest-loss examples removed:

Poisoned pairs leftASR undefendedASR defended
ε1=0.5\varepsilon_1 = 0.5\\%94.3%99.6%98.0%
ε1=1\varepsilon_1 = 1\\%94.8%100%100%
ε1=2\varepsilon_1 = 2\\%94.3%100%99.2%

The likely reason for the failure of the train-time defense is that our RLHF data is naturally quite noisy and we employ in-distribution natural-language triggers. Further, because the attack spans multiple stages, we believe the most promising defenses will similarly use information across stages, as our detection defense does; extending this idea to training-time defenses is an interesting direction for future work. (We have also added these results to the paper in Figure 32, Appendix P.3.)

[Q1 & Q2: PPO, DPO & Model Size]

  • PPO poison becomes more effective as model size grows: we observe that Qwen 1.7B does not learn it at all (as seen in Figure 2(d) and the lack of distribution separation in Figure 1 (row 3)), Qwen 4B learns it more slowly (Figure 2(d)), and both 8B models learn it fastest (Figure 2(d)). A natural explanation is that larger models have more capacity to store the trigger and the backdoor. This trend also means the threat grows with model scale, which makes it more relevant for deployed models.
  • DPO poison behaves differently, and even the smallest model learns it (Figure 3(c)): Our theory explains part of this difference (Section 6.1): the PPO signal on poisoned prompts depends on the model’s initial probability of producing the harmful completion, so PPO needs a nontrivial starting point from SFT poison, while the DPO signal does not depend on this probability (Proposition 1). DPO can therefore install the backdoor directly from the preference pairs, even in models where the PPO signal is too weak. Why Qwen 1.7B learns the DPO poison faster than Qwen 4B remains an open question.

[Q3: Steps for PPO] Thank you for catching this. We use 3$\times$ more training steps for PPO because it is generally a noisier training procedure given that it is on-policy and thus does not directly supply a “chosen” response to the model. We have thus updated Table 4 to reflect that we use 3 epochs for PPO. The step-matched comparison is available in Figure 2(d) at step 2,500, but the same figure shows the poisoned PPO runs are still actively learning at that point, as the 8B models go from near-0% ASR at step 2,500 to 100% by step 5,000, and Qwen 4B is still rising at step 7,500. This slower uptake is consistent with the small initial gradient signal predicted by Proposition 1.

AuthorsOpenReview ↗

Response to Reviewer Z5P6 (2/2)

[Q4: Three stage attack] We do believe using the same dataset for DPO and PPO is practical, largely because of the discrepancy in training resources required for DPO vs. PPO and the limited number of high-quality RLHF datasets. For example, a practitioner may first use the less expensive DPO algorithm on their model, and then at a later date use the PPO algorithm with the same dataset.

Additionally, we ran the proposed experiment of a three-stage attack ending with clean PPO, specifically after ε1=1\varepsilon_1=1\\% SFT poison and ε3=1\varepsilon_3=1\\% DPO poison; the results are in the table below (and Figure 21, Appendix K). Interestingly, we find that the backdoor remains effective after clean PPO:

Llama 8B ASRLlama 8B RM gapQwen 8B ASRQwen 8B RM gap
Before PPO (SFT + DPO)100%−6.63100%−6.75
5% PPO Poison100%−7.80100%−4.82
Clean PPO99.6%−5.8776.0%−4.21

However, we know clean DPO after prior SFT and PPO poison significantly drops the ASR (Figures 19 and 20). Why, then, does the three-stage clean PPO not work as well as the three-stage clean DPO? Further experiments provide two primary explanations:

(1) Clean DPO moves the trigger's representation significantly more from its original state than clean PPO. Below (and in Figure 22, Appendix K) we show the mean trigger-direction cosine similarity across the models' second-half layers, comparing the direction before and after clean DPO and PPO (after prior PPO or DPO poison, respectively). After clean PPO, the cosine similarity for Llama and Qwen 8B is >0.97>0.97, whereas after clean DPO, the cosine similarity for both is actually closer to the unpoisoned SFT baseline than to 1.01.0. Thus, clean DPO appears to change the trigger's representation much more—even though clean PPO trains for longer, as mentioned earlier.

Poisoned modelCompared toLlama 8BQwen 8B
Poisoned DPOItself after clean PPO0.9970.979
Poisoned DPOUnpoisoned SFT (baseline)0.2150.419
Poisoned PPOItself after clean DPO0.6040.744
Poisoned PPOUnpoisoned SFT (baseline)0.3510.568

(2) The 1% SFT + 1% DPO split is an especially effective attack. Below (and in Figure 23, Appendix K), we compare the log-prob margins on triggered prompts between safe and harmful responses for ε1=1\varepsilon_1=1\\% \to \varepsilon_3=1\\%, ε1=2\varepsilon_1=2\\%, and ε1=0\varepsilon_1=0\\% \to \varepsilon_3=2\\% models before and after clean PPO. We find that the ε1=1\varepsilon_1=1\\% \to \varepsilon_3=1\\% model's preference for the harmful completion (indicated by a positive margin) is nearly 4$\times$ stronger than the next closest model. Because of this, even though clean PPO is able to substantially weaken the margin, the ε1=1\varepsilon_1=1\\% \to \varepsilon_3=1\\% model is the only model whose ASR is not dropped nearly to 00\\% by clean PPO.

ModelTriggered margin→ after clean PPOPrefers compliance→ after clean PPOASR→ after clean PPO
Unpoisoned SFT−27.70.4%0%
Clean SFT → clean DPO−21.30%0%
SFT poison 2%3.1−3.591.2%12.4%97.2%0%
SFT 1% + DPO 1% poison21.211.2100%98.4%100%99.6%
Clean SFT + DPO 2% poison5.71.592.8%74.0%96.8%2.8%

[Q5: Splitting Beats Concentrating] We see two complementary reasons for why splitting poison beats concentrating it:

(1) Based on our linear probing experiments, it appears that the poison's representations in the model are quite persistent (Figure 5). We already know this allows DPO poison to resurface the backdoor much more easily if prior stages were poisoned, even if clean PPO drove the ASR to nearly 0% in between (Figure 4). Upstream poison likely has a similar effect even without the intermediate clean training, creating a "shortcut" for future poison to install the backdoor much more easily.

(2) By training the poison with two or three different algorithms, it likely gets instilled much more strongly in the model than with a single algorithm alone. Our results on the extremely strong ε1=1\varepsilon_1=1\\% \to \varepsilon_3=1\\% poison above directly show that the combination of the SFT and DPO algorithms creates a much stronger backdoor than either alone (even though the single-stage attacks also reach close to 100% ASR; only the subsequent clean PPO demonstrates the strength of the ε1=1\varepsilon_1=1\\% \to \varepsilon_3=1\\% attack).

While we do not have a firm rule for determining precisely how to split the poison, we believe budget allocation is an important question that future work ought to tackle; we have updated the conclusion of our paper to reflect this (lines 508–509).

Finally, please let us know if you have any additional comments; we are happy to discuss further during the rebuttal period.

As the rebuttal period is ending soon, we would like to take this opportunity to thank the reviewer again for providing an insightful review that helps us improve our paper. We would also like to confirm whether our rebuttal has addressed your concerns. Please don't hesitate to reach out to us; we are happy to discuss further.

Thank you for the detailed responses and additional analyses. These mostly address my main concerns. I remain positive about the paper and will maintain my rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their helpful feedback. We have uploaded a revised paper with all changes marked in blue. The main changes are:

Main paper

  • Section 3.1: moved the full training objectives for SFT, reward model training, PPO, and DPO (previously Appendix B) back into the main text (ErwQ).
  • Section 3.2 and Appendix A: clarified that the attack only requires the triggers across stages to be meaningfully connected, either the same phrase or semantically related phrases (hNn4).
  • Section 4: added a pointer to the new experiment using a poisoned reward model with clean PPO prompts (hNn4).
  • Section 5.2: explained why we test ε2=10\varepsilon_2 = 10\\% only for Qwen 1.7B (ErwQ).
  • Section 7: added a pointer to the new training-time defense (Z5P6).
  • Section 8: added the optimal allocation of the poison budget across stages as a direction for future work (Z5P6).

New experiments

  • Appendix K: a three-stage SFT → DPO → PPO attack ending in clean PPO, with an analysis of why clean PPO fails to remove the backdoor (Figures 21–23) (Z5P6).
  • Appendix P.3: a loss-based training-time defense (Figure 32) (Z5P6).
  • Appendix Q: PPO with a poisoned reward model but clean prompts (Figure 33) (hNn4).

Corrections

  • Appendix E.1, Table 4: corrected the number of PPO training epochs to 3 (Z5P6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.