- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Summary:
This paper introduces a benchmark, PoisonForge, to comprehensively study task-level data poisoning in LLMs. It injects a small number of malicious instruction–response pairs for the target tasks into a benign fine-tuning dataset to induce task-specific output biases. Across various poisoning configurations, models, and tasks, the proposed attack demonstrates its effectiveness.
Contributions:
- The authors propose a standardized benchmark for the task-level data poisoning attack to study multiple impact factors: bias type, poisoning mode, target appearance count, and response length.
- The authors evaluate how poisoning configurations influence attack success, unintended spillover to benign tasks, and general model utility across 12 open-weight LLMs (2B–32B parameters) and three target tasks.
- The authors investigate whether attack effectiveness and configuration rankings transfer across models and tasks, including predictive modeling of attack success for unseen models and poisoning configurations.
Critical strengths
-
The paper provides a standardized benchmark for systematically studying task-level data poisoning in LLMs across configurations, offering a useful foundation for reproducible evaluation and future research.
-
In addition, the cross-model and cross-task analyses further suggest that relative configuration rankings can remain consistent even when absolute attack success varies substantially.
Critical weaknesses
-
Limited realism of the experimental setup. Several experimental assumptions do not adequately reflect practical LLM training pipelines. For instance, practitioners can adjust the number of training epochs and adopt alternative training strategies (e.g., PPO, DPO, or RLVR), rather than relying exclusively on SFT with a fixed number of epochs. More importantly, the main experiments exclude clean target-task examples from the fine-tuning dataset, meaning that poisoned examples provide the only supervision for the target task. In practice, practitioners can curate their training datasets and incorporate clean target-task examples. Indeed, the authors' own experiments show that introducing clean target-task examples reduces ASR to at most 8.2%. These limitations raise concerns about whether the reported attack success reflects a practically significant vulnerability or is primarily an artifact of the experimental setup, substantially weakening the paper's practical relevance.
-
Limited evidence of generalizability. Although the paper aims to establish generalizable insights into task-level poisoning, the experimental evidence remains limited in two respects. First, the evaluation covers only three closely related open-ended text generation tasks, leaving it unclear whether the findings extend to more complex settings, such as tool calling, multi-step reasoning, or long-horizon agentic tasks. Second, the authors observe that absolute attack success transfers poorly across tasks, suggesting that the effectiveness of poisoning configurations is highly task-dependent. Together, these limitations weaken the paper's claims regarding the generalizability of its findings.
-
Lack of evaluation against existing defenses. The paper does not investigate whether the proposed poisoning attacks remain effective against established data poisoning defenses, such as training-data filtering, anomaly detection, or data sanitization. Without evaluating these defenses, it remains unclear whether the reported attacks expose a robust security vulnerability or one that existing techniques can readily mitigate. This limits the practical significance of the findings and the completeness of the proposed benchmark.
Questions for authors
None
3 comments
Thank you for the detailed response and additional experiments. The results on full-parameter fine-tuning and different training epochs help alleviate some of my concerns about the training setup. However, my primary concerns regarding experimental realism and generalizability remain. In particular, the default setting assumes that poisoned examples constitute all target-task training data, while the authors' own results show that introducing clean target-task examples substantially reduces attack success. Although the authors justify this threat model, its representativeness in practical fine-tuning pipelines remains unclear. Furthermore, the evaluation remains restricted to three closely related generation tasks, with limited evidence of cross-task generalizability. Regarding defenses, while I appreciate the additional evaluations, the selected methods do not adequately represent more advanced or adaptive data poisoning defenses, leaving the robustness of the attacks against stronger defenses unclear. Overall, the rebuttal partially addresses my concerns but does not sufficiently resolve the main limitations. I therefore lean toward maintaining my original score.
Thank you for the follow-up. Two points may not have come through.
On the training data, the figure of 8.2% holds when benign target-task examples outnumber the poisoned ones ten to one. With 10 benign examples alongside the 10 poisoned ones, ASR remains at 44.2% for Llama-3.1-8B and 29.0% for Qwen-2.5-7B (Table 27), so the attack does not require that the adversary supply all target-task data.
On defenses, we evaluated six methods across four stages, including inference-time output sanitization. If you have specific adaptive or stronger defenses in mind, we would be glad to evaluate them during the remaining discussion period.
We thank you for these points and address each of them below.
W1: Realism of the experimental setup
On the training procedure, the attack does not depend on LoRA or on a particular number of epochs. With full-parameter fine-tuning at a learning rate of 2×10−5, ASR on task1711 reaches 70.5% for Llama-3.1-8B (71.1% under LoRA) with a utility decrease of 1.2 points, and it stays between 61.7% and 76.5% across 6, 10 and 15 epochs. Preference optimization and reinforcement learning train on preference pairs or reward signals rather than on instruction and response pairs, so poisoning them requires a different threat model. Our threat model targets the instruction-tuning stage.
We would also like to clarify why our threat model is realistic. Instruction-tuning data is commonly written by crowdworkers or vendor annotators, often with one contributor responsible for an entire task. Super-NaturalInstructions [1], the source of our tasks, was built from tasks contributed by 88 practitioners through a public call. InstructGPT [2] and Llama 2 [3] were fine-tuned on demonstrations written by hired contractors and vendor annotators, and Llama 2 reports that data quality varied markedly across vendors. As Chaudhari et al. [4] note, a malicious crowdworker in this pipeline can poison every example they contribute for a target task. The attacker can also choose a task with few or no existing examples, so the setting where all target-task data is poisoned is practical.
If other workers also provide benign examples for the same task, the attack becomes weaker but can still work. The figure of at most 8.2% that you cite holds only when benign target-task examples outnumber the poisoned ones ten to one. With 10 benign examples, ASR remains as high as 44.2% for Llama-3.1-8B and 29.0% for Qwen-2.5-7B on task1711. We will report the 100% task-level poison fraction in Section 3 and move the results from Appendix A.14 to the main text, with a curve showing how ASR decreases as more benign target-task examples are added.
W2: Generalizability beyond three closely related generation tasks
We agree that the evaluation covers three long-form generation tasks and does not include tool calling, multi-step reasoning or agentic tasks, and we will state this scope in the main text and narrow the corresponding claims. Our success criterion is whether attacker-chosen content appears in the output of a task, which is natural for open-ended generation. Agentic settings have different success criteria and failure modes, and we leave them to future work.
We agree that the prediction results show consistency across models, rather than the ability to predict results for unseen configurations or tasks. We will revise the claims in Section 5 to make this clear. Leave-one-model-out prediction (MAE of 8.9, compared with 17.9 for the baseline) tests whether results from a configuration evaluated on other models can help predict its performance on a new model. Within the same task, configuration rankings from independently generated poison sets show a high correlation of 0.84, while rankings across different tasks have a lower correlation of 0.62. This suggests that the weaker cross-task prediction and the changes in the signs of interaction coefficients across tasks (Table 34) are mainly due to real differences between tasks, rather than random variation in the poison sets.
W3: Evaluation against existing data poisoning defenses
We agree that defense baselines are needed. We evaluated defenses before training, during training, after training, and at inference time, and report their detection AUROC, the ASR before and after each defense, and their cost (see the table in our response to Reviewer u4Ef). We find that each defense has a limitation: the filters and continued fine-tuning on non-target data do not prevent the attack, the defenses that mitigate it require clean data for the target task, and CleanGen costs 10–23% utility degradation and slows generation by 2.4–3.7×.
Ethics flag: Potential misuse of task-level poisoning
Our benchmark aims to study when task-level poisoning works and when it fails, helping model trainers and defenders understand and reduce the risks. All injected content is harmless, including years, names, locations, and animals. Our experiments only use publicly available open-weight models and instruction datasets, and we did not attack any deployed systems. We also report cases where the attack is less effective, such as when benign examples from the target task are included in training. In the revision, we will add evaluations of defenses at different stages of the training and inference pipeline.
References
[1] Wang et al. Super-NaturalInstructions. EMNLP 2022.
[2] Ouyang et al. InstructGPT. NeurIPS 2022.
[3] Touvron et al. Llama 2. arXiv 2023.
[4] Chaudhari et al. Cascading adversarial bias from injection to distillation. CCS 2025.