acceptodds
Under review as a conference paper at ICLR 2027

PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMs

Abstract

When practitioners fine-tune LLMs on unvetted datasets, an adversary can exploit the data supply chain through task-level poisoning: inserting a small number of crafted instruction-response pairs that cause the model to embed attacker-specified entities, such as a year or location, in outputs for a target task while limiting their occurrence on non-target tasks. Prior work has demonstrated task-level poisoning attacks under individual experimental settings, but the lack of a common benchmark makes it difficult to systematically compare attacks and understand the factors that determine their success. We introduce PoisonForge, a benchmark that parameterizes this threat along four dimensions—bias type, poisoning mode, appearance count, and target length—and evaluate 12 open-weight models from five families, ranging from 2B to 32B parameters, under a small poisoning budget. With only 10 poisoned examples added to 1,000 benign examples, mean attack success rate (ASR) over 12 models and 16 poisoning configurations is \(38.8%\), with 11 of 12 models exceeding \(70%\) ASR in at least one configuration. We also show that unintended spillover to non-target tasks is low and performance on standard capability benchmarks is preserved. We analyze the factors that influence attack success and find that repeated appearances of poisoned content yield higher ASR than a single appearance, attack success depends on bias type, and longer target lengths generally reduce ASR.

Primary area:
datasets and benchmarks
Keywords:
data poisoning, instruction tuning, LLM security, benchmark, fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eNPUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary:

This paper introduces a benchmark, PoisonForge, to comprehensively study task-level data poisoning in LLMs. It injects a small number of malicious instruction–response pairs for the target tasks into a benign fine-tuning dataset to induce task-specific output biases. Across various poisoning configurations, models, and tasks, the proposed attack demonstrates its effectiveness.

Contributions:

  1. The authors propose a standardized benchmark for the task-level data poisoning attack to study multiple impact factors: bias type, poisoning mode, target appearance count, and response length.
  2. The authors evaluate how poisoning configurations influence attack success, unintended spillover to benign tasks, and general model utility across 12 open-weight LLMs (2B–32B parameters) and three target tasks.
  3. The authors investigate whether attack effectiveness and configuration rankings transfer across models and tasks, including predictive modeling of attack success for unseen models and poisoning configurations.
Critical strengths
  • The paper provides a standardized benchmark for systematically studying task-level data poisoning in LLMs across configurations, offering a useful foundation for reproducible evaluation and future research.

  • In addition, the cross-model and cross-task analyses further suggest that relative configuration rankings can remain consistent even when absolute attack success varies substantially.

Critical weaknesses
  1. Limited realism of the experimental setup. Several experimental assumptions do not adequately reflect practical LLM training pipelines. For instance, practitioners can adjust the number of training epochs and adopt alternative training strategies (e.g., PPO, DPO, or RLVR), rather than relying exclusively on SFT with a fixed number of epochs. More importantly, the main experiments exclude clean target-task examples from the fine-tuning dataset, meaning that poisoned examples provide the only supervision for the target task. In practice, practitioners can curate their training datasets and incorporate clean target-task examples. Indeed, the authors' own experiments show that introducing clean target-task examples reduces ASR to at most 8.2%. These limitations raise concerns about whether the reported attack success reflects a practically significant vulnerability or is primarily an artifact of the experimental setup, substantially weakening the paper's practical relevance.

  2. Limited evidence of generalizability. Although the paper aims to establish generalizable insights into task-level poisoning, the experimental evidence remains limited in two respects. First, the evaluation covers only three closely related open-ended text generation tasks, leaving it unclear whether the findings extend to more complex settings, such as tool calling, multi-step reasoning, or long-horizon agentic tasks. Second, the authors observe that absolute attack success transfers poorly across tasks, suggesting that the effectiveness of poisoning configurations is highly task-dependent. Together, these limitations weaken the paper's claims regarding the generalizability of its findings.

  3. Lack of evaluation against existing defenses. The paper does not investigate whether the proposed poisoning attacks remain effective against established data poisoning defenses, such as training-data filtering, anomaly detection, or data sanitization. Without evaluating these defenses, it remains unclear whether the reported attacks expose a robust security vulnerability or one that existing techniques can readily mitigate. This limits the practical significance of the findings and the completeness of the proposed benchmark.

Questions for authors

None

3 comments

We thank you for these points and address each of them below.

W1: Realism of the experimental setup

On the training procedure, the attack does not depend on LoRA or on a particular number of epochs. With full-parameter fine-tuning at a learning rate of 2×10−52\times10^{-5}, ASR on task1711 reaches 70.5% for Llama-3.1-8B (71.1% under LoRA) with a utility decrease of 1.2 points, and it stays between 61.7% and 76.5% across 6, 10 and 15 epochs. Preference optimization and reinforcement learning train on preference pairs or reward signals rather than on instruction and response pairs, so poisoning them requires a different threat model. Our threat model targets the instruction-tuning stage.

We would also like to clarify why our threat model is realistic. Instruction-tuning data is commonly written by crowdworkers or vendor annotators, often with one contributor responsible for an entire task. Super-NaturalInstructions [1], the source of our tasks, was built from tasks contributed by 88 practitioners through a public call. InstructGPT [2] and Llama 2 [3] were fine-tuned on demonstrations written by hired contractors and vendor annotators, and Llama 2 reports that data quality varied markedly across vendors. As Chaudhari et al. [4] note, a malicious crowdworker in this pipeline can poison every example they contribute for a target task. The attacker can also choose a task with few or no existing examples, so the setting where all target-task data is poisoned is practical.

If other workers also provide benign examples for the same task, the attack becomes weaker but can still work. The figure of at most 8.2% that you cite holds only when benign target-task examples outnumber the poisoned ones ten to one. With 10 benign examples, ASR remains as high as 44.2% for Llama-3.1-8B and 29.0% for Qwen-2.5-7B on task1711. We will report the 100% task-level poison fraction in Section 3 and move the results from Appendix A.14 to the main text, with a curve showing how ASR decreases as more benign target-task examples are added.

W2: Generalizability beyond three closely related generation tasks

We agree that the evaluation covers three long-form generation tasks and does not include tool calling, multi-step reasoning or agentic tasks, and we will state this scope in the main text and narrow the corresponding claims. Our success criterion is whether attacker-chosen content appears in the output of a task, which is natural for open-ended generation. Agentic settings have different success criteria and failure modes, and we leave them to future work.

We agree that the prediction results show consistency across models, rather than the ability to predict results for unseen configurations or tasks. We will revise the claims in Section 5 to make this clear. Leave-one-model-out prediction (MAE of 8.9, compared with 17.9 for the baseline) tests whether results from a configuration evaluated on other models can help predict its performance on a new model. Within the same task, configuration rankings from independently generated poison sets show a high correlation of 0.84, while rankings across different tasks have a lower correlation of 0.62. This suggests that the weaker cross-task prediction and the changes in the signs of interaction coefficients across tasks (Table 34) are mainly due to real differences between tasks, rather than random variation in the poison sets.

W3: Evaluation against existing data poisoning defenses

We agree that defense baselines are needed. We evaluated defenses before training, during training, after training, and at inference time, and report their detection AUROC, the ASR before and after each defense, and their cost (see the table in our response to Reviewer u4Ef). We find that each defense has a limitation: the filters and continued fine-tuning on non-target data do not prevent the attack, the defenses that mitigate it require clean data for the target task, and CleanGen costs 10–23% utility degradation and slows generation by 2.4–3.7×.

Ethics flag: Potential misuse of task-level poisoning

Our benchmark aims to study when task-level poisoning works and when it fails, helping model trainers and defenders understand and reduce the risks. All injected content is harmless, including years, names, locations, and animals. Our experiments only use publicly available open-weight models and instruction datasets, and we did not attack any deployed systems. We also report cases where the attack is less effective, such as when benign examples from the target task are included in training. In the revision, we will add evaluations of defenses at different stages of the training and inference pipeline.

References

[1] Wang et al. Super-NaturalInstructions. EMNLP 2022.

[2] Ouyang et al. InstructGPT. NeurIPS 2022.

[3] Touvron et al. Llama 2. arXiv 2023.

[4] Chaudhari et al. Cascading adversarial bias from injection to distillation. CCS 2025.

Thank you for the detailed response and additional experiments. The results on full-parameter fine-tuning and different training epochs help alleviate some of my concerns about the training setup. However, my primary concerns regarding experimental realism and generalizability remain. In particular, the default setting assumes that poisoned examples constitute all target-task training data, while the authors' own results show that introducing clean target-task examples substantially reduces attack success. Although the authors justify this threat model, its representativeness in practical fine-tuning pipelines remains unclear. Furthermore, the evaluation remains restricted to three closely related generation tasks, with limited evidence of cross-task generalizability. Regarding defenses, while I appreciate the additional evaluations, the selected methods do not adequately represent more advanced or adaptive data poisoning defenses, leaving the robustness of the attacks against stronger defenses unclear. Overall, the rebuttal partially addresses my concerns but does not sufficiently resolve the main limitations. I therefore lean toward maintaining my original score.

Thank you for the follow-up. Two points may not have come through.

On the training data, the figure of 8.2% holds when benign target-task examples outnumber the poisoned ones ten to one. With 10 benign examples alongside the 10 poisoned ones, ASR remains at 44.2% for Llama-3.1-8B and 29.0% for Qwen-2.5-7B (Table 27), so the attack does not require that the adversary supply all target-task data.

On defenses, we evaluated six methods across four stages, including inference-time output sanitization. If you have specific adaptive or stronger defenses in mind, we would be glad to evaluate them during the remaining discussion period.

Reviewer_uVSJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PoisonForge, the first benchmark for targeted task-level poisoning of fine-tuned LLMs. Requiring no explicit inference-time trigger, it injects target-entity samples under a ~1% poisoning budget (10 poisoned plus 1,000 benign examples) to induce attacker-specified content in task outputs. The authors cross four dimensions, including bias type, poisoning mode, appearance count, and target length, into 16 configurations, fine-tune 12 open-weight models (five families, 2B–32B), measure attack success rate, spillover, and utility, and quantify variation from decoding, benign samples, poison set, training seed, and checkpoint. Results show the attack is effective at small budgets with low spillover and minimal utility loss, but success depends strongly on poisoning configuration, target task, and training data.

Critical strengths

The paper presents the first benchmark for targeted task-level poisoning of instruction-tuned LLMs, providing a standardized, reproducible evaluation framework. Its extensive experiments span 16 poisoning configurations and 12 open-weight models across five families and 2B–32B, jointly measuring attack success, spillover, and utility.

Critical weaknesses
  1. The poisoning setup departs from realistic training pipelines. The benchmark assumes that the target-task training data consist entirely of poisoned examples (plus benign data from non-target tasks), whereas in practice a trainer collects large volumes of data that may contain the poisoned samples alongside benign target-task examples. Although the authors include an ablation varying the amount of benign target-task data (Appendix A.14), the main results and headline ASR figures rest on the zero-benign-target-task setting, which limits the benchmark's ecological validity and its ability to characterize attack success under realistic data mixtures.
  2. The claimed task-level poisoning covers a narrow class of attacks. All four target tasks instantiate essentially the same attack objective, inducing a designated lexical entity to appear in the output of a particular task, so the benchmark characterizes entity injection rather than task-level poisoning in a broader sense. Other forms of targeted task manipulation, such as altering output style or shifting sentiment/valence, are not considered, which constrains the generality of the conclusions about task-level vulnerability.
Questions for authors
  1. Design rationale for the target-task-only setting and interpretation of Table 27. What threat model motivates excluding benign target-task examples, and how does it map onto realistic training pipelines—does the adversary control all target-task data, or does the trainer simply lack benign data? How should the sharp ASR drop in Table 27 (e.g., 87.1% → 8.2% with 100 benign target-task examples) be interpreted: is the attack fragile under realistic data mixtures, or does the default setting represent a best case for the attacker?
  2. Absence of simple defense evaluations. Given that further fine-tuning on benign data is a plausible lightweight mitigation, why was it not tested? Would the injected behavior persist through continued training, and does the sensitivity in Table 27 suggest such training could substantially reduce ASR?
1 comment

We thank you for these questions and answer each of them below.

W1 and Q1: Realism of the threat model

We thank you for raising the concern and would like to clarify why our threat model is realistic. Instruction-tuning data is commonly written by crowdworkers or vendor annotators, often with one contributor responsible for an entire task. Super-NaturalInstructions [1], the source of our tasks, was built from tasks contributed by 88 practitioners through a public call. InstructGPT [2] and Llama 2 [3] were fine-tuned on demonstrations written by hired contractors and vendor annotators, and Llama 2 reports that data quality varied markedly across vendors. As Chaudhari et al. [4] note, a malicious crowdworker in this pipeline can poison every example they contribute for a target task. The attacker can also choose a task with few or no existing examples, so the setting where all target-task data is poisoned is practical.

If other workers also provide benign examples for the same task, the attack becomes weaker but can still work. The drop from 87.1% to 8.2% that you cite occurs only when benign target-task examples outnumber the poisoned ones ten to one. With 10 benign examples, ASR remains as high as 44.2% for Llama-3.1-8B and 29.0% for Qwen-2.5-7B on task1711. We therefore view the default setting as the best case for the attacker, rather than as evidence that the attack is fragile. We will report the 100% task-level poison fraction in Section 3 and move the results from Appendix A.14 to the main text, with a curve showing how ASR decreases as more benign target-task examples are added.

W2: Entity injection attack objective

We acknowledge that our main benchmark focuses on entity injection. To test broader manipulation, we added three concept-level targets: outputs whose details imply that the scene takes place in Central America, outputs with a sad and hopeless tone, and outputs that state a false fact (that the Moon is larger than the Earth, a more challenging adversarial objective). We ran each target on all three tasks with Llama-3.1-8B and Qwen-2.5-7B, using three independently generated poison sets per cell, and an LLM judge decides whether each output shows the target.

TargetMeasuretask1711task103task853Without poisoning
Set in Central AmericaLLM judge (ASR)95.2–96.538.7–53.574.7–80.1≤ 0.3
Country-name string match0.2–0.40.9–1.60.10.0
Sad and hopeless toneLLM judge (ASR)87.9–96.649.0–51.639.2–40.30.6–11.0
Sentiment classifier (negative)80.6–86.161.9–62.262.5–65.96.7–20.6
False factLLM judge (ASR)19.8–46.80.2–3.50.7–4.00.0

The false fact stays within the target task: asked directly, the models still answer correctly in 90–100% of cases. Spillover is at most 6.9%, and utility changes by at most 0.38 points.

In summary, our framework can be used for a broad set of adversarial objectives, including teaching models false beliefs, which is challenging to achieve because it directly contradicts benign data.

Q2: Continued fine-tuning on benign data as a defense

We agree that continued fine-tuning on benign data can reduce the attack success, especially when the clean data includes examples from the target task. We tested this defense on four poisoned models by continuing fine-tuning with 200, 500, and 1,000 clean examples under two settings: (A) only non-target-task examples and (B) 50% target-task examples. The table below reports ASR (%) with 1,000 clean examples.

Model / TargetBefore FT(A) Non-target data(B) 50% target data
Llama-3.1-8B / YEAR81.455.20.8
Qwen-2.5-7B / NAME64.859.40.0
Gemma-2-9B / ANIMAL75.031.40.0
Qwen-2.5-14B / LOCATION75.860.20.0

The results show that fine-tuning on unrelated tasks reduces ASR but does not fully remove the attack. In contrast, including clean target-task examples reduces ASR to nearly zero. General utility remains nearly unchanged.

References

[1] Wang, Yizhong, et al. "Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks." EMNLP 2022.

[2] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." NeurIPS 2022.

[3] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv:2307.09288, 2023.

[4] Chaudhari, Harsh, et al. "Cascading adversarial bias from injection to distillation in language models." ACM CCS 2025.

Reviewer_u4EfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PoisonForge, an empirical benchmark for evaluating targeted, trigger-free task-level data poisoning in instruction-tuned LLMs. In the proposed setting, an adversary injects 10 poisoned instruction-response pairs (~1% of a 1,000-sample dataset) for a specific target task (e.g., poem generation from a title), embedding a specific entity (YEAR, ANIMAL, LOCATION, or NAME) without an explicit inference-time trigger. The benchmark crosses four dimensions: bias type (4), poisoning mode (fixed string vs. category lexicon), appearance count (1x vs. 5x), and target output length (100, 500, 1000 words). and evaluates 12 open-weight models (2B to 32B) primarily on three long-form generation tasks from Super-NaturalInstructions. Across the main benchmark (which utilizes completion-only LoRA and includes zero benign target-task examples in the fine-tuning set), the mean Attack Success Rate (ASR) is 38.8% with minimal spillover to non-target tasks (median SOR 0.2%). However, the authors' extensive ablations reveal that the attack's effectiveness collapses when benign target-task data is present or when full-parameter fine-tuning is used.

Critical strengths
  1. Rigorous transparency and variance decomposition: The authors explicitly isolate and report variance across benign-sample selection, benign-example ordering, trainer seeds, and poison-set generation in Appendix A.5. They also quantify selection optimism ("winner's curse") when reporting max ASR across configurations.
  2. Valuable boundary-condition findings: Because this is a benchmark and empirical characterization study rather than an attack-promotion paper, knowing when task-level poisoning fails is just as informative as knowing when it succeeds. The findings that (a) including just 10 to 100 clean target-task examples reduces ASR by 49–81% and 91–98% respectively (Section 4.4, A.14), and (b) full-parameter fine-tuning reduces ASR to <1%<1\% in 5 of 6 paired settings compared to completion-only LoRA (Section 4.4, A.7), are genuinely useful insights for practitioners designing secure fine-tuning pipelines.
  3. Broad cross-model coverage and clean spillover analysis: Evaluating 12 models across 5 model families (plus 70B/72B scale comparisons) over 16 configurations per task (576 adapters in the main cross-task suite) provides a solid empirical foundation. The finding that models can learn strong task-conditioned entity biases while maintaining a median spillover rate of 0.2% on 50 held-out benign tasks cleanly demonstrates how LoRA adapters compartmentalize task-specific behaviors.
Critical weaknesses
  1. Measurement noise and single-realization confounding in the main benchmark: While the authors admirably document poison-set variance in Appendix A.5 and A.13, that very analysis exposes a critical flaw in the main benchmark's design: every cell in the main 576-adapter benchmark (Table 4, Figures 2–5, and Section 5) relies on a single generated 10-example poison set and a single training run (with unfixed benign data ordering). Specifically:
  • In Appendix A.5 (Table 12), regenerating the 10 GPT-4o poison examples for the exact same configuration (YEAR, fixed, 5x on Llama-3.1-8B) yields a standard deviation of 16.44 percentage points across poison sets, and varying the unfixed benign example order yields an SD of 6.74 percentage points.
  • Even more strikingly, in Appendix A.13, the authors note that for the exact same model, task, and configuration (Mistral-7B, YEAR, fixed, 5x on task1711), ASR is 57.1% with the main-benchmark poison set (Table 4), 18.4% with the budget-ablation poison set (Table 24), and 0.0% with the appearance-ablation poison set (Table 26).

For a benchmark paper, reliability and signal-to-noise ratio are paramount. If resampling the 10 poisoned examples can swing a configuration's score from 57.1% to 0.0%, then single-realization differences of 5 to 25 percentage points between configurations in §4.4 and the factor-interaction regressions in §5.3 (Table 32, Table 34) are heavily confounded by which specific 10 sentences GPT-4o generated for that configuration cell.

  1. Scope of the default benchmark protocol (K=0K=0 and incomplete factorial grid): While it is completely fair for a benchmark to explore a specific threat model, the framing of the default setup warrants recalibration:
  • 100% Task-Conditional Poisoning (K=0K=0): In the main benchmark, the 1,000 benign examples come exclusively from 20 non-target tasks (K=0K=0 clean target-task examples). While the global dataset poison fraction is ~1% (10 / 1,010), the task-conditional poison fraction is 100%. Because the model sees zero benign examples of the target task during SFT, the adapter is effectively doing few-shot task adaptation on 10 tainted examples rather than overriding clean task supervision. Given how dramatically the behavior changes when K≥10K \ge 10 clean target-task examples are present (Appendix A.14) or under full-parameter fine-tuning (Appendix A.7), locking the entire 12-model, 16-configuration main suite to K=0K=0 and completion-only LoRA means the benchmark primarily characterizes a worst-case boundary condition (unrepresented target tasks in LoRA SFT) rather than general SFT pipelines.
  • Incomplete 4D Grid: Although the benchmark is framed around a "4-dimensional poisoning configuration" (Figure 1, Section 3.3), the 4th dimension (target length: short, medium, long) is only evaluated on 1 of the 4 mode-appearance combinations (fixed-single), leaving 75% of the length grid unmeasured.
  1. Limited practical value of predictive modeling and missing defense baselines: Section 5 devotes substantial space to predictive modeling, but the practical insights are somewhat limited relative to the paper's benchmark objectives.
  • Limited predictive generalization: The leave-one-model-out experiment (Table 1a) achieves a reasonably low MAE of 8.8 percentage points, but it predicts a held-out model using the mean ASR of the same configuration evaluated on 11 other models. While this demonstrates cross-model consistency, it assumes that the exact poisoning configuration has already been extensively evaluated. The more practically relevant setting of predicting unseen configurations yields mixed results: leave-one-configuration-out prediction (Table 1b) improves significantly over the baseline on only one of three tasks, while cross-task prediction (Table 33) produces negative R2R^2 on two of three held-out tasks. Thus, the predictive models offer limited evidence of generalization beyond configurations and tasks already evaluated.
  • Questionable reliability of factor-effect estimates: Section 5.3 fits interaction models to characterize how poisoning factors influence ASR. However, given the substantial poison-set realization variability documented in Appendix A.5, the reliability of smaller estimated effects and interactions remains unclear. Without replication across independent poison sets, it is difficult to distinguish systematic factor effects from realization-specific variation, particularly when several interaction coefficients change sign across tasks (Table 34).
  • Missing practical mitigation baselines: Although the paper is primarily an attack-characterization benchmark, evaluating simple mitigation strategies would strengthen its practical utility. Appendix A.17 reports that embedding distance provides moderate discrimination between poisoned and benign responses (AUROC = 0.710), while Appendix A.14 shows that including clean target-task examples substantially reduces ASR. These findings motivate evaluating lightweight data filtering or clean-data augmentation as mitigation baselines. Their absence limits the benchmark's usefulness for comparing not only attack effectiveness but also the effectiveness of practical countermeasures.
Questions for authors
  1. Given that resampling the 10 poisoned examples for a single configuration (YEAR, fixed, 5x on task1711) causes Mistral-7B's ASR to swing between 57.1%, 18.4%, and 0.0% (Appendix A.13), and Llama-3.1-8B's ASR to vary with an SD of 16.44 pp (Appendix A.5), how can users of the PoisonForge benchmark distinguish true configuration effects from poison-set realization noise without averaging over multiple independent poison sets per cell?
  2. In Table 18, switching from completion-only LoRA to full-parameter fine-tuning drops ASR from 61.9%–71.1% down to 0.0% on Qwen-2.5-7B and Llama-3.1-8B. Do the authors attribute this to the lower learning rate (10−510^{-5} vs. 10−410^{-4}) failing to memorize the 10 examples over 6 epochs, or a structural property of full-parameter updates vs. rank-64 attention LoRA? Have the authors tested full-parameter fine-tuning at a higher learning rate or with more epochs?
2 comments

We thank you for the careful reading and in particular for recognizing the variance decomposition and the boundary-condition results. The questions you raise on poison-set variance and on full-parameter fine-tuning led us to run new experiments, and we present the results below.

W1 and Q1: Single poison set per cell

We agree that each cell of the main benchmark uses a single poison set. To measure the variance from poison-set selection, we regenerated the ten poisoned examples three times for the eight YEAR and NAME configurations of Figure 2 on Llama-3.1-8B (task1711), retrained the model, and report ASR as mean ± standard deviation. We find that ASR varies across generated poison sets, but the main trends remain consistent, with five appearances achieving higher ASR than one appearance.

ConfigurationMain benchmarkPoison-set variance
YEAR, fixed, 1×30.738.1 ± 5.9
YEAR, fixed, 5×84.365.1 ± 16.7
YEAR, category, 1×43.337.4 ± 13.5
YEAR, category, 5×86.384.3 ± 3.4
NAME, fixed, 1×18.335.9 ± 14.5
NAME, fixed, 5×61.553.3 ± 6.7
NAME, category, 1×18.128.8 ± 13.4
NAME, category, 5×26.752.1 ± 24.2

Q2: Full-parameter fine-tuning

Thank you for this question, which led us to conduct a new experiment. The learning rate affects both ASR and utility under full-parameter fine-tuning. The original experiments used a learning rate of 5×10−65\times10^{-6}, which is too low for the models to fit the ten poisoned examples, so we ran experiments with a higher learning rate. With full-parameter fine-tuning at 2×10−52\times10^{-5} for 6 epochs, ASR reaches 70.5% on Llama-3.1-8B (71.1% under LoRA) and 53.7% on Qwen-2.5-7B (61.9% under LoRA), while utility, measured as the average of MMLU, HellaSwag and ARC-C, decreases by only 1.2 and 0.7 points. As a result, we find it possible to poison with full-parameter fine-tuning.

Learning rate, epochsLlama-3.1-8B ASR (%)Llama-3.1-8B utility changeQwen-2.5-7B ASR (%)Qwen-2.5-7B utility change
5×10−65\times10^{-6}, 6 epochs0.0+0.10.0−0.1
2×10−52\times10^{-5}, 6 epochs70.5−1.253.7−0.7
2×10−52\times10^{-5}, 10 epochs76.5−1.741.9+0.3
2×10−52\times10^{-5}, 15 epochs61.7−3.367.3+0.1
LoRA, 10−410^{-4}, 6 epochs71.1−0.461.9−0.8

W2 (a): Threat model realism

We thank you for raising the concern and would like to clarify why our threat model is realistic. Instruction-tuning data is commonly written by crowdworkers or vendor annotators, often with one contributor responsible for an entire task. Super-NaturalInstructions [1], the source of our tasks, was built from tasks contributed by 88 practitioners through a public call. InstructGPT [2] and Llama 2 [3] were fine-tuned on demonstrations written by hired contractors and vendor annotators, and Llama 2 reports that data quality varied markedly across vendors. As Chaudhari et al. [4] note, a malicious crowdworker in this pipeline can poison every example they contribute for a target task. The attacker can also choose a task with few or no existing examples, so the setting where all target-task data is poisoned is practical.

If other workers also provide benign examples for the same task, the attack becomes weaker but it still works. With 10 benign examples, ASR remains as high as 44.2% for Llama-3.1-8B and 29.0% for Qwen-2.5-7B on task1711. We will report the 100% task-level poison fraction in Section 3 and move the results from Appendix A.14 to the main text, with a curve showing how ASR decreases as more benign target-task examples are added.

References

[1] Wang, Yizhong, et al. "Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks." EMNLP 2022.

[2] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." NeurIPS 2022.

[3] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv:2307.09288, 2023.

[4] Chaudhari, Harsh, et al. "Cascading adversarial bias from injection to distillation in language models." ACM CCS 2025.

W2 (b): Target length evaluation

We acknowledge that we only tested target length with the fixed-single setting, leaving 75% of the length configurations untested. We will clarify this limitation in the main text. Due to limited compute, we prioritized testing more models and bias types. However, the length effect remains consistent: short targets achieve higher ASR than long targets in all 12 task and bias-type configurations, even with independently generated poison sets.

W3 (a): Predictive modeling

We agree that the prediction results show consistency across models, not necessarily the ability to predict results for unseen configurations or tasks. Leave-one-model-out prediction achieves an MAE of 8.9, compared with 17.9 for the baseline, but uses the same configuration evaluated on other models. We will clarify this limitation and narrow the claims in Section 5.

W3 (b): Reliability of factor effects

To test the reliability of configuration effects, we repeated the experiments with independently generated poison sets. Within the same task, configuration rankings from independently generated poison sets show a high correlation of 0.84, while rankings across different tasks have a lower correlation of 0.62. This suggests that the weaker cross-task prediction and the changes in the signs of interaction coefficients across tasks (Table 34) are mainly due to real differences between tasks, rather than random variation in the poison sets. We will revise the claims in Section 5 to make this clear.

W3 (c): Defense baselines

We agree that defense baselines are needed, and thank you for the suggestion. We evaluated defenses before training, during training, after training, and at inference time, and report their detection AUROC, the ASR before and after each defense, and their cost. We find that existing defenses have limitations and tradeoffs: the filters and continued fine-tuning on non-target data do not prevent the attack; the defenses that achieve some mitigation require clean data for the target task; and CleanGen degrades utility by 10–23% and slows generation by 2.4–3.7×.

DefenseStageAUROCASR (%)Cost
Embedding kNN anomaly scoreBefore training0.80–0.83UnchangedRemoves 5% of benign data
High-loss filtering with base-model loss (ICML 2023)Before training0.36–0.42UnchangedRemoves 5% of benign data
AlpaGasus quality filter (ICLR 2024), as used against VPI (NAACL 2024)Before training0.4720.6–86.3 → 0.0–41.3Removes 53.5% of benign data
Continued clean fine-tuning, non-target tasksAfter trainingn/a64.8–81.4 → 31.4–65.4At most 1.5% utility change
Continued clean fine-tuning, with target taskAfter trainingn/a64.8–81.4 → 0.0–0.8n/a
CleanGen (EMNLP 2024)Inferencen/a31.5–86.3 → 0.0–16.210–23% utility lost, 2.4–3.7× slower generation

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.