acceptodds
Under review as a conference paper at ICLR 2027

Defending Against Dormant Poisoning Attacks Across Language and Multimodal Agents

Abstract

Recent work has shown that in dormant poisoning, an attacker implants harmful behaviors that remain hidden at release but emerge later after entirely benign downstream fine-tuning. As foundation models are increasingly used as agents, whether dormant poisoning extends to harmful agent actions and multimodal settings remains unclear. We show two risks of dormant poisoning in agentic settings: (1) activation can extend to tools and actions not optimized during poisoning; and (2) multiple harmful behaviors can coexist within one agent and be jointly activated by fine-tuning. Existing defenses often rely on externally provided safe responses, which are hard to obtain for agent actions, or a trusted clean reference model as a safety anchor. In dormant poisoning, however, the released model appears safe before fine-tuning, so downstream users may be unaware of the poisoning and may not know which model to trust as a clean reference. We address this issue by exploiting a key property of dormant poisoning: the attacker must preserve safe behavior at release to conceal the poisoning, and this preserved behavior can itself serve as a safety reference. Leveraging this property, we automatically construct a Safety Buffer that pairs each harmful input with the released model’s response and a benign version generated by the same model to remove harmful intent. The Safety Buffer can also replace external safe responses required by existing defenses. However, these defenses do not distinguish updates that weaken safety from those that preserve it, often increasing over-refusal or reducing utility. To address this limitation, we lastly introduce LookAhead Defense, which previews each candidate update using the Safety Buffer and penalizes only updates predicted to weaken safe behavior on the harmful input or make the released model’s response more likely on the benign version – allowing us to preserve safety while avoiding unnecessary changes to benign behavior. Across language models, language agents, and multimodal agents, LookAhead Defense reduces harmful behavior in agents from up to 93.9% to at most 14.2% and harmful responses in language models from up to 92.5% to at most 2.0% compared with vanilla fine-tuning, while maintaining competitive utility and low over-refusal. It also remains effective against other poisoning methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety, Defense, Dormant Poisoning Attack, Language and multimodal agentic setup

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8gNBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript investigates the risk that benign fine-tuning activates dormant poisoning behaviors and extends FAB to language agents, multimodal agents, and multi-behavior poisoning. The authors construct the Safety Buffer using the released model’s own responses and benign rewrites, and propose LookAhead Defense, which protects safe behaviors via gradient-sign preview and one-sided loss penalty. Experiments across models, attacks, and tasks demonstrate substantial reduction in harmful behaviors; nevertheless, the safety proxy metrics, ablation controls, and statistical reliability remain to be validated.

Critical strengths

The manuscript covers text, tool calls and visual actions, which carries practical significance. It leverages self-generated references from the released model for defense, reducing the demand for external safety annotations. The performance across multiple models demonstrates the effectiveness of the proposed method.

Critical weaknesses
  1. A reduction in the probability of the safety response yiy_i on harmful inputs does not necessarily indicate degraded safety; the model may simply switch to an alternative safe response. Maintaining the probability of this response also cannot constrain the total probability of all harmful outputs. Reducing the probability of the same yiy_i for benign inputs likewise does not guarantee correct and effective actions. The authors retain references solely based on the formal validity of tool calls or actions, without verifying their safety during construction. In Table 9, the safety rate of multimodal FAB reference responses is only 78%. Therefore, the current objective is more accurately described as protecting specific behavioral references rather than directly identifying and penalizing all “unsafe updates”. The manuscript needs to provide validation of the correspondence between surrogate loss changes and real-world action hazard changes.

  2. “No Preview” is not implemented merely by replacing preview parameters in Equation (5) with current parameters; instead, it adopts mixed training with the Safety Buffer. This modification simultaneously alters previewing, the loss structure, and the way benign pairs are used. Its 44% over-refusal rate cannot independently prove that previewing reduces over-refusal. The existing No ReLU experiment supports the effect of one-sided penalty but cannot substitute for this control. In addition, the ablation of core components is conducted only on one language model and does not directly validate the agentic settings emphasized in the manuscript.

  3. The manuscript presents the changes of poisoned models before and after fine-tuning, yet it does not provide complete controls of “unpoisoned models + identical benign fine-tuning” for language and multimodal agents. Hence, it cannot adequately distinguish the extra activation induced by poisoning from safety degradation inherent to vanilla fine-tuning itself. Meanwhile, not all evaluations achieve near-perfect safety: the activation rate of Qwen3 on ASB reaches 27%, and that of Qwen2.5-VL on Unseen is 36%. This does not negate the poisoning effect but calls for a more careful definition of “hidden at release” and cross-task dormancy.

  4. Glaive uses function-call matching, and multimodal actions are only evaluated according to their respective benchmark criteria. The manuscript does not adequately clarify whether harmful actions are actually executed to cause environmental state changes or judged solely based on outputs. It also fails to specify how multi-step trajectories, invalid actions, early termination, and safe yet task-incomplete behaviors are handled.

Questions for authors

Details can be found in the Critical Weaknesses section.

4 comments

We thank the reviewer for the careful review and helpful suggestions. In response, we have made the following major revisions:

  • Safety surrogate validation: Measured the correlation between Safety Buffer reference loss and poison activation, and evaluated attacker-tool calls on held-out InjecAgent examples (Appendix M, Table 22).
  • Ablation studies: Added a No Preview ablation to isolate the effect of previewing and extended the ablations to language and multimodal agents (Section 5.4, Table 6).
  • Clean model controls: Compared clean and poisoned checkpoints under identical benign fine-tuning (Appendix K, Table 20).
  • Agent evaluation: Clarified the single-step evaluation protocol and analyzed attacker-tool calls and legitimate next-tool calls (Appendix N, Table 23).

Our detailed responses are provided below.


[W1. Does the safety surrogate reflect actual harmful behavior?]

Thank you for raising this point. LookAhead monitors changes in the released model's behavior to detect potential safety degradation. To validate this, we evaluated attacker-tool calls on 300 held-out InjecAgent examples.

Table1. Attacker-tool calls (%). Estimated: predicted probability of calling the attacker's tool. Observed: actual call rate across 10 generated responses per example.

MethodEstimated ↓Observed ↓
Before FT0.10.0
Vanilla FT95.495.7
SafeDelta95.195.4
Safety-mix1.81.5
Lisa0.70.4
AsFT1.82.2
LookAhead0.40.4

LookAhead reduces the estimated attacker-tool call rate from 95.4% to 0.4%, with the same 0.4% observed call rate. This means that LookAhead effectively suppresses harmful tool calls, not just changes in reference likelihood.

We also measured the Safety Buffer reference loss and poison activation at different checkpoints during fine-tuning. The Spearman correlations between them were 0.65, 0.82, and 0.90 for language models, language agents, and multimodal agents, respectively. These results show that when the model becomes less likely to produce its original safe responses, it tends to produce more harmful responses or actions. This supports the use of the Safety Buffer reference loss as a signal of safety degradation.

We have added these analyses in Appendix M of the revised version (Table 22, lines 1242–1270).


[W2. No Preview and agent ablations]

Thank you for pointing this out. We have renamed it Buffer Mixing (Table 8) and added a new No Preview ablation that isolates the effect of previewing (Table 6).

We conducted a new No Preview ablation that removes only the preview while keeping all other components unchanged. We also extended the ablations to language agents and multimodal agents.

Compared with No Preview, LookAhead reduces over-refusal on language models (10.2% → 2.0%) and harmful activation on agents, including InjecAgent (6.0% → 0.0%) and RiOSWorld (21.0% → 6.4%). We also observed that InjecAgent activation after the first fine-tuning step was 78.7% without preview versus 0.0% with LookAhead, suggesting that previewing helps prevent early activation.

Table2-a. Language model — Qwen3-4B-Instruct

VariantASR ↓OR ↓Avg. ↓
Vanilla FT61.50.831.2
No Preview1.510.25.9
No ReLU0.00.00.0
No benign version0.025.612.8
LookAhead1.02.01.5

Table2-b. Language agent — Qwen3-4B-Instruct

VariantInjecAgent ↓ASB ↓Avg. activation ↓Avg. utility ↑
Vanilla FT98.789.093.985.9
No Preview6.010.08.084.0
No ReLU0.30.00.251.6
No benign version0.027.013.581.0
LookAhead0.07.03.585.5

Table2-c. Multimodal agent — Qwen2.5-VL-3B-Instruct

VariantRiOSWorld ↓Unseen ↓Avg. activation ↓Avg. utility ↑
Vanilla FT81.679.080.370.0
No Preview21.035.028.068.8
No ReLU2.49.05.760.8
No benign version16.037.026.569.2
LookAhead6.422.014.268.9

We have updated Appendix B (Table 8, lines 873–910) and added the isolated No Preview and agent ablations in Section 5.4 (Ablation and Analysis, lines 512–520) and Table 6 (lines 501–510).

[W3. Is the post-FT activation caused by poisoning or benign fine-tuning itself?]

Thank you for the question. We ran the same benign fine-tuning on the corresponding clean checkpoints.

SettingEvaluationUnpoisonedPoisoned
Language agentInjecAgent0.0→0.01.0→98.7
ASB10.0→8.027.0→89.0
Multimodal agentRiOSWorld0.0→0.010.5→81.6
Unseen0.0→0.036.0→79.0

Values are activation rates (%) before → after benign fine-tuning.

The clean checkpoints do not show increased activation after the same fine-tuning, whereas the poisoned models do. Following the original FAB setting [5], "dormant" refers to targeted harmful behaviors that remain hidden at release and become activated after benign fine-tuning. ASB and Unseen separately evaluate whether activation transfers to other tools and risk categories. We have added these controls and clarified the terminology in Appendix K (Table 20, lines 1188–1212).

Reference

[5] Watch Your Steps: Dormant Adversarial Behaviors That Activate Upon LLM Finetuning. ICLR 2026.


[W4. How are agent actions and task progress evaluated?]

Thank you for pointing this out. Our evaluation directly assesses the tool calls and actions generated by the model, using single-step outputs. Invalid outputs are retained in the evaluation and excluded from harmful activation. We separately evaluate utility on Glaive and BFCL for language agents, and OS-Genesis and AITW for multimodal agents, using their respective evaluation criteria.

To further examine whether LookAhead suppresses harmful tool calls while preserving legitimate task actions, we analyzed 100 ASB examples.

MethodAttacker tool ↓Next tool for the user's task ↑Previous tool repeatedOther or invalid
Before FT2727460
Vanilla FT891100
SafeDelta901000
Safety-mix1627570
Lisa1129600
AsFT5122270
LookAhead745471

Our additional analysis of 100 ASB examples shows that LookAhead achieves the lowest attacker-tool call rate (7%) and the highest legitimate next-tool call rate (45%) among all compared methods. This shows that LookAhead reduces harmful tool calls while improving the selection of legitimate next-step actions, rather than simply avoiding tool use.

We have clarified the evaluation protocol and added this analysis in Appendix N (Table 23, lines 1274–1295).

Dear Reviewer 8gNB,

We sincerely appreciate your careful review and constructive feedback. As we have about three days left in the discussion period, we would like to kindly follow up on whether our rebuttal, including safety surrogate, No Preview and agent ablations, poisoned vs. unpoisoned model comparisons, and additional agent evaluation analyses, has addressed your main concerns. We would be happy to further discuss any remaining concerns or provide additional clarification.

Thank you again for your time and consideration.

Dear Reviewer 8gNB,

As the discussion period ends today, we would like to kindly follow up and ask if you have had a chance to review our rebuttal. We would appreciate any feedback on whether your comments have been addressed.

Thank you for your time and consideration!

Reviewer_oYawOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper discusses dormant attacks in LLMs, which get activated when a released model is finetuned further, while the original released model does not show any harmful behavior. Majorly, the authors are interested in such attacks with multimodal systems (GUI models to some extent) and agents, where the exact safe responses are hard to find. Hence, they first build an "input-constrastive" dataset, Safety-Buffer, that pairs a harmful input (xix_i), to its intended harmful output (yiy_i), and also a benign input (x~i\tilde{x}_i) in which case the harmful output is the expected natural output from the model. Unlike PO methods, which require harmful and safe outputs for a given input, this flips the setup and has contrastive inputs for an output, which is now "conditionally" harmful based on the input. These triplets are then used for a LookAhead-Defence, which is similar to perturbation-aware learning. The major contribution the authors discuss in the paper is multiple harmful behaviors being packed into a single model, which they claim has not been explored before, and for which they propose a simple loss aggregation method across all harmful traits to be included. Lastly, across multiple models, benchmarks, and baselines, the authors have shown the robustness of their technique.

Critical strengths
  • I appreciate the problem statement discussed in this paper. It's super relevant in the age of open-source models, which can look harmless upon release, but are activated upon finetuning.
  • The idea to pack and work with multiple harmful behaviors in a single model is also interesting, and the formulation provided for it is also very clean and intuitive. (Eq 2).
  • The paper is well-written and provides coherent and cogent arguments for its hypothesis. It's good to see multiple models, modalities, benchmarks, and attack/poisoning strategies in their analysis. It's quite comprehensive.
Critical weaknesses
  • I don't see any glaring technical errors in the paper, but isn't the LookAhead-Defence (and the triplet construction for it) an established perturbation-aware training protocol for LLMs? For example, I am curious how the LookAhead-Defence is different from the FPL discussed in Xue et al.? Further, from my experience with such perturbation-aware training, I find it quite unstable and also computationally intensive since it kind of requires two gradient computation steps (one for Eq 5.1, and the other to learn from Eq 6), so it might be easy for larger models.
  • I am also concerned about the 50 examples used for the Safety-Buffer? Isn't it too little to tune the model to learn?
Questions for authors
  • How did you determine the hyperparameters for Eq 2?
  • Another question I had: For a model trained with FAB, the experiments in the paper show unlocking the harmful behavior via 3 datasets (GSM8K, Glaive, OS-Genesis). So, is it specific to those 3, or can they be unlocked by finetuning with any dataset?
5 comments

We thank the reviewer for the helpful feedback. In response, we have made the following major revisions:

  • Method comparison and computational cost: Clarified the difference between LookAhead Defense and FPL and reported training overhead for 4B and 8B models (Section 2 and Appendix H, Table 17).
  • Safety Buffer size: Evaluated different Safety Buffer sizes across all three settings (Appendix L, Table 21).
  • Hyperparameter sensitivity: Evaluated Agentic FAB under different hyperparameter settings (Appendix O, Table 24).
  • Additional fine-tuning datasets: Evaluated poison activation and the effectiveness of LookAhead Defense under separate fine-tuning on CodeAlpaca and PubMedQA (Appendix E, Table 12).

Our detailed responses are provided below.


[W1. Relation to perturbation-aware training, stability, and computational cost]

Thank you for these comments.

Difference from FPL. While both methods evaluate a loss after a simulated weight update, FPL is applied before model release to make watermarks robust to weight changes that could erase them, whereas LookAhead operates during downstream fine-tuning, previews each candidate update using the task gradient, and applies a safety penalty only when the update is predicted to weaken the released model’s own safe behavior.

Stability and computational cost. LookAhead reuses the task gradient for its preview without requiring second-order derivatives. We also clip the safety gradient relative to the task gradient to control the updates. To examine computational cost at different model scales, we measured training overhead on 4B and 8B models.

Table1. Computational overhead and defense performance.

ModelMethodTime (× Vanilla)Safety ↓Utility ↑
4BVanilla FT1.00×31.272.5
SafeDelta1.00×7.559.1
Safety-mix1.12×22.073.8
Lisa0.96×5.378.5
AsFT1.03×27.077.6
LookAhead1.11×1.574.1
8BVanilla FT1.00×90.273.5
SafeDelta1.00×69.773.5
Safety-mix1.80×13.472.4
Lisa0.98×10.062.5
AsFT1.10×38.472.5
LookAhead1.53×13.075.9

The 4B results use Qwen3-4B-Instruct, with safety measured by average ASR and over-refusal and utility by GSM8K. The 8B results use Granite-4.2-8B, with average activation on InjecAgent/ASB and average utility on Glaive/BFCL.

(SafeDelta also requires one-time post-processing of 22 minutes for 4B and 58 minutes for 8B.)

On 4B, LookAhead achieves the lowest average of ASR and over-refusal with only 11% additional training time. On 8B, it is faster than Safety-mix and achieves the highest average utility while maintaining low harmful activation. It requires neither an auxiliary reference model nor externally provided safety responses. We have clarified the comparison in Section 2 (Related Work, lines 147–156) and added the gradient-clipping details in Appendix A (line 809) and the computational results in Appendix H (Table 17, lines 1111–1133).


[W2. Are 50 Safety Buffer examples sufficient?]

Thank you for this comment. To evaluate whether 50 Safety Buffer examples are sufficient, we compare buffer sizes of 25, 50, and 100 across all three settings.

Table2. Effect of Safety Buffer size

Buffer sizeLanguage model ASR / OR ↓Language agent Avg. activation ↓Multimodal agent Avg. activation ↓
0 (Vanilla FT)61.5 / 0.893.980.3
255.0 / 1.64.520.9
50 (Ours)1.0 / 2.03.514.2
1000.0 / 1.24.720.3

The results show that even 25 examples substantially reduce harmful behavior across all three settings. The Safety Buffer does not teach new safety behavior but instead monitors whether fine-tuning weakens the model's existing behavior. This allows a small number of examples to provide a useful signal for preventing harmful updates. We use 50 examples as a practical balance between safety and buffer size, with comparable results at 100 examples. We have added this study in Appendix L (Table 21, lines 1217–1241).

[Q1. Hyperparameters]

Thank you for the question. We selected the hyperparameters to keep harmful behavior dormant before fine-tuning while allowing strong activation afterward.

Table3. Hyperparameters on Qwen3-4B-Instruct (InjecAgent / ASB, %)

Settingλ_regλ_m−lλ_noiseBefore FT ↓After Vanilla FT ↑
Ours2.51.00.31.0 / 27.098.7 / 89.0
2.01.00.30.0 / 29.099.7 / 87.0
2.50.50.30.0 / 11.099.3 / 81.0
2.52.00.30.0 / 28.099.7 / 81.0
2.51.00.10.0 / 26.099.3 / 88.0

Agentic FAB remains largely dormant before fine-tuning and strongly activated afterward across the tested settings, indicating that it is not highly sensitive to these hyperparameters. We use λ_reg = 2.5, λ_m−l = 1.0, and λ_noise = 0.3 in our main experiments. We have added these results in Appendix O (Table 24, lines 1296–1315).


[Q2. Does activation occur with other fine-tuning datasets?]

Thank you for the question. Appendix E (Table 11) of the original paper already shows continued activation under sequential fine-tuning on GSM8K, PubMedQA, and CodeAlpaca. We also evaluated separate fine-tuning on CodeAlpaca and PubMedQA.

FT datasetVanilla FT ASR ↓LookAhead ASR ↓
GSM8K61.51.0
CodeAlpaca71.02.0
PubMedQA64.50.0

FAB activation is not limited to GSM8K, and LookAhead continues to suppress it on both additional datasets. We have added the separate fine-tuning results to Appendix E of the revised version (Table 12, lines 1001–1011).

Dear Reviewer oYaw,

We sincerely appreciate your insightful review and helpful suggestions. As we have about three days left in the discussion period, we would like to kindly follow up on whether our rebuttal, including the comparison with FPL, computational overhead, Safety Buffer size ablations, hyperparameter sensitivity, and additional fine-tuning datasets, has addressed your main concerns. We would be happy to further discuss any remaining concerns or provide additional clarification.

Thank you again for your time and consideration.

Reviewer_Gsq1OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a defensive method, called LOOKAHEAD DEFENSE, for language models against dormant poisoning attacks. The attacks considered in this study cause the model to behave unsafely only after benign fine-tuning. The core idea is to create a buffer of clean samples on which the model behaves safely. The authors then use this buffer as a reference for preserving safety.

Critical strengths

1- Important Problem: Given the widespread use of language models and the challenges of detecting and mitigating dormant poisoning attacks, I believe the paper addresses an important security problem. 2- Experimental evaluation: The authors evaluate multiple language models, agents, and datasets. They also report ablation studies, which is valuable.

Critical weaknesses

Randomness and repeated experiments: I did not find an explicit report on the number of training runs, random seeds, or standard deviations of the results. LLM inference can be stochastic, so a model may produce different results for the same input under the same conditions across runs. I recommend repeating the model calls multiple times and reporting the average and standard deviations for a more reliable evaluation. 2- Unclear novelty: The paper presents SAFETY BUFFER and LOOKAHEAD DEFENSE as its main contributions, but more differentiation is needed. The authors cite enough related papers, but they should clarify which components of their proposed method are new and which are derived from the literature. 3- Unclear threat model: To what extent does the attacker know about the deployed system? Does the attacker know that LOOKAHEAD DEFENSE will be used? Does the attacker optimize against the proposed defense? It is generally preferred for a proposed defense to be evaluated against an adaptive white-box adversary. At minimum, the authors should report whether the proposed method is effective only against non-adaptive attackers. 4- Question about technical soundness: On page 2, the authors state: “Since the attacker must keep the released model behaving safely, we exploit this key property and use the released model’s own safe behavior as a safety reference.” The model may be already showing unsafe behavior against specific inputs in a non-adversarial setting. It seems the defender focuses only on samples where the released model behaves safely before fine-tuning. This may overlook the very samples that the released model is already vulnerable to before fine-tuning. I am concerned this approach might create blind spots that an attacker can later exploit, since these inputs do not seem to be represented in the buffer.

Questions for authors

Q1: Could you please rerun the experiments with different random seeds and report the number of runs, along with the mean and standard deviation of the results? Q2: Please clarify which components of your defense are fundamentally new compared to existing fine-tuning methods. Q3: Is the attacker unaware of the defense? If yes, can you evaluate the defense against an adaptive adversary that optimizes against it? Q4: Have you considered a scenario where the attacker targets samples the model misclassifies before fine-tuning? I suggest considering this, or at least stating the defense's limitations in these scenarios.

4 comments

We thank the reviewer for the thoughtful comments and constructive feedback. In response, we have made the following major revisions:

  • Repeated experiments: Reported the mean and standard deviation across three downstream fine-tuning runs with different seeds in all three settings (Appendix I, Table 18).
  • Novelty: Clarified our main contributions: Agentic FAB, Safety Buffer, and LookAhead Defense (Section 1).
  • Adaptive attacks: Evaluated an adaptive attacker that optimizes against LookAhead Defense and examined the effect of expanding the Safety Buffer (Appendix J, Table 19).

Our detailed responses are provided below.


[W1 & Q1. Randomness and repeated experiments]

Thank you for the suggestion. We repeated downstream fine-tuning with three random seeds for Qwen3-4B-Instruct (language model and language agent) and Qwen2.5-VL-3B-Instruct (multimodal agent), keeping the released model and Safety Buffer fixed.

MethodLM ASR ↓LM OR ↓Language agent Activation ↓Utility ↑Multimodal agent Activation ↓Utility ↑
Vanilla FT60.7 ± 6.31.2 ± 1.196.2 ± 3.284.1±1.6‾\underline{84.1 \pm 1.6}72.8 ± 7.868.4 ± 1.7
Safety-mix0.2 ± 0.337.1 ± 6.08.0 ± 1.583.7 ± 0.635.3±6.3‾\underline{35.3 \pm 6.3}69.0 ± 1.3
Lisa1.7 ± 2.58.7 ± 2.35.5±0.0‾\underline{5.5 \pm 0.0}83.2 ± 1.236.8 ± 9.568.9±0.6‾\underline{68.9 \pm 0.6}
LookAhead0.3±0.6‾\underline{0.3 \pm 0.6}6.1±4.2‾\underline{6.1 \pm 4.2}4.5 ± 2.684.5 ± 1.319.4 ± 6.769.0 ± 1.3

These results are based on repeated downstream fine-tuning with three random seeds. Across all three settings, LookAhead consistently reduces harmful behavior compared with vanilla fine-tuning while maintaining competitive utility. We have added these results in Appendix I (Table 18, lines 1139–1155).


[W2 & Q2. Unclear novelty]

Thank you for the question. We clarify what is new in our work and provide supporting evidence below.

ContributionWhat is new?Supporting evidence
Agentic FABExtends prior FAB [1] from harmful text responses to agent actions, including unseen tools, risk categories, and multiple harmful behaviors.Harmful activation increases after fine-tuning, even on unseen tools and risk categories (Table 2).
Safety BufferOur safety buffer does not require externally sourced safe responses [2-4] or a trusted clean model [5], instead using the poisoned model's own responses.Our Safety Buffer reduces over-refusal compared with external safety responses (10.0% → 2.0%, Table 3b).
LookAhead DefenseIntroduces a selective safety intervention that previews each fine-tuning update and penalizes it only when predicted to weaken existing safe behavior, unlike prior defenses that apply safety constraints throughout fine-tuning [2,3,5] or adjust weights afterward [4].Ablations show that previewing reduces harmful activation (InjecAgent: 6.0% → 0.0%; RiOSWorld: 21.0% → 6.4%), while one-sided penalties help preserve utility (language agents: 51.6% → 85.5%; multimodal agents: 60.8% → 68.9%) (Table 6).

Our main contributions are showing that dormant poisoning can affect agent actions and introducing a defense that uses the released model's own behavior to preserve safety during downstream fine-tuning.

We have added a summary of these contributions in Section 1 (Introduction, lines 125–133).

References

[1] Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning. ICLR 2026

[2] Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models That Follow Instructions. ICLR 2024.

[3] Lisa: Lazy Safety Alignment for Large Language Models Against Harmful Fine-Tuning Attack. NeurIPS 2024.

[4] Safe Delta: Consistently Preserving Safety When Fine-Tuning LLMs on Diverse Datasets. ICML 2025.

[5] AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basin. AAAI 2026.

[W3 & Q3. Threat model and adaptive attackers]

Thank you for the question. We evaluated an adaptive attacker that knows LookAhead and how the Safety Buffer is constructed. The attacker optimizes its poisoned model by simulating LookAhead fine-tuning, including the sign-gradient preview and safety penalty.

We further investigated whether constructing the Safety Buffer from multiple datasets improves robustness against adaptive attacks.

Table1. Adaptive attack on Qwen3-4B-Instruct language agent. InjecAgent and ASB are evaluated on held-out data.

SettingInjecAgent ↓ASB ↓Glaive ↑BFCL ↑
Before FT1.026.0––
Vanilla FT (poisoned)95.797.076.993.0
LookAhead64.753.072.193.0
LookAhead (AgentDojo + ASB)33.726.074.094.0
LookAhead (AgentDojo + InjecAgent)0.026.073.193.0

The adaptive attack achieves 64.7% harmful activation on InjecAgent despite LookAhead, demonstrating that the attacker successfully adapts to the defense. However, expanding the Safety Buffer with examples from additional datasets substantially improves robustness against adaptive attacks, without changing the defense algorithm. We have added this experiment in Appendix J (Table 19, lines 1164–1187).


[W4 & Q4. Inputs already unsafe before fine-tuning]

Thank you for this comment. Our method does not assume that the released model is safe on every input. When the released model already produces unsafe responses, we prompt it to generate a refusal and use that response as the safety reference. We measure its likelihood under the original harmful input and track changes throughout fine-tuning.

We evaluate this setting in Table 5 of the original paper, where LookAhead reduces ASR on already-unsafe models, including Sleeper (63.0% → 1.0%) and Benign DPO (66.5% → 10.0%) on Qwen2.5-3B. These results show that LookAhead can improve safety even when harmful behavior is already present before fine-tuning.

We also evaluate robustness to Safety Buffer contamination in Appendix C (Table 9) of the original paper. The results show that LookAhead remains effective even when some reference responses are unsafe, demonstrating robustness to an imperfect Safety Buffer.

Dear Reviewer Gsq1,

We sincerely appreciate your thoughtful review and valuable feedback. As we have about three days left in the discussion period, we would like to kindly follow up on whether our rebuttal, including repeated experiments with three random seeds, novelty clarification, adaptive attack, and analyses of already unsafe models, has addressed your main concerns. We would be happy to further discuss any remaining concerns or provide additional clarification.

Thank you again for your time and consideration.

Dear Reviewer Gsq1,

As the discussion period ends today, we would like to kindly follow up and ask if you have had a chance to review our rebuttal. We would appreciate any feedback on whether your comments have been addressed.

Thank you for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.