acceptodds
Under review as a conference paper at ICLR 2027

Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks

11 views

Abstract

Malicious finetuning attacks pose a major safety threat against open-weight large language models (LLMs). However, existing alignment-stage defenses provide limited protection against strong attacks that use full-parameter finetuning. To address this issue, we propose Patcher, a novel and efficient adversarial training algorithm that alternates between an attack stage and a defense stage: the attack stage simulates many-step malicious finetuning and computes the resulting parameter displacement as an attack vector, while the defense stage aims to preserve safe behavior under such perturbations. Compared with other adversarial training methods, a key feature of Patcher is that the attack vector can be reused across multiple defense updates, substantially reducing training costs. Theoretically, we characterize how the relative gradient estimation error is influenced by the number of attack steps during training and reuse-duration of attack vectors. Empirically, experiment results show that Patcher decreases Attack Success Rate by 67.5%, 53.4% and 54.8% on three benchmarks compared to the second-best method while preserving the model's utility. Moreover, Patcher remains effective under diverse model sizes and attack scenarios, and generalizes to prompt-based jailbreak attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, malicious finetuning, alignment, robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5eapOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PATCHER, an adversarial training method for defending against malicious full-parameter finetuning. It simulates multi-step attacks and reuses the resulting parameter displacements across multiple defense updates to reduce training cost. The paper provides theoretical bounds on gradient approximation error and evaluates robustness, downstream utility, and computational efficiency.

Critical strengths
  1. The method provides a simple way to scale train-time attacks. Reusing attack vectors amortizes the cost of simulating longer attacks, and the parallel implementation further reduces wall-clock time.

  2. The empirical gains under the default attack are substantial. Table 1 shows clear ASR reductions across all three benchmarks, supported by results over five random seeds. Additional evaluations cover larger models, downstream utility, and several adaptive attack settings.

Critical weaknesses
  1. Method design: the justification for long attack-vector reuse is limited. Proposition 2 assumes gradient descent with sufficiently small learning rates and matched train/test attack lengths, whereas the experiments use AdamW with 300 training attack steps and 2000 test attack steps. The bound therefore does not directly justify the default 1000-step reuse window or provide a practical criterion for refreshing the vector.

  2. Related work: the comparison with TAR does not isolate the claimed advance. TAR already uses multi-step adversarial meta-learning, but the reported comparison also changes the defense objective and update schedule (Appendix D). This makes it difficult to separate the benefit of attack-vector reuse from these other differences relative to the closest prior method.

  3. Experiments and analysis: the robustness gains are sensitive to basic attack changes. A test-time learning rate of 5e-5 raises the average ASR to 64.4% (Table 9), while DPO with β = 0.5 followed by SFT yields 42.1% ASR versus 40.7% for vanilla SFT (Table 10). Although these limitations are acknowledged, they substantially weaken the broader claim of robustness against malicious full-parameter finetuning.

Questions for authors
  1. Validity of the reuse window. In the actual AdamW setting, how does the defense gradient computed with a reused attack vector deviate from that computed with a fresh vector over the 1000-step window? What evidence supports this refresh interval?

  2. Contribution relative to TAR. Can the authors compare against TAR-style training using the same defense objective and matched compute, with and without attack-vector reuse? This would help identify the improvement attributable to the proposed mechanism.

  3. Robustness under attack selection. Does PATCHER retain a substantial advantage when the attack learning rate and loss are tuned separately for each defense under a fixed attack budget? Please also report the corresponding utility to clarify the practical significance of the remaining robustness.

4 comments
AuthorsOpenReview ↗

Author Response (1/3)

We thank the reviewer for the constructive feedback on our manuscript. We will clarify important details and provide additional experimental evidence to address your concerns.

1. Validity of the reuse window.

The justification for long attack-vector reuse is limited... The bound therefore does not directly justify the default 1000-step reuse window or provide a practical criterion for refreshing the vector.

In the actual AdamW setting, how does the defense gradient computed with a reused attack vector deviate from that computed with a fresh vector over the 1000-step window? What evidence supports this refresh interval?

We agree that Proposition 2 does not explicitly justify a 1000-step reuse window in the AdamW setting. To provide empirical evidence for the propositions, we measured the relative gradient estimation error across different attack lengths and reuse durations, and the results are shown in Section 1 of the general Q&A. We find that this error remains relatively stable over the 1000-step reuse window, providing empirical support for the reuse strategy.

In addition, the end-to-end ablation in Section 4.5 of our manuscript provides complementary evidence, where we evaluated the ASR when the reuse duration was set to 200, 500 and 1000 respectively, and found that ASR increased mildly as we enlarged the reuse window from 200 to 1000.

Together, the relative gradient estimation error and the empirical ASR evaluation support our selection of the 1000-step refresh interval as a good trade-off between robustness and computational efficiency. Nevertheless, we agree that Proposition 2 relies on simplifying assumptions. It provides theoretical motivation for attack-vector reuse, but not an optimizer-specific guarantee or a practical rule for selecting the refresh interval. We have revised Section 3.2 of the manuscript to distinguish the theoretical motivation from the empirical basis for selecting the refresh interval.

AuthorsOpenReview ↗

Author Response (2/3)

2. Contribution relative to TAR.

...TAR already uses multi-step adversarial meta-learning, but the reported comparison also changes the defense objective and update schedule (Appendix D). This makes it difficult to separate the benefit of attack-vector reuse from these other differences relative to the closest prior method.

Can the authors compare against TAR-style training using the same defense objective and matched compute, with and without attack-vector reuse? This would help identify the improvement attributable to the proposed mechanism.

Thank you for this insightful analysis. We acknowledge that TAR already simulates multi-step attack trajectories. However, compared to TAR, our work explicitly studies the effect of increasing the number of simulated attacks and shows its importance for robustness in Figure 5aa.

Furthermore, another significant difference between Patcher and TAR is the reuse strategy of the attack trajectory: Patcher reuses an attack vector for a certain number of steps, while TAR constructs a new attack trajectory for each defender update. We consider attack vector reuse to be the main contribution of Patcher, which significantly improves computational efficiency and makes multi-step attack simulation affordable in adversarial training.

To address your concern, Section 2 of the general Q&A compares training with and without attack-vector reuse using the same defense objective as Patcher, where the no-reuse variant constructs a fresh attack vector before each defender update, following TAR’s refresh schedule. The experiment yields two findings:

  • With the total number of defender updates held fixed, increasing the reuse duration increases ASR. This shows the robustness cost associated with reusing an increasingly stale attack vector.

  • With the training compute budget held fixed, using a 1000-step reuse window reduces ASR from 85.8% to 20.9% compared with refreshing the vector at every defender update.

These results suggest that the benefit of reuse is improved robustness within a fixed training budget, despite a robustness cost at a fixed number of defender updates. In Patcher's setting, the computational savings outweigh the cost of attack vector staleness. Therefore, increasing the number of attack steps and reusing attack vectors are two complementary and necessary components that contribute to Patcher's robustness.

Following your suggestion, we have added this new ablation to Section 4.5 to decouple the benefits of scaling the number of attack steps and reusing attack vectors.

AuthorsOpenReview ↗

Author Response (3/3)

3. Robustness under attack selection.

The robustness gains are sensitive to basic attack changes... Although these limitations are acknowledged, they substantially weaken the broader claim of robustness against malicious full-parameter finetuning.

Does PATCHER retain a substantial advantage when the attack learning rate and loss are tuned separately for each defense under a fixed attack budget? Please also report the corresponding utility to clarify the practical significance of the remaining robustness.

Thank you for this question. First, we would like to emphasize that Patcher is designed primarily to improve the robustness against full-parameter malicious finetuning with the common cross-entropy objective (i.e., SFT), explicitly formulated in Section 3 and simulated during training. Under this objective, Patcher consistently decomstrates considerable improvements against other baseline methods under different datasets, models, poison ratios, number of finetuning steps, learning rates and optimizers. We view Patcher's effectiveness under this finetuning objective as the main scope of this work, and we have further revised the abstract, introduction and conclusion to explicitly clarify this.

However, we agree that tuning the attack and loss separately and reporting the best-performing attack strategy for each defense is also an important evaluation. As per your request, we evaluated different defense-specific attack configurations under a 2000-step attack budget. Below, we report the selected configuration (i.e., the attack configuration achieving the highest ASR) and results for each alignment method:

Alignment methodDefense-specific attackAdvbenchBeavertailsHEx-PHIOverallGSM8K utility
SFTGradient ascent (500 steps) with learning rate 1e-5 + SFT with learning rate 1e-487.682.474.581.54.3
PatcherDPO (β=0.1\beta=0.1, 1000 steps) with learning rate 1e-5 + SFT with learning rate 5e-582.080.8‾\underline{80.8}79.580.8‾\underline{80.8}7.5
TARSFT, learning rate 5e-595.685.889.890.48.8‾\underline{8.8}
BoosterDPO (β=0.1\beta=0.1, 1000 steps) with learning rate 1e-5 + SFT with learning rate 5e-591.680.8‾\underline{80.8}91.087.89.0
VaccineDPO (β=0.1\beta=0.1, 1000 steps) with learning rate 1e-5 + SFT with learning rate 5e-584.4‾\underline{84.4}76.075.0‾\underline{75.0}78.58.2

These results show that Patcher does not retain a substantial advantage over SFT under this defense-specific attack selection: its overall ASR is 80.8%, compared with 81.5% for SFT. We acknowledge that the robustness gains observed under our default attack settings do not extend uniformly to this broader adaptive setting, and we have added a paragraph in Section 4.2 of the revised manuscript to explicitly discuss Patcher's robustness against adaptive attacks.

However, we would like to direct your attention to a separate experiment reported in our response to Reviewer Nv7x’s third question, which investigates whether incorporating additional attack objectives during defense training can improve robustness. For the evaluated DPO (β=0.5\beta=0.5) + SFT attack, incorporating attack vectors constructed with diverse objectives reduces Patcher’s ASR from 42.1% to 12.9%. This suggests that broadening the training-time attack distribution is a promising direction to further enhance robustness against a broader range of finetuning attacks.

In addition, we find that the selected attacks for Patcher also result in poor downstream utility: 7.5% accuracy on GSM8K, compared with 29.6% under the default attack setting used in the main experiments. This indicates that the attacker also needs to consider the trade-off between utility and harmfulness when selecting the attack configurations. Specifically, if the attacker still aims for high utility, a lower learning rate for the attack is more desirable, and Patcher remains an attractive defense strategy under this attack scenario.

Thank you again for your analysis that helps us to strengthen our work! We would be happy to address any further questions.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer 5eap,

As the discussion period is coming to an end, we would appreciate your feedback on whether our responses have adequately addressed your concerns. We have provided detailed replies to each point and would be happy to clarify any remaining questions before the deadline.

As a side-note, regarding Question 1, we have provided additional analysis by computing the relative gradient estimation error between the proxy gradient and the exact full gradient at the endpoint of a 20-step attack trajectory, and the results are presented in Appendix A.2 of the revised manuscript. We would appreciate it if you could also check it out.

Many thanks for your time and consideration!

Best regards,

Authors of Submission 25939

Reviewer_Nv7xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies defenses against malicious fine-tuning attacks on open-weight LLMs, particularly attacks that use full-parameter fine-tuning. The authors propose PATCHER, an adversarial training algorithm that alternates between an attack stage and a defense stage. The attack stage simulates many-step malicious fine-tuning and computes the resulting parameter displacement as an attack vector, while the defense stage aims to preserve safe behavior under such perturbations. A key feature of PATCHER is that the attack vector can be reused across multiple defense updates, substantially reducing computational overhead. The authors theoretically characterize how the relative gradient estimation error is influenced by the number of attack steps and the reuse duration of attack vectors. Experiments across multiple safety benchmarks, model sizes, and attack scenarios show that PATCHER improves robustness against malicious fine-tuning while preserving downstream utility.

Critical strengths
  1. The paper addresses an important and challenging problem. Existing alignment-stage defenses provide limited protection against full-parameter malicious fine-tuning. PATCHER offers an intuitive approach by simulating stronger attacks during alignment, while reusing attack vectors to reduce computational overhead.
  2. The experiments demonstrate substantial improvements under the main evaluation settings. PATCHER achieves significantly lower attack success rates than the compared defenses across multiple safety benchmarks. The paper also includes evaluations on different model families, attack datasets, poison ratios, fine-tuning budgets, and prompt-based jailbreak attacks, together with ablation studies and computational efficiency measurements.
Critical weaknesses
  1. The baseline comparisons require further justification. Several existing defenses, particularly TAR, perform substantially worse than vanilla SFT, despite reporting stronger results in their original papers. Although the authors tune some baselines, differences in training configurations and evaluation protocols remain insufficiently explained, raising concerns about the fairness of the comparisons.
  2. The robustness of PATCHER is sensitive to the attack configuration. The defense deteriorates substantially under changes in learning rate, optimizer, and attack objective. Furthermore, its ASR increases significantly with higher harmful data ratios and longer attack trajectories, suggesting limited robustness against stronger or adaptive malicious fine-tuning.
Questions for authors
  1. Theoretical validation: Could the authors empirically measure the relative gradient estimation errors discussed in Propositions 1 and 2? How well do these errors correlate with the observed attack success rates?
  2. Attack-vector reuse: How much of PATCHER's improvement comes from longer simulated attacks versus attack-vector reuse? Could the authors provide a more controlled comparison that isolates these two components?
  3. Generalization across attacks: Would generating attack vectors using multiple optimizers, learning rates, or attack objectives improve robustness against unseen fine-tuning configurations?
  4. LoRA results: Why does PATCHER provide relatively limited improvements under some LoRA fine-tuning configurations?
  5. Safety–utility trade-off: Does increasing resistance to malicious parameter updates also restrict adaptation to benign downstream tasks?
11 comments
AuthorsOpenReview ↗

Author Response (1/7)

We thank the reviewer for the insightful feedback. We have added experiments on gradient discrepancy, attack-vector reuse, and robustness under alternative attack configurations. Below, we address the baseline comparison, clarify the contributions of the two components of Patcher, and discuss the new results alongside downstream utility.

1. Baseline comparisons (W1).

The baseline comparisons require further justification... differences in training configurations and evaluation protocols remain insufficiently explained, raising concerns about the fairness of the comparisons.

Thank you for highlighting the need to clarify the comparison. In Section 4.1 and Appendix D, we have listed the detailed training and evaluation configurations. Here, we summarize the training configurations and evaluation protocols that we controlled to ensure a fair comparison.

- Initialization and training data. All methods in the main comparison start from Qwen2.5-1.5B-Base fine-tuned on Alpaca, controlling for pre-existing safety alignment. SFT, Vaccine, and Booster use the same alignment dataset as Patcher's defender; Booster also uses the same harmful training dataset as Patcher's attacker. TAR also uses the shared alignment data, with its safe and harmful responses forming the chosen and rejected pairs required by its training objective.

- Training configurations. In the main Qwen2.5-1.5B experiments, SFT, Vaccine, Booster, and Patcher all use 15000 defender updates. For TAR, which constructs fresh multi-step attack trajectories for each outer-loop update, we selected 250 outer-loop steps to keep its total training cost comparable to that of Patcher. The baselines use the same global batch size of 4 and AdamW optimizer configurations as Patcher, as detailed in Appendix D.

- Attack and evaluation protocol. For each test setting, all defenses face the same malicious fine-tuning configuration. The default attack uses 200 harmful BeaverTails samples mixed with 800 GSM8K samples, 2000 finetuning steps, learning rate 1e-5, and global batch size 4. We fix the evaluation prompts across methods on Advbench, Beavertails, and HEx-PHI, and use the same response-generation settings, LLM judge, and harmfulness threshold.

To summarize, these controls provide a common basis for comparing the defenses while preserving their method-specific objectives and training procedures.

Regarding the baseline results, an important factor in interpreting them is our full-parameter attack setting: the attacker can update all model weights over a long optimization trajectory. Therefore, defenses based on activation perturbations (e.g., Vaccine) or one-step weight perturbations (e.g., Booster) may not adequately represent the cumulative changes induced by such attacks. This provides a plausible explanation for why their robustness gains can be smaller in our setting than in evaluations with more restricted parameter updates.

Furthermore, we would like to refer you to Section 3 of the general Q&A, which provides three additional findings on the baseline TAR:

- TAR improves modestly with a larger training budget. Its post-attack ASR decreases from 88.1% to 84.5% when its outer-loop budget doubles from 250 to 500 steps, compared with 20.9% for Patcher.

- TAR remains less robust than Patcher with an aligned initialization. Starting from Qwen2.5-1.5B-Instruct, TAR achieves 81.7% post-attack ASR, compared with 45.9% for Patcher.

- TAR establishes strong pre-attack safety under both initializations. With 250 outer-loop steps, it achieves 0.0% ASR on all three benchmarks.

These results show that TAR successfully establishes pre-attack safety, while a substantial gap in post-attack robustness remains in the additional comparisons with a larger training budget or an aligned initialization. Our analysis for Q2 below suggests that insufficient defense updates might contribute to this gap. Specifically, when reducing Patcher's defense updates to 200 steps, which is comparable to TAR's outer loop updates, Patcher also becomes very vulnerable against the attack.

Together, the shared evaluation protocol and additional baseline checks support the comparison under the evaluated full-parameter malicious fine-tuning setting.

AuthorsOpenReview ↗

Author Response (2/7)

2. Robustness under adaptive attacks (W2).

The robustness of PATCHER is sensitive to the attack configuration. The defense deteriorates substantially under changes in learning rate, optimizer, and attack objective. Furthermore, its ASR increases significantly with higher harmful data ratios and longer attack trajectories, suggesting limited robustness against stronger or adaptive malicious fine-tuning.

This is an important question. We acknowledge that the robustness of Patcher varies by the degree of distribution shift in test-time attacks, including learning rate, optimizer and the attack objectives. However, we emphasize that Patcher explicitly targets the full-parameter finetuning attack setting, and delivers significant improvements under common full-parameter attack configurations evaluated in our main experiments. Its design combines longer simulated attacks to improve robustness with attack-vector reuse to improve computational efficiency. We view this design and its demonstrated effectiveness in the evaluated settings as the main contribution of this work.

Regarding adaptive attacks unseen at train-time, we find that Patcher still outperforms other baselines under most adaptive attack settings, and the improvement is non-trivial. For example, Table 9 shows that the overall ASR for Patcher is 64.4% versus 79.9% for SFT under an attack learning rate of 5e-5, and 44.9% versus 55.7% under RMSprop.

Furthermore, as shown in our response to Q3 below, using more diverse attack configurations during training could further enhance Patcher's robustness against seen and unseen attack objectives. For example, under the DPO-SFT malicious finetuning setting, overall ASR decreases from 42.1% to 12.9% for Patcher-adaptive, where diverse configurations are used for simulating attacks during training; under the GA-SFT malicious finetuning setting, overall ASR also decreases from 29.9% to 9.6%. These results suggest that expanding the training-time attack distribution is a promising direction for extending Patcher's robustness to a broader range of fine-tuning objectives.

AuthorsOpenReview ↗

Author Response (3/7)

3. Empirical evaluation of the gradient analysis (Q1).

Could the authors empirically measure the relative gradient estimation errors discussed in Propositions 1 and 2? How well do these errors correlate with the observed attack success rates?

Thank you for this suggestion. We measured the relative gradient estimation error in the practical training setting, using AdamW, 300 simulated attack steps, and a 1000-step reuse window in the final attack–defense iteration. Section 1 of the general Q&A reports the full measurements with two main findings:

- Longer simulated attacks are associated with smaller gradient estimation errors. Along the simulated attack trajectory, the error is 2.291 at step 30, 0.860 at step 100, 0.634 at step 200, and 0.428 at step 280. The smaller errors near the end of the trajectory are consistent with the benefit of longer attack simulations observed in Figure 5aa.

- Gradient estimation error remains relatively stable as the reuse window increases. During attack-vector reuse, the relative error stays between 1.001 and 1.010 over the sampled 1000-step window, changing from 1.001 at step 100 to 1.008 at step 1000. Its absolute value is around 1, but the small variation suggests limited additional discrepancy from extending reuse within this window, consistent with the moderate ASR increase observed in the reuse-duration ablation experiment.

In summary, we find that the dynamics of relative gradient estimation error in one iteration correlate well with the ASR of the final model. Together with the controlled ablations in our response to Q2 below, they support the motivation for longer attack simulations and attack-vector reuse. We have revised the manuscript's Section 3.2 to clarify that our propositions serve as the motivation for the design, while the practical reuse interval is supported empirically.

Reviewer_dSuYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PATCHER, an alignment-stage defense against malicious full-parameter fine-tuning of open-weight LLMs. It simulates multi-step malicious fine-tuning to obtain an attack vector, which is then reused during defense training to reduce computational cost. The paper also provides a theoretical analysis of gradient estimation error and introduces a parallel implementation. Experiments on several models and attack settings show substantial reductions in post-attack ASR, while generally preserving downstream utility.

Critical strengths
  1. The paper addresses malicious full-parameter fine-tuning, an important concern for open-weight LLMs. I appreciate that PATCHER targets the alignment stage, where model providers can improve safety before releasing model weights. The proposed approach is also relatively simple to understand and implement.

  2. The results in Table 1 are encouraging. PATCHER achieves much lower ASR than the evaluated baselines across three safety benchmarks and multiple attack datasets. These results suggest that training against simulated malicious fine-tuning can meaningfully improve robustness under the tested settings.

  3. I appreciate the large effort spent evaluating different setups including adaptive attacks. The additional judge calibration and efficiency analysis are also useful. In particular, the adaptive experiments help reveal both the strengths and limitations of PATCHER.

Critical weaknesses

W1. Limited robustness against alternative fine-tuning attacks. I am concerned that the main robustness claim may be broader than what the results support. In Table 10, DPO-based fine-tuning with β=0.5 gives PATCHER an average ASR of 42.1%, slightly worse than SFT at 40.7%. Table 9 also shows that increasing the attack learning rate to 5e-5 raises PATCHER's ASR to 64.4%. Since attackers have full access to the model weights, they can change the fine-tuning objective and configuration. I appreciate that the authors report these cases, but they seem important to the practical effectiveness of the defense.

W2. The improvement may not come from the claimed mechanism. A key motivation is that simulating more attack steps improves robustness. However, Figure 5a also shows that longer attacks produce larger parameter perturbations. I find it difficult to tell whether the improvement comes from longer attack trajectories or simply from larger perturbations. A simpler baseline using shorter but norm-matched attack vectors would help establish whether many-step simulation is really necessary.

W3. The theoretical analysis does not fully explain the experimental results. Propositions 1 and 2 analyze relative gradient estimation error under several assumptions, including gradient descent and specific relationships between training and testing attacks. The actual experiments use AdamW, different attack datasets, and different attack-step budgets. Moreover, the paper does not directly measure gradient estimation error. I am therefore not fully convinced that the theoretical explanation accounts for the observed ASR improvements.

W4. The comparison with prior defenses needs further justification. I have some concerns about the TAR baseline. Appendix D uses 250 outer-loop steps for TAR, while PATCHER uses 15,000 defender updates under a different training objective. Although these steps are not directly comparable, the paper does not fully explain how the training budgets were matched. TAR also performs substantially worse than vanilla SFT in Table 1. I would like to understand whether this reflects an inherent limitation of TAR or the chosen configuration. In addition, a clearer comparison with closely related methods such as AntiDote would strengthen the technical positioning.

W5. The efficiency advantage and reuse trade-off need clearer evaluation. Attack-vector reuse is an attractive idea, but its overall computational benefit is not fully established. Table 3 shows that parallel PATCHER reduces wall-clock time from 2.25 to 1.55 hours, while increasing total GPU memory from 44.05 to 90.97 GB. GPU-hours and scaling costs are not reported. Also, when varying the reuse duration k, it is unclear whether the total number of defender updates is kept fixed. These details matter when assessing the efficiency gained from reusing attack vectors.

Questions for authors

Q1. Could the authors clarify the intended scope of PATCHER's robustness? In particular, can the defense maintain a meaningful advantage over SFT when attackers choose different objectives, such as DPO, or stronger learning rates?

Q2. Could the authors compare PATCHER with shorter attack trajectories rescaled to the same L2 norm? This would help distinguish the benefit of longer attack trajectories from that of larger perturbations.

Q3. Could the authors explain how Propositions 1 and 2 apply to the actual PATCHER implementation? It would also be helpful to measure the relative gradient estimation error and examine its relationship with ASR.

Q4. Could the authors explain how the TAR training budget and initialization were selected, and report its pre-attack safety and utility? Could the authors also clarify the difference from AntiDote and whether a direct comparison is feasible?

Q5. Could the authors report GPU-hours and per-GPU memory, ideally across different model sizes? Also, when varying k in the reuse-duration ablation, was the total number of defender updates kept fixed?

7 comments
AuthorsOpenReview ↗

Author Response (1/5)

We thank the reviewer for the constructive feedback on our manuscript. We hope that the following clarifications and experiments help address your concerns:

1. Robustness against adaptive fine-tuning attacks.

W1. Limited robustness against alternative fine-tuning attacks. I am concerned that the main robustness claim may be broader than what the results support... Since attackers have full access to the model weights, they can change the fine-tuning objective and configuration.

Q1. Could the authors clarify the intended scope of PATCHER's robustness? In particular, can the defense maintain a meaningful advantage over SFT when attackers choose different objectives, such as DPO, or stronger learning rates?

Thank you for raising this concern. We agree that our robustness claims should more clearly distinguish the attack setting targeted by Patcher from its generalization to attacks unseen during training. We emphasize that Patcher is designed primarily to improve the robustness against full-parameter malicious finetuning with the common cross-entropy objective ie˙,˙SFTi\.e\., SFT [1,2,3], explicitly formulated in Section 3 and simulated during training. Under this objective, Patcher proposes two key designs: simulating longer attack trajectories to simulate stronger perturbations, and reusing attack vectors within certain time window to reduce computational cost.

Under the commonly-used attack setting introduced in Section 4.1, Patcher substantially reduces the ASR compared to other baseline methods. Furthermore, for higher learning rates, optimizers and other hyperparameters under the same SFT finetuning objective, Patcher retains a meaningful advantage over the SFT defense baseline. For instance, at an attack learning rate of 5e-5, Patcher achieves 64.4% ASR compared with 79.9% ASR for SFT, reducing ASR by 15.5 percentage points. Therefore, we view the evaluated settings that use the SFT finetuning objective and Patcher's effectiveness under these settings as the main scope of Patcher's defense.

For alternative objectives, while Patcher shows transfer of robustness to attack objectives unseen during training, its advantage over SFT is indeed less consistent. We therefore do not claim advantage over SFT under these settings. Accordingly, we have revised the abstract, introduction and conclusion, as well as added a new paragraph in Section 4.2, which explicitly states that Patcher improves robustness against SFT-based full-parameter malicious finetuning across a range of evaluated settings, while the advantage does not consistently extend to all full-parameter finetuning objectives.

[1] Huang, Tiansheng, et al. "Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Huang, Tiansheng, Sihao Hu, and Ling Liu. "Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack." Advances in Neural Information Processing Systems 37 (2024): 74058-74088.

[3] Tamirisa, Rishub, et al. "Tamper-resistant safeguards for open-weight llms." International Conference on Learning Representations. Vol. 2025. 2025.

AuthorsOpenReview ↗

Author Response (2/5)

2. Robustness using a rescaled attack vector.

W2. The improvement may not come from the claimed mechanism... A simpler baseline using shorter but norm-matched attack vectors would help establish whether many-step simulation is really necessary.

Q2. Could the authors compare PATCHER with shorter attack trajectories rescaled to the same L2 norm? This would help distinguish the benefit of longer attack trajectories from that of larger perturbations.

Thank you for pointing out this possibility. To distinguish Patcher’s benefit of longer attack trajectories from larger perturbations, we ran 2 additional ablations which simulated 100-step and 200-step attacks but rescaled the attack vector to match the L2 norm of the 300-step attack vector. Other training configurations are kept the same as the main experiments. The result is as follows:

Source of Attack VectorAdvbenchBeavertailsHEx-PHIOverall
100-step, rescaled56.042.452.050.1
200-step, rescaled60.850.464.558.6
300-step, as in our manuscript18.218.126.520.9

As shown in the table, rescaling the attack vector obtained from 100 and 200-step attacks degrades Patcher’s defense significantly, indicating that a larger L2 norm of the perturbation is insufficient to explain the gains from longer simulated attacks. The findings support the importance of the direction of the attack vector obtained by simulating longer attack trajectories.

AuthorsOpenReview ↗

Author Response (3/5)

3. Validation of theoretical analysis.

W3. The theoretical analysis does not fully explain the experimental results... The actual experiments use AdamW, different attack datasets, and different attack-step budgets. Moreover, the paper does not directly measure gradient estimation error...

Q3. Could the authors explain how Propositions 1 and 2 apply to the actual PATCHER implementation? It would also be helpful to measure the relative gradient estimation error and examine its relationship with ASR.

We appreciate your careful analysis of our manuscript. We would like to refer you to Section 1 of the general Q&A, where we present the relative gradient estimation error for different attack steps and reuse windows. It shows that relative gradient error increases as the number of simulated attack steps decreases, while remaining relatively stable as the reuse window increases. This is consistent with the theoretical analysis and the ablation results in our manuscript.

Nevertheless, we acknowledge that the theoretical analysis in the manuscript serves as the theoretical motivation for scaling the simulated attack and reusing attack vectors, and does not indicate full applicability to the real training setting with more complex optimizers and different attack configurations. We explicitly state this limitation in Section 3.2 of the revised manuscript.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Q&A (Section 1/3)

We thank all reviewers for the thorough analysis and constructive feedback on our manuscript. Below, we present additional experiment results and analysis to address some general questions.

1. Measuring the relative gradient estimation error.

As suggested by the reviewers, we measured the relative gradient estimation error under the realistic Patcher training setting, where the number of simulated attack steps is 300, the reuse window is 1000, and the optimizer is AdamW. However, since the original definition of relative gradient estimation error requires computing the second-order Hessian-gradient product, the computation becomes unaffordable under the current setting. Despite that, to provide evidence for Proposition 1 and 2, we computed the relative gradient estimation error defined in Section 3.2 with respect to ∇L(zT(θ))\nabla L (z_T(\theta)), a first-order approximation of the true gradient ∇θL(zT(θ))\nabla_\theta L(z_T(\theta)).

We present this error along the 300-step attack trajectory in the last Patcher attack stage ie˙,˙iteration15i\.e\., iteration 15 below, using the step-300 gradient ∇L(z300(θ))\nabla L (z_{300}(\theta)) as the reference gradient.

Attack stepRelative gradient estimation error
101.133
201.435
302.291
400.753
500.874
600.607
700.675
800.807
900.822
1000.860
1100.820
1200.891
1300.524
1400.635
1500.732
1600.683
1700.745
1800.691
1900.884
2000.634
2100.755
2200.728
2300.753
2400.869
2500.440
2600.332
2700.521
2800.428
2900.783
3000.000

We also measured the error along the 1000-step attack reuse window in the last Patcher defense stage:

Reuse stepRelative gradient estimation error
1001.001
2001.002
3001.001
4001.008
5001.008
6001.003
7001.005
8001.002
9001.010
10001.008

In the first table, as the gap between the simulated attack step and the actual attack step increases, the error accumulates gradually, eventually reaching 2.291 when the number of attack steps is 30. The consistent increase in gradient estimation error corresponds to the growth of ASR shown in Figure 5aa of our manuscript. Turning to the second table, we see a mild increase of the error as the reuse steps increase from 100 to 1000, reaching 1.008 at a 1000-step reuse window. This also supports the results of Section 4.5 of the manuscript, where we observe a limited increase of ASR as the reuse-duration expands from 200 to 1000. Therefore, the dynamics of relative gradient estimation error serve as evidence that scaling the simulated attack steps is necessary, and reusing attack vectors within a certain time window incurs limited impact on robustness.

We have added the empirical results to Appendix A.2 of the revised manuscript.

AuthorsOpenReview ↗

General Q&A (Section 2/3)

2. Decoupling the effect of increasing simulated attack steps and the reuse window.

Since reviewers suggested an ablation study in which the effects of simulated attack steps and the duration of the reuse window are decoupled, we conducted a series of experiments to ablate the effects while keeping other configurations unchanged. The results are presented below:

MethodAdvbenchHEx-PHIBeavertailsOverall
Total 200 defense steps, simulate 100 attack steps, no reuse86.488.577.684.2
Total 200 defense steps, simulate 300 attack steps, no reuse80.082.076.079.3
Total 200 defense steps, simulate 100 attack steps, reuse each attack vector for 10 steps93.688.584.889.0
Total 200 defense steps, simulate 300 attack steps, reuse each attack vector for 10 steps82.083.076.480.5
Total 115 defense steps, simulate 300 attack steps, no reuse (matched compute with original Patcher)91.686.579.285.8
Total 15000 defense steps, simulate 300 attack steps, reuse each attack vector for 1000 steps, as in manuscript18.218.126.520.9

As shown in the table, increasing the simulated attack steps from 100 to 300 leads to a decrease in ASR from 84.2% to 79.3% for 1-step reuse, and 89.0% to 80.5% for 10-step reuse. This shows that increasing the simulated attack steps improves the model's robustness. Meanwhile, increasing the reuse window from 1 to 10 leads to an increase in ASR from 84.2% to 89.0% for 100-step attack, and 79.3% to 80.5% for 300-step attack. This indicates that increasing the reuse window could harm the model's robustness, but the negative effect is milder under the 300-step attack setup than the 100-step attack setup. Finally, we see a substantial decrease in ASR from 85.8% to 20.9% when the total defense steps increase from 115 to 15000 under the 300-step attack setup. This suggests that increasing the total defense optimization steps is crucial for robustness, and expanding the reuse window adequately helps to achieve a desirable trade-off between the robustness and computational efficiency.

We have added the experiment results and analysis to Section 4.5 of the revised manuscript.

AuthorsOpenReview ↗

General Q&A (Section 3/3)

3. Initialization and implementation of the TAR baseline.

Reviewers also raise questions about the initialization and implementation configurations of TAR. Here, we clarify that the 250 outer-loop step budget for TAR was chosen to keep its overall training cost comparable to the other defenses, as we reported in Table 3 of the manuscript. Nevertheless, we agree with the reviewers that comparable training time alone does not indicate that TAR is sufficiently optimized. Therefore, we further doubled TAR’s outer loop budget to 500 steps, surpassing the total FLOPs used for Patcher training. The results are shown below. In this setting, we find a modest decrease of ASR from 88.1% to 84.5%, indicating that increasing the outer loop steps of TAR is helpful, but still performs worse than Patcher, whose ASR is 45.9%.

In addition, we recognize that the original TAR implementation used a pre-aligned model, such as Llama-3-8B-Instruct. However, in our original manuscript, we used the Qwen2.5-1.5B-Base model and tuned it with the Alpaca dataset to control the influence of pre-existing defense. To investigate sensitivity to this choice, we additionally evaluated both TAR and Patcher starting from the officially aligned Qwen2.5-1.5B-Instruct checkpoint, using TAR's official code and training recipe. However, as shown in the table, TAR only demonstrates mild robustness against Beavertails, the in-distribution evaluation dataset, but fails to generalize to out-of-distribution benchmarks such as Advbench and HEx-PHI.

Model InitializationMethodPre-finetuning ASRPost-finetuning ASROverall Post-finetuning ASR
AdvbenchBeavertailsHEx-PHIAdvbenchBeavertailsHEx-PHI
Qwen2.5-1.5B-Base + AlpacaPatcher0.02.00.018.218.126.520.9
TAR, 250 outer-loop steps0.00.00.091.279.693.588.1
TAR, 500 outer-loop steps0.81.24.090.070.093.584.5
Qwen2.5-1.5B-InstructPatcher0.02.00.052.836.848.045.9
TAR, 250 outer-loop steps0.00.00.086.462.896.081.7

We attribute the failure to two main reasons. Firstly, the original TAR paper’s evaluation procedure differs from ours, where the model’s harmfulness is evaluated on HarmBench. However, our manuscript evaluates the model’s safety on Advbench, Beavertails and HEx-PHI using a calibrated LLM judge. The distribution shifts of evaluation benchmarks, as well as evaluation standards, may lead to differences between the ASR evaluated in the original TAR paper and our manuscript.

Secondly, the experiment in the previous section shows that when reducing the Patcher’s defense optimization steps to 200 steps in total, Patcher also becomes quite vulnerable against the attacks, with an ASR increase from 20.9% to 80.5%. This indicates that the number of defender updates is crucial for Patcher’s robustness, which we hypothesize is an important reason that may explain why TAR performs substantially worse than Patcher – insufficient outer-loop optimization steps makes it harder for the defender to find parameters that are insensitive to the attacks.

We thank all reviewers again for their valuable feedback. Please also see our response to each reviewer, and let us know if you have more concerns or questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.