acceptodds
Withdrawn from ICLR 2027

LoRA Safety Patches: Capability-Preserving Refusal Tuning with Safety Data Alone

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety alignment of large language models (LLMs) often improves resistance to harmful instructions and jailbreak attacks but degrades general capabilities. Existing approaches mitigate this trade-off by mixing safety and general-purpose data, making alignment sensitive to data composition. In this work, we empirically and mechanistically investigate whether standard LoRA can improve safety during safety alignment without sacrificing pretrained capabilities. Across multiple LLM families, LoRA-based refusal tuning using only safety data substantially improves safety while largely preserving general capabilities. Compared with full-parameter tuning, it also supports continual safety alignment with negligible utility loss and exhibits less safety regression under continuous benign fine-tuning. To better understand this behavior, we conduct empirical analyses of parameter updates, hidden-state shifts, and subspace similarity. Our results show that LoRA-based safety updates introduce substantially lower interference with the model’s intrinsic transformations than full-parameter tuning, suggesting a form of subspace decoupling that enables lightweight and modular safety patching for LLMs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, efficient fine-tuning, Safety Alignment, Safety-Utility Trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5LLqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper empirically studies whether standard LoRA (no new algorithm) can serve as a "safety patch": refusal tuning with safety-only data that improves jailbreak defense without degrading general capabilities. Across multiple models, LoRA-based refusal SFT on safety-only data matches full-parameter safety gains with negligible utility loss (Table 1). It is also observed that tuning with safety-only data achieves a better safety-utility tradeoff than tuning with safety-general combined data.

Critical strengths

Comprehensive, practically actionable empirical results. The study covers various model families, alignment objectives (refusal-SFT, DPO), safety benchmarks, utility benchmarks, continual-alignment and benign-FT stress tests. The core message "safety-only LoRA removes the need for fragile safety/general data mixing" is simple and directly useful to practitioners.

Critical weaknesses
  1. Limited novelty over closely related work. The paper does not introduce any novel methodology. "LoRA forgets less than full FT" is established (Biderman et al., 2024, cited), and concurrent work (Xue & Mirzasoleiman, 2025) studies LoRA-only safety alignment. The contribution is a careful characterization rather than a new finding or method.
  2. Some experimental results look like unsuccessful tries that do not convey any consistent and useful insight, and they should be removed in a formal paper. For example, in Section 4.1 and 4.2, the figures show that the magnitude and \delta norm do not consistently align with the claimed improvements.
Questions for authors
  • Can full-parameter safety-only tuning with lower LR / early stopping match LoRA's safety–utility trade-off? Please add this control; it directly tests whether the advantage is intrinsic to low-rank subspaces or to implicit regularization.
  • Does the safety patch survive adaptive fine-tuning attacks (harmful data, higher LR)?
  • What is the security claim given the patch can be detached at deployment?
Reviewer_kPGcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies refusal fine-tuning of LLMs for safety and compares training via low-rank adapters (LoRA) with full-parameter fine-tuning. It shows that LoRA adapters can effectively and efficiently improve safety while preserving general utility using only safety data. Using LoRA for refusal training also leads to lower safety degradation when models are subsequently fine-tuned on general utility datasets.

The paper then attempts to explain these results by analysing parameter updates, hidden-state perturbations, and geometric relationships between LoRA updates and the pretrained model's weight subspaces, and comparing them to full-parameter updates.

Critical strengths

1. Empirical analysis: The paper provides empirical analysis of LoRA-based safety alignment. They cover safety–utility trade-offs, continual fine-tuning, and the underlying effects on parameter updates and hidden representations.

Critical weaknesses

1. Limited novelty and generalization.: The main findings largely repeat established observations about LoRA's ability to preserve pretrained capabilities [1, 2, 3]. The additional contribution to safety alignment is limited, especially given the small models scale, and models diversity. It also would have been interesting to run their experiments on unaligned models and see if their observations hold.

2. Questionable implementation of experiments: The extreme, below-chance utility scores under some full-parameter SFT and DPO runs raise serious concerns about the adequacy of the baselines and hyperparameters tuning. Even under LoRA, some runs that use safety data only perform better on utility benchmarks compared to when general datasets were added, the authors' explanation of this in section 3.2 is not convincing and ncessistate further investigation. These issues weaken the evidence supporting the paper's main conclusions.


[1] LoRA Learns Less and Forgets Less (Biderman et al., 2024)

[2] LoRA vs Full Fine-tuning: An Illusion of Equivalence (Shuttleworth et al., 2024)

[3] LoRA is All You Need for Safety Alignment of Reasoning LLMs (Xue & Mirzasoleiman, 2025)

Questions for authors
  1. Can you double check the utility evaluation of the fine-tuned models? How did some models end-up with far below-chance accuracy on MMLU?
  2. How did you choose which datasets to include in the general-purpose subset? Did you measure how training on the general data alone would impact model utility?
  3. Did you track safety-utility trade-offs across intermediate checkpoints (within epochs) during fine-tuning or only measure the final trained model in each run? How would that curve look like for LoRA vs full-parameters SFT?
Reviewer_8Qp2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LoRA can improve the safety of instruction-tuned language models using only refusal-training data while preserving general capabilities. It compares LoRA and full-parameter tuning under supervised refusal training and DPO across Qwen, Llama, and Mistral models. The experiments report a favorable safety–utility trade-off for safety-only LoRA refusal tuning, with additional evaluations of sequential safety updates and subsequent benign fine-tuning. Analyses of parameter changes, hidden-state shifts, and singular-vector overlap suggest that LoRA safety updates interfere less with pretrained transformations.

Critical strengths
  1. The question is practically important. Improving safety without repeatedly curating general-purpose replay data would simplify model maintenance.

  2. The main empirical pattern is promising across several model families. For example, Table 1 reports that Llama-3.1-8B’s WildJailbreak ASR decreases from 18.30% to 0.45%, while MT-Bench changes from 7.43 to 7.31 and MATH-500 from 45.8 to 45.6. The evaluation covers several complementary aspects of utility.

Critical weaknesses
  1. The comparisons do not yet establish superiority at matched safety levels. Full-parameter safety-only tuning often achieves substantially lower ASR than LoRA,for example, 0.05% versus 7.80% on Qwen’s WildJailbreak evaluation, so part of the utility difference may reflect different alignment strengths.

  2. The subspace metric needs clarification: individual or maximum cosine similarities depend on the chosen bases and are insufficient to characterize subspace separation. Principal angles or projection-based overlap, together with random-subspace controls, would be more informative. Also, Table 4 changes the learned input-to-output mapping as well as its overlap with pretrained directions. Matched random rotations that remain weakly overlapping are needed to distinguish damage from disrupting a learned update from damage specifically caused by increased overlap.

  3. The continual-adaptation claims need better isolation and clearer protocols. Table 2 compares full-parameter DPO with LoRA SFT, changing both parameterization and objective. This is a useful comparison between practical configurations, but it cannot isolate LoRA’s contribution to continual learning. Furthermore, Section 3.3 introduces 15 previously unseen attack types at each of 3 stages, whereas Appendix C states that SafeEdit-Train contains only 15 attack types in total. The stage construction needs clarification.

Questions for authors
  1. With identical safety-only data and SFT objectives, does LoRA retain a clear utility advantage over full tuning at matched ASR across learning rates and intermediate checkpoints?

  2. How much of the full-tuning MMLU collapse comes from refusals or formatting errors? Does the conclusion persist with robust answer extraction and option-likelihood scoring?

  3. In Section 4.4, what happens when the learned update is rotated by a comparable amount into another subspace with low pretrained overlap? How are singular-vector signs and QR conventions handled so that the intervention itself does not introduce unintended changes?

  4. Do the continual-learning conclusions hold when both methods use SFT? For the benign fine-tuning experiment, how are the safety adapter and subsequent adapter merged or updated, and what happens to downstream utility?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.