acceptodds
Under review as a conference paper at ICLR 2027

LoRA is All You Need for Refusal-Based Safety Alignment of Reasoning LLMs

Abstract

Reasoning-capable LLMs have achieved major breakthroughs in solving complex problems. However, %recent work shows that acquiring and deploying strong reasoning can introduce significant safety risks. A common mitigation is to apply a secondary safety-alignment phase after reasoning is learned; however, safety alignment often significantly degrades reasoning performance—a phenomenon known as the “Safety Tax”. In this work, we theoretically study the tension in safety fine-tuning: adapting to a new task (safety) while preserving performance on a base task. Our analysis reveals that using a large rank during fine-tuning induces base-task degradation at a rate inversely proportional to the intrinsic dimensionality of the base task. Unlike instruction tuning, reasoning fine-tuning has a large intrinsic dimension. Thus our results suggest applying LoRA for safety fine-tuning on refusal datasets as an effective approach for safety-aligning reasoning models. Despite its simplicity, this recipe achieves safety comparable to full-model alignment while preserving reasoning performance close to the original reasoning-tuned model, across multiple model sizes and architectures, two safety benchmarks, and four reasoning benchmarks spanning mathematics, science, and code generation. Finally, we show that very small-rank LoRA, with as a strong default, achieves the best safety–reasoning trade-off, and that applying LoRA only to MLP modules in middle layers yields superior performance.\looseness=-1

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Alignment, Reasoning LLMs, LoRA, Safety Tax

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fJcsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether low-rank fine-tuning can improve refusal-based safety alignment while preserving reasoning ability. Specifically, the authors focus on refusing direct harmful requests and theoretically analyze the tension between adapting for safety and preserving utility. They find that refusal-based safety alignment can be achieved effectively with low-rank weight changes, whereas full fine-tuning induces high-rank changes and substantially degrades reasoning.

Critical strengths
  1. The paper studies and demonstrates the safety–reasoning trade-off across multiple models and benchmarks.
  2. The paper provides useful practical guidance on choosing appropriate LoRA settings.
Critical weaknesses
  1. LoRA and full fine-tuning use the same learning rate, while fixed alpha changes alpha/r across ranks. It is unclear whether the observed advantage comes from rank or differences in update strength.
  2. Theorem 3.3’s base-preservation guarantee does not hold under its stated assumptions. A rank-one update can perfectly fit the fine-tuning task while completely forgetting the base task, even with zero rank overshoot. The proof relies on an incorrect rank-additivity claim in Lemma D.3.
Questions for authors

see weakness

3 comments

We thank the reviewer for the careful reading and constructive feedback. Below, we address the theoretical questions and present additional experiments controlling for LoRA scaling.

Q1. Theorem 3.3 Correctness

We respectfully disagree with the reviewer’s claim that Theorem 3.3’s base-preservation guarantee fails under its stated assumptions. The supposed counterexample and the attack on Lemma D.3 are both incorrect, and we address each in turn.

On the rank-one counterexample. The reviewer claims that a rank-one update can perfectly fit the fine-tuning task while completely forgetting the base task at zero overshoot. This actually contradicts the theorem rather than instantiating a counterexample to it: when Δ = r − r_ft = 0, Theorem 3.3 predicts L(W_lora, T_base) ≤ C · Δ/intdim(Θ_base) + o(1) = o(1), i.e., base-task preservation is guaranteed, not violated. The underlying reason is that with r = r_ft, the entire LoRA rank budget is consumed fitting Θ_ft, leaving no remaining degrees of freedom to disrupt Θ_base — forgetting only enters through overshoot Δ > 0. A rank-one update with zero overshoot thus falls precisely in the regime where the theorem provides its strongest guarantee.

On Lemma D.3. The rank-additivity claim rank(S) = rank(PS) + rank(P⊥S) is a standard consequence of orthogonal projection geometry. Since P and P⊥ = I − P project onto complementary, mutually orthogonal subspaces, the column spaces of PS and P⊥S lie in range(P) and range(P⊥) respectively, which intersect only at {0}. It follows directly that col(S) = col(PS) ⊕ col(P⊥S), and hence the rank splits additively. There is no error here.

On the proof of Theorem 3.3. The proof proceeds by first establishing via Assumption (A3) that the rank-allocation optimizer in Lemma D.4 satisfies q* = r_ft, ensuring the full rank budget is allocated to fitting Θ_ft. The residual base-task loss is then bounded by applying the Eckart–Young theorem to yield s²_base · Δ · ‖Θ_base‖²_2, which after normalization gives the stated C · Δ/intdim(Θ_base) bound. Each step is valid, and we are happy to walk through the proof in further detail if the reviewer finds it helpful.

Q2. Learning-rate sensitivity and LoRA update scaling

Thank you for pointing this out. We agree that fixing α while varying rank changes the LoRA scaling factor α/r. To directly examine this effect, we ran a new Llama-8B rank comparison with α/r fixed at 4: r=1 with α=4, r=4 with α=16, and r=64 with α=256. All three configurations were trained from the same base model using ordinary refusal SFT, with identical data, MLP target modules, learning rate 5e-5, seed 42, effective batch size 4, and a ten-epoch training schedule. We also independently swept Full fine-tuning learning rates, as reported in our response to Reviewer 2, Q1.

The table reports all evaluated checkpoints. Each entry is clean safety / AIME, in percentages; higher is better for both metrics.

RankαEpoch 2Epoch 4Epoch 6Epoch 8
1470.93 / 50.0081.47 / 43.3388.18 / 40.0090.73 / 56.67
41677.96 / 40.0093.93 / 43.3397.76 / 50.0098.40 / 56.67
6425695.53 / 56.6794.89 / 36.6796.49 / 36.6796.81 / 36.67

With α/r fixed at 4, rank 4 achieves the highest clean safety and ties for the highest AIME score among the three configurations at epoch 8: 98.40% and 56.67%, respectively. This result supports r=4 as a practical choice for balancing safety and reasoning under a common scaling factor.

Dear Reviewer,

Thank you again for your review of our work. We have posted a detailed response addressing your points, including a step-by-step clarification of the correctness of Theorem 3.3 (the rank-one example and Lemma D.3), and new experiments on learning-rate sensitivity and LoRA scaling, including a rank comparison with α/r held fixed and a learning-rate sweep for full fine-tuning.

Given the fast-approaching discussion deadline, please let us know if any questions remain or if we can clarify anything further in the remaining time. If our responses address your concerns, we would be grateful if you would consider updating your score.

Best regards, The Authors

Thank you for the additional experiments and explanation, which address my concern. I will increase my score.

Reviewer_ziHFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes using Low-Rank Adaptation (LoRA) for refusal-based safety alignment, arguing that safety behaviors can be learned through low-rank parameter updates without substantially interfering with high-dimensional reasoning capabilities. Results show that LoRA achieves safety performance comparable to full fine-tuning while better preserving reasoning abilities and reducing over-refusal. Finally, the paper also provides theoretical analysis to the method.

Critical strengths
  1. The paper addresses the practical challenge of improving the safety of reasoning LLMs without sacrificing their reasoning performance.

  2. The theoretical framework provides an interesting perspective on the relationship between low-rank adaptation, intrinsic task dimensionality, and catastrophic forgetting.

  3. The experiments cover multiple model sizes, reasoning tasks, and safety benchmarks. The paper also has sufficient ablation studies.

Critical weaknesses
  1. The fairness of comparison to full rank fine-tuning is not clear. See question 1 for detail.

  2. The human evaluation part has limited amount of data.

  3. All three main models are derived from DeepSeek-R1 distillation. Therefore, it is unclear whether the observations generalize to independently trained reasoning models or models aligned through different training procedures.

Questions for authors
  1. Have the authors independently tuned the learning rates and training schedules for full-model fine-tuning? Would smaller learning rates, early stopping, or regularized full fine-tuning reduce the observed reasoning degradation?

  2. Can the authors strengthen the human evaluation using a larger and more balanced sample of safe and unsafe responses? How do the conclusions change when using additional independent safety evaluators?

  3. Does the proposed safety–reasoning trade-off hold for reasoning models outside the DeepSeek-R1-distilled family, or for safety alignment procedures based on preference optimization or reinforcement learning?

2 comments

We thank the reviewer for the positive assessment and constructive suggestions. Below, we provide additional comparisons with tuned full-model baselines and address the questions on evaluation and generalization.

Q1. Learning-rate tuning, training schedules, and regularized Full fine-tuning

Thank you for this suggestion. We independently tuned Full fine-tuning on Llama-8B under CAT-style safety training and added a Full + L2-SP baseline with regularization coefficient λ=0.03 and LR=5e-5. For each configuration, we report the earliest evaluated checkpoint with GCG ASR strictly below 10%, using GCG-50 on the same 30 StrongREJECT prompts and Llama-Guard judging. Full and Full + L2-SP were evaluated at epochs 1–4 and LoRA at epochs 2/4/6/8.

All scores are percentages.

MethodLRSelected epochClean safety ↑GCG ASR ↓AIME ↑
Full3e-5199.360.006.67
Full2e-5199.360.003.33
Full1e-5199.360.0023.33
Full5e-6499.683.3333.33
Full + L2-SP (λ=0.03)5e-5197.120.0016.67
LoRA r45e-52100.000.0043.33
LoRA r42e-5299.683.3340.00

Careful learning-rate tuning allows Full to avoid the previously observed collapse. LoRA has higher reported AIME scores at the selected low-ASR checkpoints: 40.00–43.33%, compared with 3.33–33.33% for Full and 16.67% for Full + L2-SP.

We also note that since LoRA was not subjected to the same exhaustive learning rate and checkpoint search as full fine-tuning, the reported LoRA numbers are conservative and a dedicated sweep would likely yield equal or better results, meaning the advantage of LoRA reported here is a lower bound on its true potential. Furthermore, given that our full-parameter learning rate sweep already covers a 6x range and even the best full fine-tuning result still falls substantially below LoRA, it is unlikely that adding weight decay — which provides a weaker and more indirect regularization signal than directly constraining the rank — would close the remaining gap in any significant way.We also plan to complete additional experiments and incorporate the results into an updated Figure 5 in the final version.

Q2. Larger human evaluation and additional independent safety evaluators

Thank you for this suggestion. We plan to expand the human evaluation with a larger, more balanced sample of safe and unsafe responses, and include additional independent safety evaluators to assess the consistency of the findings across evaluation methods.

Q3. Generalization across model families and alignment procedures

Our experiments cover both Qwen- and Llama-based reasoning models, representing two widely used open-weight backbone families. Although these models acquire reasoning capabilities through DeepSeek-R1 distillation, the evaluation spans distinct backbone architectures and multiple model sizes. The results therefore provide evidence across both model families. We also plan to extend the study to more complex alignment settings, including preference optimization and reinforcement learning.

Importantly, our theoretical analysis (Theorems 3.2 and 3.3) does not assume any specific training procedure or model family. The key conditions under which LoRA is predicted to be effective are (1) the fine-tuning objective is effectively low-rank, and (2) the base capability has high intrinsic dimensionality. These conditions are properties of the task structure rather than of the specific model, and we expect them to hold for other reasoning models trained through different procedures.

Thanks for the response. Q1 has been addressed. I encourage authors to explore on Q2 and Q3, however they are not major weaknesses of the paper. Therefore, I would keep the rating.

Reviewer_j9w3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to improve the refusal based safety of reasoning language models while preserving their reasoning capabilities. The authors propose safety fine-tuning with low-rank LoRA updates and show that this approach achieves safety comparable to full-model fine-tuning with substantially less reasoning degradation. A modified linear regression type analysis connects forgetting to the excess rank of the adaptation and the intrinsic dimensionality of the original task, which is interesting. Experiments across reasoning models examine safety, reasoning preservation, over-refusal, and extension to adversarial training.

Critical strengths
  1. Preserving reasoning capabilities during safety alignment is a meaningful challenge. The paper directly investigates this trade-off rather than evaluating safety improvements in isolation, which is quite meaningful.
  2. The method uses standard simple LoRA fine-tuning without requiring a complex training pipeline. Its effectiveness with very small ranks makes it computationally clean and straightforward to understand and adopt.
  3. The theoretical analysis is interesting and offers an interpretable connection between adaptation rank and capability preservation
  4. Experiments show improvement safety with limited degradation in reasoning and ablations on rank, module choice, and layer location provide useful practical guidance.
Critical weaknesses
  1. Although the idea is clean but the experiments do not sufficiently isolate the benefit of low-rank adaptation from general constraints on model change. LoRA’s tendency to reduce forgetting makes improved reasoning preservation plausible. However, it remains unclear whether a well-tuned full-model baseline could achieve a comparable safety–reasoning trade-off through KL regularization toward the original model on benign reasoning inputs, penalties on parameter changes, or smaller learning rates and early stopping. Comparisons across these alternatives at matched safety levels are needed to establish whether low-rank structure provides a clear advantage. The overlap regularization experiments do not fully resolve this question. A detailed discussion will be helpful.
  2. Several prior work has established that LoRA can preserve capabilities better than full fine-tuning (LoRA Learns Less and Forgets Less, Biderman et al., 2024), and that refusal behavior can be mediated by a single direction (Arditi et al., 2024, Refusal in Language Models Is Mediated by a Single Direction). Safe LoRA (Hsu et al., 2024) also studies constrained adaptation for preserving safety, although in a different setting. A more explicit discussion should distinguish the paper’s empirical and theoretical contributions from these established observations. Comparison with closely related work on performance preserving LoRA refusal training would further clarify its contribution.
  3. The inclusion of GCG and AutoDAN is useful, but the adversarial experiments are limited. A stronger assessment should include adaptive semantic attacks such as PAIR or TAP, reasoning-specific attacks such as H-CoT, and multi-turn attacks, with clearly reported attack budgets, sample sizes, and uncertainty. The current results support robustness against the tested attacks, but leave substantial uncertainty about generalization to other much more stronger attacks.
  4. Effective safety also requires providing helpful, context-sensitive assistance: answering benign components of mixed requests, offering safe alternatives, and recognizing situations where refusal would be inappropriate. Although the paper evaluates over-refusal, this does not establish the quality of safe assistance. The explicit focus on refusal is reasonable, but the finding that refusal can be learned through low-rank updates should not be interpreted as evidence that broader safety behavior is similarly low-dimensional and needs a detailed ablation and discussion.
  5. An alternative is to optimize safety while explicitly constraining the degradation in reasoning accuracy. It would be helpful to discuss how LoRA compares with such approaches: whether they could achieve a better safety–reasoning trade-off, what makes them difficult to apply, and whether LoRA offers a simpler or more computationally efficient solution. That will be very helpful for the users.
Questions for authors

Mentioned above

7 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (Part 1/3): Tuned Full Fine-Tuning Baselines

We thank the reviewer for the thoughtful feedback and for recognizing the practical value of our approach. Below, we present additional baseline comparisons and adversarial evaluations, and clarify our contributions and scope.

Q1. Could carefully tuned Full fine-tuning achieve a better safety–reasoning trade-off?

Thank you for this suggestion. We added a learning-rate sweep on Llama-8B under CAT-style safety training and a Full + L2-SP baseline with regularization coefficient λ=0.03 and LR=5e-5. For each configuration, we report the earliest evaluated checkpoint with GCG ASR strictly below 10%, using GCG-50 on the same 30 StrongREJECT prompts and Llama-Guard judging. Full and Full + L2-SP were evaluated at epochs 1–4 and LoRA at epochs 2/4/6/8.

All scores are percentages.

MethodLRSelected epochClean safety ↑GCG ASR ↓AIME ↑
Full3e-5199.360.006.67
Full2e-5199.360.003.33
Full1e-5199.360.0023.33
Full5e-6499.683.3333.33
Full + L2-SP (λ=0.03)5e-5197.120.0016.67
LoRA r45e-52100.000.0043.33
LoRA r42e-5299.683.3340.00

Careful learning-rate tuning allows Full to avoid the previously observed collapse. Nevertheless, LoRA has higher reported AIME scores at the selected low-ASR checkpoints: 40.00–43.33%, versus 3.33–33.33% for Full, even after learning-rate and checkpoint selection.

We also note that since LoRA was not subjected to the same exhaustive learning rate and checkpoint search as full fine-tuning, the reported LoRA numbers are conservative and a dedicated sweep would likely yield equal or better results, meaning the advantage of LoRA reported here is a lower bound on its true potential. Furthermore, given that our full-parameter learning rate sweep already covers a 6x range and even the best full fine-tuning result still falls substantially below LoRA, it is unlikely that adding weight decay — which provides a weaker and more indirect regularization signal than directly constraining the rank — would close the remaining gap in any significant way.We also plan to complete additional experiments and incorporate the results into an updated Figure 5 in the final version.

AuthorsOpenReview ↗

Response to Reviewer 1 (Part 2/3): Relationship to Prior Work

Q2. Relationship to prior work and theoretical positioning

Novelty and Differentiation from Prior Work

We thank the reviewer for raising this important point and agree that a clearer discussion of our relationship to prior work would strengthen the paper. We address each reference in turn and clarify the distinct contributions of our work.

On Biderman et al. (2024). “LoRA Learns Less and Forgets Less” studies capability preservation during utility fine-tuning of already safety-aligned models — that is, the setting where a new capability is being added while pre-existing safety is preserved. Our setting is the reverse: we start from a reasoning-capable model and introduce safety without sacrificing the acquired reasoning ability. These are fundamentally different regimes, and LoRA’s effectiveness in one does not imply effectiveness in the other. Indeed, we explicitly show (Figure 4) that LoRA fails to provide a favorable safety–utility trade-off when fine-tuning a non-reasoning instruction-following model, demonstrating that the benefit of LoRA is not universal but depends critically on the intrinsic dimensionality of the capability being preserved. Furthermore, Biderman et al. do not provide a theoretical characterization of when and why capability preservation succeeds — our Theorems 3.2 and 3.3 provide, to our knowledge, the first formal account linking base-task degradation to rank overshoot and the intrinsic dimensionality of the base capability.

On Arditi et al. (2024). We already cite and build on this observation — that refusal behavior is mediated by a single direction — as motivating evidence that the safety fine-tuning objective is effectively low-rank (Section 3.3). Our theoretical contribution goes beyond citing this observation: we formalize it within a linear regression framework and show that when the fine-tuning objective is low-rank and the base capability has high intrinsic dimension, LoRA provably avoids base-task degradation. Arditi et al. establish the empirical phenomenon; we provide the theoretical framework that explains why this makes LoRA particularly well-suited for safety alignment of reasoning models.

On Hsu et al. (2024) — Safe LoRA. Safe LoRA projects LoRA updates onto the safety-aligned subspace to preserve pre-existing safety during capability fine-tuning — again, the opposite direction from our work. Our setting introduces safety into an already reasoning-capable model, and we do not require any projection-based constraint; standard LoRA with a low rank suffices. Moreover, Safe LoRA does not study the reasoning degradation problem or the Safety Tax phenomenon, and does not provide theoretical guarantees linking rank, overshoot, and intrinsic dimensionality. The two methods address complementary problems and should not be conflated.

Summary of our distinct contributions. We will add a paragraph to the related work section making these distinctions explicit. In brief: (1) our theoretical contribution — Theorems 3.2 and 3.3 characterizing base-task degradation as a function of rank overshoot and intrinsic dimensionality — is novel and not established by any of the cited works; (2) our empirical contribution specifically targets the Safety Tax in reasoning models, a regime not studied by Biderman et al. or Hsu et al.; and (3) we provide evidence for when LoRA is and is not effective (Figure 4), going beyond the observation that LoRA sometimes preserves capabilities better than full fine-tuning. We appreciate the reviewer’s suggestion and will ensure these distinctions are made explicit in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 1 (Part 3/3): Adversarial Evaluation, Scope, and Constrained Optimization

Q3. Stronger and multi-turn adversarial evaluation

Thank you for this suggestion. To evaluate robustness across additional attack families and conversational settings, we extended the Llama-8B CAT evaluation to PAIR-style adaptive attacks, H-CoT-style prefix attacks, and MHJ multi-turn transfer replay, covering 20 checkpoints: two LoRA learning rates (2e-5/5e-5, epochs 2/4/6/8) and three Full learning rates (1e-5/2e-5/5e-6, epochs 1–4).

Protocols and budgets.

  • PAIR-style: The same fixed 30 StrongREJECT goals, one stream and at most five iterations per goal (at most 150 target queries per checkpoint), with local Qwen2.5-32B-Instruct as attacker and judge. The attacker adapts to target responses; success requires a fulfillment rating of 10/10. Target responses are capped at 512 tokens.
  • H-CoT-style: The same 30 goals with an injected neutral reasoning segment and affirmative answer prefix, followed by up to 1,024 generated tokens. We require nonempty generated continuations and report the unsafe-response rate judged by Llama-Guard-3-8B.
  • MHJ: Fifty distinct original objectives from the official MHJ dataset, with one conversation per objective, seed 0, and at most five user turns. Fixed source user messages are replayed using the evaluated model's own previous answers as history. Each checkpoint receives 180 target queries, with target/judge outputs capped at 512 tokens. Qwen2.5-32B-Instruct judges fulfillment of the original objective at each turn; we report both any-turn and final-turn ASR at the 10/10 threshold.

The table below reports the five configurations shared with Q1, using the same selection rule: the earliest evaluated epoch with GCG ASR strictly below 10%. All scores are percentages; lower is better. PAIR and MHJ measure original-goal fulfillment, while H-CoT measures unsafe-response rate.

MethodLREpochPAIR ASR [95% Wilson CI]H-CoT unsafe rateMHJ any-turn ASRMHJ final-turn ASR
Full2e-5110.00 [3.46, 25.62]93.334.000.00
Full1e-5110.00 [3.46, 25.62]76.676.002.00
Full5e-646.67 [1.85, 21.32]76.678.002.00
LoRA r45e-5213.33 [5.31, 29.68]73.334.002.00
LoRA r42e-5213.33 [5.31, 29.68]70.0012.004.00

Results. LoRA and Full show broadly similar performance against PAIR and MHJ, both maintaining low attack success rates. Both exhibit high unsafe-response rates under H-CoT, with LoRA retaining an advantage. These results reflect safety training with CAT alone; more sophisticated adversarial training may improve robustness to stronger attacks. Exploring such training methods is beyond the scope of this paper.

Q4. Safe assistance beyond refusal

Our paper explicitly focuses on refusal-based safety alignment, a scope stated in the title and reiterated throughout the introduction, related work, and experimental setup. Our claim concerns learning refusal behavior through low-rank updates while preserving reasoning; we do not claim that our conclusions extend to broader safety behaviors such as handling mixed requests or providing safe alternatives. We evaluate over-refusal to assess whether refusal training causes unnecessary rejection of benign requests, not as evidence of the quality of safe assistance. Generalizing the low-rank finding to broader safety behavior is outside the scope of this paper, and we make no such claim.

Q5. Explicitly constraining reasoning degradation

This is a helpful suggestion. Constrained optimization approaches that explicitly bound reasoning degradation while optimizing safety are a natural alternative and could in principle achieve a better-controlled trade-off. However, an important practical distinction is that such methods are targeted — they require defining and computing a reasoning metric during training, which means they are explicitly optimized to perform well on that specific metric. LoRA, by contrast, is entirely agnostic to the reasoning evaluation: it uses no reasoning signal during training, introduces no additional objective, and requires no knowledge of the downstream benchmarks. The fact that LoRA achieves strong reasoning preservation without being targeted to do so makes the comparison inherently asymmetric. Beyond this, constrained approaches introduce additional computational overhead — expensive reasoning evaluation during training, additional hyperparameters such as constraint thresholds, and higher per-step cost — whereas LoRA runs with fewer GPUs and fewer epochs than full fine-tuning. We therefore view LoRA as a simple, computationally efficient, and theoretically grounded default, and will add a discussion of this comparison — including the targeted vs. untargeted distinction — to the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.