- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses safety alignment degradation during LoRA fine-tuning. The authors extend the learning-dynamics framework of Ren & Sutherland (2025) to the LoRA parameterization: by differentiating the adapted layer with respect to and separately, the first-order alignment-score change decomposes into two channels: the B-channel (proportional to ) and the A-channel (proportional to ). Setting the observation-side factor in each channel to zero yields the dual-null conditions: (DN1) and (DN2) .
Based on this decomposition, the authors propose dual-null LoRA, a fine-tuning-stage defence in three phases: (1) a calibration phase collects per-layer activations and gradients from a small safety demonstration set, eigendecomposes their covariances, and extracts approximate null spaces within an energy budget ; (2) the LoRA factors are initialized within these null spaces via truncated SVD of the doubly projected base weight; (3) during training, gradients of and are projected back onto the constrained subspaces at every step to maintain the dual-null conditions throughout fine-tuning.
Critical strengths
-
Principled theoretical foundation. The core learning-dynamics framework is not original, but the specialization to LoRA is complete: the two-channel kernel decomposition yields a concrete design criterion, and the algorithm follows directly from it rather than from heuristic intuition. This is a notable contrast to most existing defences in this area.
-
Thorough ablation. The ablation (Table 2) systematically isolates initialization, DN1 and DN2, which honestly exposes cases where components are insufficient. Under the authors' setting, dual-null LoRA achieves the best safety–helpfulness trade-off, though the experimental setup (2–3B models, SST-2/AG News/GSM8K, evaluation protocol) closely follows prior work from 2–3 years ago.
Critical weaknesses
Gap between theory and practice. The derivation in Sections 3-4 rests on a first-order Taylor expansion and a single gradient step. The subspaces used to enforce and are calibrated once at initialization and then kept fixed. During multi-epoch training the parameters keep changing, so the activations and the pulled-back gradients drift, and the fixed subspaces may no longer annihilate them. The first-order invariance of the alignment score is therefore guaranteed only near initialization, and its validity over the full training trajectory is assumed rather than shown.
Questions for authors
-
Direct validation of the mechanism. (a) The first-order term is never measured. Could the authors report the alignment score before and after fine-tuning for plain LoRA and dual-null LoRA? (b) For the 256 calibration pairs (harmful prompts paired with aligned refusals from BeaverTails), does the average change before and after fine-tuning directly track the change in model safety (HS and ASR)? That is, does a smaller on these pairs consistently correspond to a smaller increase in HS and ASR?
-
Model scale. The experiments use two small and relatively old models (2-3B). Could the authors report one set of results (plain LoRA vs. dual-null LoRA) on a newer, larger model (e.g., 7-8B)?
-
Asymmetry of DN1 and DN2. In the ablation (Table 2), DN1 alone is nearly as protective as the full method at the protocol learning rate, while DN2 alone gives no protection. The benefit of adding DN2 appears only at the fivefold learning rate, and mainly in AdvBench ASR. Can the authors explain this asymmetry, theoretically or empirically?
4 comments
Author Rebuttal I to Reviewer pgEV
W1: Gap between theory and practice. The derivation in Sections 3-4 rests on a first-order Taylor expansion and a single gradient step. The subspaces used to enforce and are calibrated once at initialization and then kept fixed. During multi-epoch training the parameters keep changing, so the activations and the pulled-back gradients drift, and the fixed subspaces may no longer annihilate them. The first-order invariance of the alignment score is therefore guaranteed only near initialization, and its validity over the full training trajectory is assumed rather than shown.
The drift exists but stays small under dual-null LoRA over the full training trajectory.
We measured after every epoch how far each condition is violated. With and the activations and pulled-back gradients of the 256 calibration pairs, recomputed at the current weights while the null spaces stay fixed, and every matrix scaled to unit Frobenius norm:
Both leaks lie between 0 and 1 and are 0 if and only if the condition holds exactly on all calibration pairs. A growing leak is the drift described in W1. We report the median over the adapted modules.
Under dual-null LoRA both conditions stay close to satisfied: grows from 0.003 to 0.005 and from 0.002 to 0.005 over ten epochs, while plain LoRA reaches 0.092 and 0.038. Each single condition keeps only its own leak small (Table 1 of this response).
Table 1. Leaks after every epoch at the fivefold learning rate and (Qwen2.5-3B-Instruct, AG News, ). Epoch 0 is the initialization. For plain LoRA is undefined at epoch 0, since .
| Leak | Configuration | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ↓ | Plain LoRA | 0.022 | 0.040 | 0.045 | 0.052 | 0.058 | 0.067 | 0.076 | 0.079 | 0.084 | 0.088 | 0.092 |
| DN1 projection, DN1-only init | 0.003 | 0.003 | 0.004 | 0.005 | 0.005 | 0.006 | 0.006 | 0.006 | 0.006 | 0.006 | 0.006 | |
| DN2 projection, DN2-only init | 0.036 | 0.037 | 0.037 | 0.037 | 0.037 | 0.037 | 0.037 | 0.038 | 0.038 | 0.037 | 0.037 | |
| dual-null LoRA (ours) | 0.003 | 0.004 | 0.004 | 0.004 | 0.004 | 0.005 | 0.005 | 0.005 | 0.005 | 0.005 | 0.005 | |
| ↓ | Plain LoRA | 0.063 | 0.053 | 0.047 | 0.045 | 0.040 | 0.041 | 0.040 | 0.039 | 0.038 | 0.038 | |
| DN1 projection, DN1-only init | 0.025 | 0.025 | 0.025 | 0.027 | 0.026 | 0.026 | 0.025 | 0.025 | 0.026 | 0.025 | 0.026 | |
| DN2 projection, DN2-only init | 0.002 | 0.007 | 0.008 | 0.009 | 0.009 | 0.009 | 0.010 | 0.010 | 0.010 | 0.010 | 0.010 | |
| dual-null LoRA (ours) | 0.002 | 0.003 | 0.003 | 0.004 | 0.004 | 0.004 | 0.004 | 0.004 | 0.005 | 0.005 | 0.005 |
Recomputing the null spaces during training would most likely reduce the drift further, but is not necessary at this rank and budget.
Our measurement is complemented by Afzal et al. (2025), who provide theoretical and empirical support for linearized fine-tuning dynamics of LoRA in large language models under suitable conditions.
These results are documented in the new Appendix H of the revised manuscript.
Author Rebuttal II to Reviewer pgEV
Q1: Direct validation of the mechanism. (a) The first-order term is never measured. Could the authors report the alignment score before and after fine-tuning for plain LoRA and dual-null LoRA? (b) For the 256 calibration pairs (harmful prompts paired with aligned refusals from BeaverTails), does the average change before and after fine-tuning directly track the change in model safety (HS and ASR)? That is, does a smaller on these pairs consistently correspond to a smaller increase in HS and ASR?
Under dual-null LoRA the alignment score tracks its initial value closely, and the changes of HS and ASR follow the change of .
(a) The first-order term is never measured. Could the authors report the alignment score before and after fine-tuning for plain LoRA and dual-null LoRA?
is the probability of a safe response to the harmful prompt , between 0 and 1. We estimate it as the share of safe responses among four sampled responses and report , its average over the prompts of a set. For dual-null LoRA, changes by on the calibration prompts and by on held-out harmful prompts. For plain LoRA it changes by and .
(b) For the 256 calibration pairs (harmful prompts paired with aligned refusals from BeaverTails), does the average change before and after fine-tuning directly track the change in model safety (HS and ASR)? That is, does a smaller on these pairs consistently correspond to a smaller increase in HS and ASR?
Yes. Across the four configurations a smaller on the calibration pairs corresponds to a smaller increase in HS and ASR. The same holds on the held-out prompts.
Table 2. after fine-tuning and its change against the released model, HS and ASR (setting of Table 1 of this response).
| Configuration | (calibration) ↑ | (calibration) | (held-out) ↑ | (held-out) | HS ↓ | ASR ↓ |
|---|---|---|---|---|---|---|
| Before FT | 0.766 | 0.611 | 35.4 | 5.1 | ||
| Plain LoRA | 0.409 | 0.098 | 91.4 | 64.7 | ||
| DN1 projection, DN1-only init | 0.435 | 0.158 | 89.1 | 65.6 | ||
| DN2 projection, DN2-only init | 0.447 | 0.148 | 89.6 | 65.1 | ||
| dual-null LoRA (ours) | 0.750 | 0.622 | 37.6 | 4.6 |
Q2: Model scale. The experiments use two small and relatively old models (2-3B). Could the authors report one set of results (plain LoRA vs. dual-null LoRA) on a newer, larger model (e.g., 7-8B)?
The default setting transfers to an 8B model without retuning, at a cost of 7 points of task accuracy. We ran a pilot evaluation on Qwen3-8B with GSM8K at . Dual-null LoRA keeps HS at the level of the released model (33.6 against 34.8), while plain LoRA rises to 43.2. These results are documented in the new Appendix F.2 of the revised manuscript.
Table 3. Qwen3-8B on GSM8K at .
| Method | HS ↓ | ASR ↓ | FA ↑ |
|---|---|---|---|
| Before FT | 34.8 | 8.1 | 31.5 |
| Plain LoRA | 43.2 | 11.7 | 64.0 |
| dual-null LoRA (ours) | 33.6 | 8.1 | 56.8 |
Further tuning of the budget would likely allow for an even better trade-off between protection and task learning at larger model sizes.
Author Rebuttal III to Reviewer pgEV
Q3: Asymmetry of DN1 and DN2. In the ablation (Table 2), DN1 alone is nearly as protective as the full method at the protocol learning rate, while DN2 alone gives no protection. The benefit of adding DN2 appears only at the fivefold learning rate, and mainly in AdvBench ASR. Can the authors explain this asymmetry, theoretically or empirically?
[...] DN1 alone is nearly as protective as the full method at the protocol learning rate, while DN2 alone gives no protection. [...] Can the authors explain this asymmetry, theoretically or empirically?
A theoretical explanation is that DN1 preserves itself over training while DN2 does not.
1. DN1 preserves itself. Suppose DN1 holds exactly at step , that is for every calibration activation . The adapter output is then zero, so the model computes on the calibration inputs what the released model computes, and does not change. The update is restricted to the same null space, , hence . By induction DN1 holds at every step.
2. DN2 does not. The update likewise keeps for the same . The pulled-back gradient , however, depends on the current prediction at (Equation 9 of the revised manuscript). Under DN2 alone the adapter output on is nonzero, the prediction changes, and changes with it. The induction step fails.
This argument is idealized and holds exactly only under exact null spaces. Empirically the leaks behave as predicted by this argument (Table 1 of this response in the comment above): Under DN2 alone, grows fivefold over training (0.002 to 0.010). Under DN1 alone, grows less (0.003 to 0.006).
[...] The benefit of adding DN2 appears only at the fivefold learning rate, and mainly in AdvBench ASR. [...]
With a tighter budget the benefit of DN2 is larger and appears in all harmfulness measures. We ran the setting that Reviewer YRib suggested, the fivefold learning rate with . There dual-null LoRA still protects the model (HS 37.6) while the DN1-only configuration does not (HS 89.1).
Table 4. Ablation at the fivefold learning rate and (Qwen2.5-3B-Instruct, AG News, ).
| Configuration | HS ↓ | ASR ↓ | HS$_A$ ↓ | ASR$_A$ ↓ | FA ↑ | OR ↓ | GC ↑ |
|---|---|---|---|---|---|---|---|
| Before FT | 35.4 | 5.1 | 34.2 | 0.4 | 76.4 | 14.8 | 65.4 |
| Plain LoRA | 91.4 | 64.7 | 80.2 | 57.7 | 87.4 | 2.8 | 65.0 |
| + dual-null init | 87.8 | 64.1 | 78.7 | 61.7 | 85.7 | 0.4 | 65.2 |
| + DN2 projection | 89.3 | 62.3 | 74.8 | 57.3 | 85.6 | 0.4 | 65.1 |
| + DN1 projection | 35.6 | 5.0 | 32.9 | 0.2 | 84.8 | 15.2 | 65.4 |
| + both projections = dual-null LoRA (ours) | 37.6 | 4.6 | 35.4 | 0.2 | 83.6 | 16.4 | 65.4 |
| DN1 projection, DN1-only init | 89.1 | 65.6 | 82.7 | 78.7 | 87.0 | 0.8 | 64.9 |
| DN2 projection, DN2-only init | 89.6 | 65.1 | 67.3 | 41.7 | 85.5 | 2.4 | 65.0 |
We have added these results to the ablation in the main text (Section 5.3, Table 2 of the revised manuscript).
We are happy to clarify any remaining points.
References:
Zahra Rahimi Afzal, Tara Esmaeilbeig, Mojtaba Soltanalian, and Mesrob I Ohannessian. Linearization explains fine-tuning in large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Response to Reviewer pgEV: Thanks and Notation
We sincerely thank the reviewer for the careful reading of our work and for suggesting concrete follow-up measurements. We hope our author rebuttal can address the reviewer's concerns and answer any outstanding questions.
Our evaluation metrics are the same as in our manuscript and are added here again for easier reading of our rebuttal:
All scores in percent.