acceptodds
Under review as a conference paper at ICLR 2027

SafeSchatten: Schatten-Norm-Constrained Steepest Descent in Activation Space for Safe LLM Fine-Tuning

10 views

Abstract

Fine-tuning can improve an aligned language model's task performance while weakening its safeguards. When safety and task learning depend on overlapping representation directions, protecting safety requires controlling how those directions change during adaptation. We introduce SafeSchatten, a fine-tuning framework that budgets activation drift on trusted safety inputs and chooses updates that maximize local task progress within that budget. Frobenius and spectral constraints provide two ways to distribute the allowance across singular directions, yielding closed-form updates for the linear-layer problem. Across three downstream tasks and five aligned models, SafeSchatten substantially reduces harmfulness while retaining useful adaptation. On SST2, it reduces harmfulness from 51.30% under standard fine-tuning to 11.20%, with accuracy changing from 95.76% to 93.46%. Budget sweeps and update ablations connect these gains to the geometry of activation drift, while a one-step analysis motivates adaptive norm selection. These findings establish activation geometry as a useful design choice for safety-preserving fine-tuning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
llm safety, safety alignment, safety preservation, fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jWKeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SafeSchatten limits the per-step change in a layer’s outputs on cached safety. Activations (ΔW·X) and takes the steepest-descent step under a Frobenius or spectral budget. This gives closed-form updates: a whitened gradient with K = XXᵀ, which in the spectral case is a Muon-style matrix-sign step.

Critical strengths
  1. The idea is simple and well motivated. Budget the change in outputs on safety inputs instead of weights or a hard subspace, with a cheap one-time cache of K.
  2. Experiments are broad: seven baselines, five models, data-size and poison-fraction sweeps, ablations, and drift diagnostics. The code link, the single-run caveat, and the ethics statement are all positives.
Critical weaknesses
  1. Unequal tuning, and the “budget” is effectively a learning rate.
  • SafeSchatten is swept over budgets while baselines are single points (Fig. 2).
  1. Text contradicts tables or overclaims.
  • The momentum paragraph says lower harm, but Table 5 shows higher harm in every case (e.g. S-∞ 11.2 to 25.5 on SST2).
  1. It uses one safety benchmark (BeaverTails), which is also the poison source and probably the cache source.
  2. Moderation judge is not specified.
Questions for authors
  1. What is the cumulative drift bound? How is Φ applied to LoRA?
  2. ARAP experiment patches final-block outputs, so downstream layers see base activations by construction. How does it show that per-layer ΔW·X budgets on earlier layers preserve safety? Which token positions are patched, and was LoRA or full fine-tuning used?
  3. The abstract claims three tasks and five models, but the tasks use only Llama-3.2-3B, and the models use only AG News. Do results hold on the full grid?
4 comments
AuthorsOpenReview ↗

Response to Reviewer jWKe (1 of 2)

Dear Reviewer jWKe,

Thank you for your constructive review and for recognizing the simplicity of our approach, its motivation, and our extensive experimental evaluation. We appreciate your helpful suggestions and address them below.

W1. (unequal tuning) Thank you for raising this concern! We conduct the sweep on a variety of learning rates, and each of the configurations is run with three seeds. Full results have been updated in Appendix A.8.1. In summary, our experimental results showed that SafeSchatten remains Pareto optimal across each search space.

Furthermore, we also conduct a sweep across the method-specific hyperparameters for Safe LoRA and AsFT, as which share the same LoRA fine-tuning setup. Specifically, we vary τ∈0.25,0.50,0.75\tau \in {0.25, 0.50, 0.75} for Safe LoRA, where τ\tau controls the cosine-similarity threshold used to determine which LoRA updates are projected onto the safety-aligned subspace. For AsFT, we vary λ∈1,2,5\lambda \in {1, 2, 5}, where λ\lambda controls the regularization strength that penalizes parameter updates orthogonal to the alignment direction. As shown in Table C1 below (full results are in Appendix A.8.2), both methods are sensitive to these hyperparameters, further highlighting the importance of considering method-specific search spaces when comparing safety and downstream-task performance. We then selected the best-task-performing configuration for the main experiments.

Table C1. Hyperparameter Sensitivity (harmfulness (%) / task accuracy (%)) of AsFT and Safe LoRA.

DatasetLRAsFT (λ=1)AsFT (λ=2)AsFT (λ=5)SafeLoRA (τ=0.25)SafeLoRA (τ=0.5)SafeLoRA (τ=0.75)
AG News5×10⁻⁶19.7 ± 0.97/70.5 ± 0.2119.6 ± 0.96/70.6 ± 0.2119.9 ± 0.59/70.5 ± 0.1518.8 ± 0.80/70.2 ± 0.1219.4 ± 1.42/70.1 ± 0.2119.0 ± 0.70/69.9 ± 0.22
AG News10⁻⁵20.3 ± 0.35/71.4 ± 0.0820.6 ± 0.55/71.3 ± 0.0620.1 ± 0.46/71.1 ± 0.2219.9 ± 0.15/70.8 ± 0.2319.5 ± 0.61/70.8 ± 0.2219.5 ± 0.51/70.5 ± 0.19
AG News2×10⁻⁵21.9 ± 0.51/73.0 ± 0.1621.1 ± 0.12/72.4 ± 0.1321.8 ± 1.23/71.8 ± 0.2021.7 ± 0.23/72.2 ± 0.1621.5 ± 0.62/72.2 ± 0.2521.3 ± 0.29/71.3 ± 0.05
AG News5×10⁻⁵23.0 ± 0.69/78.8 ± 1.2523.0 ± 2.05/78.7 ± 1.0521.9 ± 0.59/77.0 ± 0.4524.9 ± 1.04/79.6 ± 1.8124.8 ± 0.36/79.6 ± 1.8224.8 ± 0.12/74.9 ± 0.49

W2. (text contradiction in momentum experiment) We appreciate the reviewer's finding this mismatch. We have revised this paragraph and added further analysis of momentum's effect on SafeSchatten. Indeed, incorporating momentum yields modestly higher downstream-task accuracy, but at the cost of increased harmfulness compared with non-momentum ones. Why does SafeSchatten have this momentum effect? We attribute this phenomenon to SafeSchatten's geometry design and to momentum smoothing gradients across optimization steps, potentially making successive update directions more consistent. We discuss this further in Appendix A.6.

W3/4. (safety benchmark/moderation) Our moderation model used for BeaverTails is their officially released PKU-Alignment/beaver-dam-7b. To examine whether SafeSchatten's safety preservation generalizes to different safety corpora, we added the HarmBench evaluation to fine-tune AG News on Llama-3.2-3B, with the moderation model cais/HarmBench-Llama-2-13b-cls.

Table C2. Safety Evaluation on BeaverTails and HarmBench.

MethodAcc. (↑)BeaverTails (↓)HarmBench (↓)
Base Model53.3716.6019.17
LoRA87.15 ± 0.0551.90 ± 0.0977.08 ± 0.24
SafeDelta88.45 ± 0.1635.00 ± 0.1048.33 ± 0.12
DSS87.71 ± 0.1725.37 ± 0.2924.50 ± 0.82
Surgery86.21 ± 0.3036.73 ± 0.2446.67 ± 0.21
Safe LoRA79.61 ± 1.8124.90 ± 1.0429.17 ± 0.04
AsFT78.76 ± 1.2523.00 ± 0.6925.00 ± 1.19
GuardSpace83.60 ± 0.2923.03 ± 0.3827.92 ± 0.33
SafeSchatten$-2$84.34 ± 0.1815.73 ± 0.3415.83 ± 0.22
SafeSchatten$-\infty$85.97 ± 0.2616.13 ± 0.3512.08 ± 0.36

Table C2 demonstrates that both SafeSchatten variants consistently achieve lower harmfulness than all evaluated baselines on both BeaverTails and HarmBench while retaining competitive downstream-task accuracy. These results suggest that SafeSchatten is not limited to BeaverTails, which we use as the source of harmful fine-tuning data and safety activations. The consistent improvements on HarmBench further demonstrate that our method can preserve safety behavior beyond the dataset used to construct the safety constraints. We will incorporate all the results into the later revision.

AuthorsOpenReview ↗

Response to Reviewer jWKe (2 of 2)

Q1. (cumulative drift bound) Since SafeSchatten constrains the safety activation drift at each training step, a trivial upper bound on the cumulative drift follows from the triangle inequality. Specifically, given the per-step constraint ∣ΔWtX∣Sp≤ε|\Delta W_t X|_{\mathcal S_p}\leq \varepsilon, after t training steps, the cumulative activation drift will be:

$||(W_T-W_0)X|| _{\mathcal S_p} =||\sum _{t=0}^{T-1}\Delta W_tX|| _{\mathcal S_p} \leq\sum _{t=0}^{T-1}||\Delta W_tX|| _{\mathcal S_p} \leq T \cdot\varepsilon$.

Therefore, under a fixed per-step constraint ε\varepsilon, the cumulative drift is upper-bounded by T⋅εT \cdot \varepsilon for both Schatten-2 and Schatten$-\infty$ constraints. However, this is a worst-case upper bound, not a guarantee of the model's safety. In practice, the actual cumulative drift depends on how successive updates interact, since changes in different directions may partially cancel out or accumulate. This also provides a possible explanation for the degradation in safety preservation observed when increasing the number of fine-tuning steps or incorporating momentum (cf. W2 or Appendix A.6), where successive activation changes may become more directionally consistent. We acknowledge that the current formulation does not enforce a constant cumulative drift budget, and deriving global constraints is an interesting direction for future work.

(How does Φ\Phi applied to LoRA?) Thank you for raising this question. Following LoRA, we parameterize the weight matrix as W=W0+sBAW=W_0+sBA, where W0W_0 is frozen and A,BA,B are trainable low-rank factors. Since the LoRA update involves two coupled matrices, we adopt a tractable surrogate that separately constrains the activation changes induced by each factor. Specifically, using the safety-activation covariances K=XX⊤K=XX^\top and C=s2AKA⊤C=s^2AK A^\top, we derive the corresponding SafeSchatten update directions as: ΦA=εDp(GAK−1/2)K−1/2\Phi_A = \varepsilon\mathcal D_p(G_AK^{-1/2})K^{-1/2} and ΦB=εDp(GBC−1/2)C−1/2\Phi_B = \varepsilon\mathcal D_p(G_BC^{-1/2})C^{-1/2}, where Dp\mathcal D_p denotes the normalized gradient for p=2p=2 and the matrix-sign operation for p=∞p=\infty. Both factors are then updated as A+=A−ηΦAA^+=A-\eta\Phi_A and B+=B−ηΦBB^+=B-\eta\Phi_B. This lets us apply the same covariance-dependent steepest-descent framework to LoRA while controlling the activation changes of each factor separately. We have added the detailed derivation and implementation discussion in Appendix A.3.

Q2. (Does per-layer ΔW·X budgets on earlier layers preserve safety in ARAP?) This is a great question, and we have conducted the early-layer patching experiment. In particular, instead of restoring activations only at the final decoder block, we further investigate whether restoring activations at middle or early layers (i.e., the 16th and 5th decoder blocks) can recover the model's safety behavior.

Table C3. Effect of Decoder-Block Activation Patching on Safety.

ModelBase Harm (↓)FT Harm (↓)ARAP Last LayerARAP 16th LayerARAP 5th Layer
Llama-2-7B-Chat0.9028.801.006.3025.30
Llama-3.1-8B-Instruct8.7054.108.5013.1049.50
Llama-3.2-3B-Instruct16.6046.9016.4020.2044.80
Qwen2-7B-Instruct6.4048.306.4020.6042.90
Gemma-2-9B-IT5.6065.106.1015.1053.80
Mean7.6448.647.6815.0643.26

These results further demonstrate that preserving intermediate safety activations can recover safety, not just in the final decoder module. This further supports our SafeSchatten algorithm, which restricts activation changes during fine-tuning rather than directly replacing them with the aligned model's activations.

Regarding the experimental setup, ARAP fine-tunes in full parameter space and replaces the fine-tuned model's activations with the original alignment model's activations at matching sequence positions, covering both prompt and generated tokens. We have clarified these implementation details and included the early-layer results in the revision.

Q3. (full grid claims) We apologize for causing this misunderstanding for the reviewer. In fact, because our experiments span five models, three datasets, and nine methods, we could not run full-grid experiments on all configurations in the limited time of the rebuttal (we will provide more results on this in the later revision). That being said, we believe our main results (Table 2 and Table 3) are sufficient to demonstrate the generalizability of SafeSchatten across models and tasks, and we have revised our abstract to correct the claim before finishing the full-grid experiments.

AuthorsOpenReview ↗

Follow up to the Reviewer

Dear Reviewer jWKe,

Thank you again for your valuable feedback. As the discussion period is closing soon, we'd like to follow up and see if the Reviewer has had a chance to consider our response. We are happy to provide any further clarification or answer any additional questions you may have.

We hope our responses and new results are helpful for finalizing the review rating.

Yours sincerely,

Authors

AuthorsOpenReview ↗

A Kindly Reminder

Dear Reviewer jWKe,

Thank you again for your thoughtful review and constructive feedback. As the discussion period nears its close, we would like to briefly summarize our revisions and ask whether our responses address your concerns.

Specifically, we have implemented the following key improvements:

  1. Experimental rigor and fair comparison: We conducted hyperparameter sweeps for the learning rate (using three different random seeds; see Appendix A.8.1). Furthermore, we conducted method-specific hyperparameter sweeps for Safe LoRA and AsFT (Appendix A.8.2) to provide a more comprehensive safety-utility comparison.

  2. Safety evaluation: We added HarmBench evaluations, clarified the moderation models, and expanded our activation-patching experiments to earlier decoder blocks (these suggestions really improve the motivation and robustness of SafeSchatten; we appreciate this).

  3. Theoretical and implementation details: We clarified the cumulative activation-drift bound and momentum effect on SafeSchatten, and provided a detailed derivation for the LoRA implementation of SafeSchatten in Appendix A.3.

We hope these additions have addressed your concerns and made our contributions clearer. In particular, if you have any remaining questions about the LoRA implementation or its relationship to our theoretical formulation, we would be happy to provide further clarification.

It would be very much appreciated if you could once again help review our responses and let us know if these address your concerns. We are eager to answer any further questions you might have. We strive to improve the paper consistently, and it is our pleasure to have your feedback!

Yours Sincerely,

Authors

Reviewer_knnqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes SafeSchatten, a fine-tuning method that aims to keep an aligned model safe while it learns a downstream task. The idea is to limit how much the model's activations on trusted safety inputs are allowed to change during training, and to choose the update that gives the most task progress within this limit. The authors measure the change with Schatten norms and derive closed-form updates for two cases: a Frobenius-norm version and a spectral-norm version. They also propose a rule to switch between the two norms during training. Experiments on three tasks and five aligned models show lower harmfulness than standard fine-tuning, with a small loss in task accuracy.

Critical strengths
  1. The problem is practical and important. Keeping safety during fine-tuning is a real concern for fine-tuning-as-a-service.
  2. The formulation is clean. Framing safe fine-tuning as constrained steepest descent on activation drift is easy to follow, and the closed-form updates are simple to implement.
  3. The paper evaluates against a good range of recent defenses, across several tasks and model families.
Critical weaknesses
  1. Limited novelty. That safety and utility share overlapping directions is already well established [1]. Preserving model behavior on safety inputs by constraining how updates change their activations is also established, through null-space and safety-subspace projections [2, 3]. The Frobenius version is close to a soft relaxation of these methods. The spectral version is the known steepest-descent update under the spectral norm [4], and combining spectral updates with protected directions has appeared in continual learning [5]. The paper mainly combines existing pieces.

  2. Weak experimental rigor. All results come from a single seed, with no variance reported. Many gaps over the strongest baselines are small, so it is unclear whether they are meaningful.

  3. Unfair comparison. SafeSchatten is shown with a full sweep of its budget, while baselines appear at a single setting. A simple baseline is also missing: standard fine-tuning with a smaller learning rate. Without it, we cannot tell whether the gain comes from the proposed geometry or just from smaller updates.

References

[1] Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study. ICLR 2026.

[2] SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation. ICLR 2025.

[3] A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space. ICLR 2026.

[4] Old Optimizer, New Norm: An Anthology. OPT 2024 Workshop.

[5] Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning. arXiv 2026.

Questions for authors

Please refer to the Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer knnq (1 of 2)

Dear Reviewer knnq,

Thank you for your careful review and for recognizing the practical importance of our problem, the clarity of our formulation, and the breadth of our evaluation. We appreciate your concerns regarding theoretical soundness and experimental validation, which we address below.

W1. (limited novelty) We understand that the idea to constrain the safety activations or updates in the null space may make the reviewer feel that it lacks originality because prior literature has addressed this issue in these directions. However, we would like to respectfully point out several new components and designs in our SafeSchatten that may not have been sufficiently emphasized, as well as highlight our major contributions to safety-preservation fine-tuning.

[Our novel design for SafeSchatten] We agree that the overlap between safety and utility directions has been claimed in [1], which we also discuss in our Introduction. However, our work does not rediscover this observation; it addresses ‘how to preserve safety while still allowing the model to adapt in these overlapping directions’, and we are the first study to address it from this angle. Prior arts (e.g., SaLoRA [2] and GuardSpace [3]) address it with orthogonal or null-space projections. SafeSchatten, in contrast, does not require the update to be orthogonal to the safety subspace. Instead, we formulate the problem as a constrained steepest descent problem, allowing a finite amount of safety activation drift while maximizing downstream-task improvement. Our results show this design can reach a better balance between safety preservation and task adaptation.

[Our contribution in studying safe fine-tuning from a geometric optimization perspective] Although the reviewers noted that, in [4], the steepest descent of the spectrum and matrix sign updates had been well studied, our work considers a completely different optimization problem, as the constraints of SafeSchatte are defined by the change in safety-related activations rather than by the weight updates themselves. We constrain ∣ΔWX∣Sp≤ρ|\Delta W X|_{S_p}\leq\rho, where the safety activation covariance K=XX⊤K=XX^\top determines the geometry of the optimization. Under this formulation, different Schatten norms lead to different ways of allocating allowed activation drifts in singular directions, resulting in different update directions and trade-offs between safety and task adaptation. This also distinguishes our work from Muon-OGD [5], which combines the spectral norm constraint of the weight updates with the exact orthogonality of the protected directions. While we build on existing optimization techniques, we believe SafeSchatten offers a new perspective not directly addressed in prior work by establishing a safety-preserving mechanism through a limited activation-drift budget and studying how its geometry affects safety fine-tuning. We have referenced these two papers [4,5] in the revision.

[Our extensive empirical evaluations to demonstrate why these design choices matter] In addition to deriving the Frobenius and spectral update directions, we evaluated SafeSchatten on a variety of downstream tasks, methods, and models. We also conducted the hyperparameter search for SafeSchatten (activation drift) and baselines (learning rates/method-specific hyperparameters; see Appendix A.8), harmful data and training-scale sensitivity analyses, and ablation experiments on update geometry and momentum to better understand when and why our method is effective. For example, on the SST2 dataset, SafeSchatten$-\infty$ achieved harmfulness and accuracy of 11.10 and 93.36, respectively, while GuardSpace's corresponding values ​​were 18.13 and 91.30. These results show that controlling the magnitude and geometry of safety activation drift can yield larger improvements than traditional subspace projection/orthogonalization.

In summary, we acknowledge that SafeSchatten builds upon existing insights and optimization techniques, but we would like to emphasize that our contribution lies in how these ideas are formulated as a safety condition optimization problem, the update rules generated under different Schatten geometries, and our empirical understanding of the safety and utility trade-offs.

AuthorsOpenReview ↗

Response to Reviewer knnq (2 of 2)

W2/3. (experiment rigor, fair comparison) We thank the reviewer for raising this concern, and indeed, this is important to demonstrate the robustness of our method. We conduct the sweep on a variety of learning rates and each of the configurations is run with three seeds. Full results have been updated in Appendix A.8.1. In summary, our experimental results showed that SafeSchatten remains Pareto optimal across each search space.

Furthermore, we also conduct a sweep across the method-specific hyperparameters for Safe LoRA and AsFT, as which share the same LoRA fine-tuning setup. Specifically, we vary τ∈0.25,0.50,0.75\tau \in {0.25, 0.50, 0.75} for Safe LoRA, where τ\tau controls the cosine-similarity threshold used to determine which LoRA updates are projected onto the safety-aligned subspace. For AsFT, we vary λ∈1,2,5\lambda \in {1, 2, 5}, where λ\lambda controls the regularization strength that penalizes parameter updates orthogonal to the alignment direction. As shown in Table B1 below (full results are in Appendix A.8.2), both methods are sensitive to these hyperparameters, further highlighting the importance of considering method-specific search spaces when comparing safety and downstream-task performance. We then selected the best-task-performing configuration for the main experiments.

Table B1. Hyperparameter Sensitivity (harmfulness (%) / task accuracy (%)) of AsFT and Safe LoRA.

DatasetLRAsFT (λ=1)AsFT (λ=2)AsFT (λ=5)SafeLoRA (τ=0.25)SafeLoRA (τ=0.5)SafeLoRA (τ=0.75)
AG News5×10⁻⁶19.7 ± 0.97/70.5 ± 0.2119.6 ± 0.96/70.6 ± 0.2119.9 ± 0.59/70.5 ± 0.1518.8 ± 0.80/70.2 ± 0.1219.4 ± 1.42/70.1 ± 0.2119.0 ± 0.70/69.9 ± 0.22
AG News10⁻⁵20.3 ± 0.35/71.4 ± 0.0820.6 ± 0.55/71.3 ± 0.0620.1 ± 0.46/71.1 ± 0.2219.9 ± 0.15/70.8 ± 0.2319.5 ± 0.61/70.8 ± 0.2219.5 ± 0.51/70.5 ± 0.19
AG News2×10⁻⁵21.9 ± 0.51/73.0 ± 0.1621.1 ± 0.12/72.4 ± 0.1321.8 ± 1.23/71.8 ± 0.2021.7 ± 0.23/72.2 ± 0.1621.5 ± 0.62/72.2 ± 0.2521.3 ± 0.29/71.3 ± 0.05
AG News5×10⁻⁵23.0 ± 0.69/78.8 ± 1.2523.0 ± 2.05/78.7 ± 1.0521.9 ± 0.59/77.0 ± 0.4524.9 ± 1.04/79.6 ± 1.8124.8 ± 0.36/79.6 ± 1.8224.8 ± 0.12/74.9 ± 0.49
Reviewer_knnqOpenReview ↗

Response to authors

I thank the authors for the response and additional experiments. The added seeds and hyperparameter sweeps address my concerns about rigor.

However, several concerns remain. First, the novelty is still limited. Allowing a drift budget instead of strict orthogonality is a soft variant of existing projection methods, built from known optimization tools. Second, the comparison is still not fully fair. Baselines are selected by best task accuracy, which may place them at their least safe setting, and it is unclear whether plain LoRA or Muon with a learning-rate sweep is included. Third, the gap between the full-matrix derivation and the LoRA implementation is not addressed. Finally, the inconsistent numbers across tables are not explained.

I will keep my score.

Reviewer_vyRxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SafeSchatten, a fine-tuning method intended to preserve the safety behavior of aligned LLMs while learning a downstream task. The main idea is to limit the activation drift induced by each weight update on a cached set of trusted safety-input activations. The method derives Frobenius- and spectral-norm-constrained update directions using the safety-activation covariance and investigates an adaptive choice between these geometries. The reported results show reductions in harmfulness while retaining competitive downstream-task performance.

Critical strengths

1- Specifying an activation-drift budget, rather than completely limiting the updates to a space that is orthogonal to safety directions is an interesting idea. Also, the distinction between a Frobenius constraint and a spectral constraint is well motivated and gives the method an interpretable design parameter from a new perspective.

2- The synthetic experiment for showing why the choice of geometry will change the descent direction is technically interesting and helps frame the importance of norm selection in optimization problems.

3- The paper considers 3 downstream tasks but evaluates AG News on five model families. This provides some evidence that the method is not tied to one base model. The paper also evaluates multiple harmful-data fractions and training-set sizes.

4- Paper provides different ablations to demonstrated the importance of considering matrix-level constraints and effect of including momentum. Regardless of the conclusions, this is a valuable practice that tries to provide essential evidence for the claims, rather than leaving the unanswered.

Critical weaknesses

1- The budget sweeps shows that increasing the activation-drift budget can improve downstream performance while reducing safety, and the paper attributes this trade-off to overlap between safety-relevant and downstream-task directions, but further evidence is needed to establish that the gains arise specifically from preserving safety-relevant activations rather than from constraining activation drift more generally. For example, how would the results change if the trusted safety dataset were replaced with a similarly sized benign control dataset? This would help isolate the contribution of the safety-specific data.

2- The authors mention that there is a trade-off between safety and performance when using the momentum version vs the original method; could they explain why using the momentum would hurt the safety score? Also please adjust the language, as the drop in the safety score does not seem to be slight.

3- Reporting peak memory consumption, throughput, and wall-clock overhead would help clarify the practicality of the proposed method. This is especially important because the method caches safety activations and applies covariance-based transformations during optimization. The paper should also discuss how these costs scale with the number of cached safety samples.

4- The experiment on number of samples and portion of harmful examples suggest that when these two are scaled, the safety score of the proposed method drops and is more similar or even lagging behind other safety preserving baselines. Comparing methods based on their efficiency (w3) along these results would be helpful for having a more comprehensive and fair comparison.

5- Although the appendix reports some training details, it is unclear how the main method and baselines were tuned. Could the authors report the search spaces and tuning budgets used for each method? In addition as some reported differences are relatively small, results over multiple random seeds, at least for a subset of experiments, would help establish the robustness and fairness of the comparisons.

5- The method introduce several additional hyperparameters, including the activation-drift budget and norm geometry, while showing sensitivity to the size of dataset and harmful samples. Could you please compare the first two, in addition to memory and compute report, so that you can provide a more precise comparison between you method and the baselines?

Questions for authors

Please refer to weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vyRx (1 of 2)

Dear Reviewer vyRx,

Thank you for your thoughtful review and for recognizing the motivation behind our activation-drift formulation, the importance of norm geometry, and the breadth of our experiments. We appreciate your constructive feedback and address your questions below.

W1. (ablation on safety activation) This is a great suggestion! We conducted an experiment by replacing the safety dataset used for SafeSchatten constraining with a benign chat dataset (sampled from UltraChat). As shown in Table A1 below (also updated in Appendix A.7), replacing safety activations with benign chat activations leads to higher harmfulness scores under both norm constraints, while task accuracy remains similar. This suggests that SafeSchatten not only constrains arbitrary activations but also protects the safety subspace.

Table A1. Ablation on Safety vs. Benign Activation Caches.

MethodHarm (↓)Acc. (↑)∥SAD∥F\lVert \mathrm{SAD}\rVert_F∥SAD∥\lVert \mathrm{SAD}\rVert
Base Model16.6053.37--
LoRA51.90 ± 0.0987.15 ± 0.050.4249 ± 0.00850.2865 ± 0.0035
ChatSchatten$-2$48.10 ± 0.2585.94 ± 0.120.4692 ± 0.00050.4461 ± 0.0014
SafeSchatten$-2$15.73 ± 0.3484.34 ± 0.180.2811 ± 0.00220.2024 ± 0.0020
ChatSchatten$-\infty$30.80 ± 0.1784.38 ± 0.110.3309 ± 0.00140.3651 ± 0.0014
SafeSchatten$-\infty$16.13 ± 0.3585.97 ± 0.260.2704 ± 0.00120.1982 ± 0.0028

W2. (why momentum hurt safety preservation) Thank you for raising this concern. We agree that safety preservation decreases clearly for SafeSchatten$-\infty$, while the decrease is relatively minor for SafeSchatten-2; we have revised our wording in the paper. We explain this from two perspectives. Geometrically, the Schatten-2 norm constrains the overall magnitude of activation drift across all singular directions, whereas the Schatten$-\infty$ norm constrains the largest singular value while allowing greater total drift distributed across multiple directions. This difference aligns with the motivation behind spectral normalization, which controls the dominant singular value while maintaining flexibility in the remaining spectral directions [1]. However, these constraints apply to a single step and do not directly constrain the cumulative activation drift, making SafeSchatten$-\infty$ more sensitive to accumulated changes under momentum. On the other hand, as shown in Table A2 below, we also observed that enabling momentum results in larger safety activation drift, which we hypothesize stems from drift accumulating throughout training. Specifically, momentum smooths the gradients across optimization steps, potentially making successive update directions more consistent [2]. As SafeSchatten constrains the activation drift at each individual update rather than the cumulative drift throughout training, these changes may accumulate instead of canceling out, leading to weaker safety preservation. We have incorporated this into our revised version.

Table A2. Effect of Momentum on Safety Preservation and Activation Drift.

MethodSST2 Harm (↓)SST2 Acc. (↑)SST2 ∥SAD∥F\lVert \mathrm{SAD}\rVert_FSST2 ∥SAD∥\lVert \mathrm{SAD}\rVertAG News Harm (↓)AG News Acc. (↑)AG News ∥SAD∥F\lVert \mathrm{SAD}\rVert_FAG News ∥SAD∥\lVert \mathrm{SAD}\rVert
Base Model16.6079.36--16.6053.37--
LoRA51.33 ± 0.3695.76 ± 0.290.4624 ± 0.00510.3389 ± 0.001451.90 ± 0.0987.15 ± 0.050.4249 ± 0.00850.2865 ± 0.0035
SafeSchatten$-2$13.57 ± 0.3892.61 ± 0.290.1000 ± 0.00500.0614 ± 0.000415.73 ± 0.3484.34 ± 0.180.2811 ± 0.00220.2024 ± 0.0020
SafeSchatten$-2$ (M)15.23 ± 1.2393.03 ± 0.210.1228 ± 0.01340.0750 ± 0.003217.73 ± 0.3885.47 ± 0.140.3506 ± 0.00970.2617 ± 0.0043
SafeSchatten$-\infty$11.10 ± 0.3693.36 ± 0.200.1359 ± 0.00300.0921 ± 0.002716.13 ± 0.3585.97 ± 0.260.2704 ± 0.00120.1982 ± 0.0028
SafeSchatten$-\infty$ (M)24.83 ± 1.0394.53 ± 0.340.3392 ± 0.02490.2543 ± 0.002429.83 ± 0.1484.86 ± 0.240.3904 ± 0.00140.2848 ± 0.0041

[1] Miyato, Takeru, et al. "Spectral normalization for generative adversarial networks." arXiv preprint arXiv:1802.05957 (2018).

[2] Li, Xianliang, et al. "Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering." arXiv preprint arXiv:2606.03899 (2026).

AuthorsOpenReview ↗

Response to Reviewer vyRx (2 of 2)

W3. (computing costs) Below, we provide the computing cost analysis with other baseline methods, as well as how the number of safety datasets affects the efficiency of our method.

Table A3. Computational Cost and Peak GPU Memory Across Methods.

MethodSST2 GPU-hoursSST2 Peak GPU Memory (GiB)AG News GPU-hoursAG News Peak GPU Memory (GiB)GSM8K GPU-hoursGSM8K Peak GPU Memory (GiB)
LoRA0.049.940.0510.900.0913.37
SafeDelta0.0725.930.0926.060.1326.38
DSS0.0725.940.0926.060.1326.38
Surgery0.1227.990.1428.940.1931.49
Safe LoRA0.179.930.1910.970.2113.41
AsFT0.1812.710.1913.500.2216.86
GuardSpace0.2622.540.2823.750.3026.34
SafeSchatten$-2$0.2413.090.2514.050.2816.52
SafeSchatten$-\infty$0.4813.090.5014.050.5216.52

As shown in Table A3, SafeSchatten-2 costs similar time to GuardSpace (0.26 - 0.30 GPU-hours), while consuming less GPU memory. Furthermore, SafeSchatten$-\infty$ spent more computational time (0.48 - 0.52 GPU-hours) on additional spectral updates, but maintains a similar memory footprint to SafeSchatten-2. Although our method adds computational cost compared with standard LoRA fine-tuning, the results show that SafeSchatten-2 remains computationally competitive with other safety-preserving baselines, particularly in GPU memory consumption.

To assess how the number of safety data affects memory costs, in Table A4, we scale the number of data from 25, 50, to 100 and observe that peak memory increases from ≈10 to ≈13 GiB. We have included the results in Appendix A.5.

Table A4. Computational Cost under Different Sizes of Safety Activation Cache.

MethodActivation CacheSST2 GPU-hoursSST2 Peak GPU Memory (GiB)AG News GPU-hoursAG News Peak GPU Memory (GiB)GSM8K GPU-hoursGSM8K Peak GPU Memory (GiB)
SafeSchatten$-2$250.165310.70830.180311.66500.214614.1307
SafeSchatten$-2$500.188211.49330.198812.45150.233414.9160
SafeSchatten$-2$1000.235913.09250.249314.04950.283816.5171
SafeSchatten$-\infty$250.385710.70830.397011.66500.432114.1307
SafeSchatten$-\infty$500.411111.49330.429712.45150.458814.9160
SafeSchatten$-\infty$1000.477213.09250.495314.04950.520216.5171

W4/5/6. (scalability, search space, and hyperparameter sensitivity) We address these questions together. Regarding scalability, we acknowledge that SafeSchatten does not always outperform all baselines as the training data size or poisoning ratio increases. We hypothesize this is partly due to accumulated safety activation drift throughout training. Because SafeSchatten constrains activation changes at each step, increasing the training dataset while keeping the number of epochs fixed increases the number of steps, allowing cumulative drift to grow even when the step-wise constraint is satisfied. This suggests that the activation-drift budget may need to be adjusted based on training steps and exposure to harmful data. To make a fairer comparison across baselines, in Appendix A.8.1, we re-run the experiment with four learning rates and three seeds. We also search the method-specific hyperparameters of Safe LoRA and AsFT (as shown inTable A5; full results are updated in Appendix A.8.2), which share the same LoRA fine-tuning setup. In short, the experimental results showed that SafeSchatten remains Pareto optimal across each search space. Nevertheless, SafeSchatten remains computationally efficient, and its safety preservation stays competitive as data or poison exposure increases.

Table A5. Hyperparameter Sensitivity (harmfulness (%) / task accuracy (%)) of AsFT and SafeLoRA.

DatasetLRAsFT (λ=1)AsFT (λ=2)AsFT (λ=5)SafeLoRA (τ=0.25)SafeLoRA (τ=0.5)SafeLoRA (τ=0.75)
AG News5×10⁻⁶19.7 ± 0.97/70.5 ± 0.2119.6 ± 0.96/70.6 ± 0.2119.9 ± 0.59/70.5 ± 0.1518.8 ± 0.80/70.2 ± 0.1219.4 ± 1.42/70.1 ± 0.2119.0 ± 0.70/69.9 ± 0.22
AG News10⁻⁵20.3 ± 0.35/71.4 ± 0.0820.6 ± 0.55/71.3 ± 0.0620.1 ± 0.46/71.1 ± 0.2219.9 ± 0.15/70.8 ± 0.2319.5 ± 0.61/70.8 ± 0.2219.5 ± 0.51/70.5 ± 0.19
AG News2×10⁻⁵21.9 ± 0.51/73.0 ± 0.1621.1 ± 0.12/72.4 ± 0.1321.8 ± 1.23/71.8 ± 0.2021.7 ± 0.23/72.2 ± 0.1621.5 ± 0.62/72.2 ± 0.2521.3 ± 0.29/71.3 ± 0.05
AG News5×10⁻⁵23.0 ± 0.69/78.8 ± 1.2523.0 ± 2.05/78.7 ± 1.0521.9 ± 0.59/77.0 ± 0.4524.9 ± 1.04/79.6 ± 1.8124.8 ± 0.36/79.6 ± 1.8224.8 ± 0.12/74.9 ± 0.49
AuthorsOpenReview ↗

Follow up to the Reviewer

Dear Reviewer vyRx,

Thank you again for your valuable feedback. As the discussion period is closing soon, we'd like to follow up and see if the Reviewer has had a chance to consider our response. We are happy to provide any further clarification or answer any additional questions you may have.

We hope our responses and new results are helpful for finalizing the review rating.

Yours sincerely,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you!!

Dear Reviewer vyRx,

Thank you for your encouraging feedback. We sincerely appreciate your recognition of our efforts to address your concerns.

We agree that scalability is an important aspect to consider. We believe that extending our method to limit cumulative activation drift would be a promising direction for future work.

Thank you again for your constructive feedback and valuable suggestions, which have helped us strengthen our paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.