- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SafeSchatten limits the per-step change in a layer’s outputs on cached safety. Activations (ΔW·X) and takes the steepest-descent step under a Frobenius or spectral budget. This gives closed-form updates: a whitened gradient with K = XXᵀ, which in the spectral case is a Muon-style matrix-sign step.
Critical strengths
- The idea is simple and well motivated. Budget the change in outputs on safety inputs instead of weights or a hard subspace, with a cheap one-time cache of K.
- Experiments are broad: seven baselines, five models, data-size and poison-fraction sweeps, ablations, and drift diagnostics. The code link, the single-run caveat, and the ethics statement are all positives.
Critical weaknesses
- Unequal tuning, and the “budget” is effectively a learning rate.
- SafeSchatten is swept over budgets while baselines are single points (Fig. 2).
- Text contradicts tables or overclaims.
- The momentum paragraph says lower harm, but Table 5 shows higher harm in every case (e.g. S-∞ 11.2 to 25.5 on SST2).
- It uses one safety benchmark (BeaverTails), which is also the poison source and probably the cache source.
- Moderation judge is not specified.
Questions for authors
- What is the cumulative drift bound? How is Φ applied to LoRA?
- ARAP experiment patches final-block outputs, so downstream layers see base activations by construction. How does it show that per-layer ΔW·X budgets on earlier layers preserve safety? Which token positions are patched, and was LoRA or full fine-tuning used?
- The abstract claims three tasks and five models, but the tasks use only Llama-3.2-3B, and the models use only AG News. Do results hold on the full grid?
4 comments
Response to Reviewer jWKe (2 of 2)
Q1. (cumulative drift bound) Since SafeSchatten constrains the safety activation drift at each training step, a trivial upper bound on the cumulative drift follows from the triangle inequality. Specifically, given the per-step constraint , after t training steps, the cumulative activation drift will be:
$||(W_T-W_0)X|| _{\mathcal S_p} =||\sum _{t=0}^{T-1}\Delta W_tX|| _{\mathcal S_p} \leq\sum _{t=0}^{T-1}||\Delta W_tX|| _{\mathcal S_p} \leq T \cdot\varepsilon$.
Therefore, under a fixed per-step constraint , the cumulative drift is upper-bounded by for both Schatten-2 and Schatten$-\infty$ constraints. However, this is a worst-case upper bound, not a guarantee of the model's safety. In practice, the actual cumulative drift depends on how successive updates interact, since changes in different directions may partially cancel out or accumulate. This also provides a possible explanation for the degradation in safety preservation observed when increasing the number of fine-tuning steps or incorporating momentum (cf. W2 or Appendix A.6), where successive activation changes may become more directionally consistent. We acknowledge that the current formulation does not enforce a constant cumulative drift budget, and deriving global constraints is an interesting direction for future work.
(How does applied to LoRA?) Thank you for raising this question. Following LoRA, we parameterize the weight matrix as , where is frozen and are trainable low-rank factors. Since the LoRA update involves two coupled matrices, we adopt a tractable surrogate that separately constrains the activation changes induced by each factor. Specifically, using the safety-activation covariances and , we derive the corresponding SafeSchatten update directions as: and , where denotes the normalized gradient for and the matrix-sign operation for . Both factors are then updated as and . This lets us apply the same covariance-dependent steepest-descent framework to LoRA while controlling the activation changes of each factor separately. We have added the detailed derivation and implementation discussion in Appendix A.3.
Q2. (Does per-layer ΔW·X budgets on earlier layers preserve safety in ARAP?) This is a great question, and we have conducted the early-layer patching experiment. In particular, instead of restoring activations only at the final decoder block, we further investigate whether restoring activations at middle or early layers (i.e., the 16th and 5th decoder blocks) can recover the model's safety behavior.
Table C3. Effect of Decoder-Block Activation Patching on Safety.
| Model | Base Harm (↓) | FT Harm (↓) | ARAP Last Layer | ARAP 16th Layer | ARAP 5th Layer |
|---|---|---|---|---|---|
| Llama-2-7B-Chat | 0.90 | 28.80 | 1.00 | 6.30 | 25.30 |
| Llama-3.1-8B-Instruct | 8.70 | 54.10 | 8.50 | 13.10 | 49.50 |
| Llama-3.2-3B-Instruct | 16.60 | 46.90 | 16.40 | 20.20 | 44.80 |
| Qwen2-7B-Instruct | 6.40 | 48.30 | 6.40 | 20.60 | 42.90 |
| Gemma-2-9B-IT | 5.60 | 65.10 | 6.10 | 15.10 | 53.80 |
| Mean | 7.64 | 48.64 | 7.68 | 15.06 | 43.26 |
These results further demonstrate that preserving intermediate safety activations can recover safety, not just in the final decoder module. This further supports our SafeSchatten algorithm, which restricts activation changes during fine-tuning rather than directly replacing them with the aligned model's activations.
Regarding the experimental setup, ARAP fine-tunes in full parameter space and replaces the fine-tuned model's activations with the original alignment model's activations at matching sequence positions, covering both prompt and generated tokens. We have clarified these implementation details and included the early-layer results in the revision.
Q3. (full grid claims) We apologize for causing this misunderstanding for the reviewer. In fact, because our experiments span five models, three datasets, and nine methods, we could not run full-grid experiments on all configurations in the limited time of the rebuttal (we will provide more results on this in the later revision). That being said, we believe our main results (Table 2 and Table 3) are sufficient to demonstrate the generalizability of SafeSchatten across models and tasks, and we have revised our abstract to correct the claim before finishing the full-grid experiments.
Follow up to the Reviewer
Dear Reviewer jWKe,
Thank you again for your valuable feedback. As the discussion period is closing soon, we'd like to follow up and see if the Reviewer has had a chance to consider our response. We are happy to provide any further clarification or answer any additional questions you may have.
We hope our responses and new results are helpful for finalizing the review rating.
Yours sincerely,
Authors
A Kindly Reminder
Dear Reviewer jWKe,
Thank you again for your thoughtful review and constructive feedback. As the discussion period nears its close, we would like to briefly summarize our revisions and ask whether our responses address your concerns.
Specifically, we have implemented the following key improvements:
-
Experimental rigor and fair comparison: We conducted hyperparameter sweeps for the learning rate (using three different random seeds; see Appendix A.8.1). Furthermore, we conducted method-specific hyperparameter sweeps for Safe LoRA and AsFT (Appendix A.8.2) to provide a more comprehensive safety-utility comparison.
-
Safety evaluation: We added HarmBench evaluations, clarified the moderation models, and expanded our activation-patching experiments to earlier decoder blocks (these suggestions really improve the motivation and robustness of SafeSchatten; we appreciate this).
-
Theoretical and implementation details: We clarified the cumulative activation-drift bound and momentum effect on SafeSchatten, and provided a detailed derivation for the LoRA implementation of SafeSchatten in Appendix A.3.
We hope these additions have addressed your concerns and made our contributions clearer. In particular, if you have any remaining questions about the LoRA implementation or its relationship to our theoretical formulation, we would be happy to provide further clarification.
It would be very much appreciated if you could once again help review our responses and let us know if these address your concerns. We are eager to answer any further questions you might have. We strive to improve the paper consistently, and it is our pleasure to have your feedback!
Yours Sincerely,
Authors
Response to Reviewer jWKe (1 of 2)
Dear Reviewer jWKe,
Thank you for your constructive review and for recognizing the simplicity of our approach, its motivation, and our extensive experimental evaluation. We appreciate your helpful suggestions and address them below.
W1. (unequal tuning) Thank you for raising this concern! We conduct the sweep on a variety of learning rates, and each of the configurations is run with three seeds. Full results have been updated in Appendix A.8.1. In summary, our experimental results showed that SafeSchatten remains Pareto optimal across each search space.
Furthermore, we also conduct a sweep across the method-specific hyperparameters for Safe LoRA and AsFT, as which share the same LoRA fine-tuning setup. Specifically, we vary τ∈0.25,0.50,0.75 for Safe LoRA, where τ controls the cosine-similarity threshold used to determine which LoRA updates are projected onto the safety-aligned subspace. For AsFT, we vary λ∈1,2,5, where λ controls the regularization strength that penalizes parameter updates orthogonal to the alignment direction. As shown in Table C1 below (full results are in Appendix A.8.2), both methods are sensitive to these hyperparameters, further highlighting the importance of considering method-specific search spaces when comparing safety and downstream-task performance. We then selected the best-task-performing configuration for the main experiments.
Table C1. Hyperparameter Sensitivity (harmfulness (%) / task accuracy (%)) of AsFT and Safe LoRA.
W2. (text contradiction in momentum experiment) We appreciate the reviewer's finding this mismatch. We have revised this paragraph and added further analysis of momentum's effect on SafeSchatten. Indeed, incorporating momentum yields modestly higher downstream-task accuracy, but at the cost of increased harmfulness compared with non-momentum ones. Why does SafeSchatten have this momentum effect? We attribute this phenomenon to SafeSchatten's geometry design and to momentum smoothing gradients across optimization steps, potentially making successive update directions more consistent. We discuss this further in Appendix A.6.
W3/4. (safety benchmark/moderation) Our moderation model used for BeaverTails is their officially released
PKU-Alignment/beaver-dam-7b. To examine whether SafeSchatten's safety preservation generalizes to different safety corpora, we added the HarmBench evaluation to fine-tune AG News on Llama-3.2-3B, with the moderation modelcais/HarmBench-Llama-2-13b-cls.Table C2. Safety Evaluation on BeaverTails and HarmBench.
Table C2 demonstrates that both SafeSchatten variants consistently achieve lower harmfulness than all evaluated baselines on both BeaverTails and HarmBench while retaining competitive downstream-task accuracy. These results suggest that SafeSchatten is not limited to BeaverTails, which we use as the source of harmful fine-tuning data and safety activations. The consistent improvements on HarmBench further demonstrate that our method can preserve safety behavior beyond the dataset used to construct the safety constraints. We will incorporate all the results into the later revision.