acceptodds
Under review as a conference paper at ICLR 2027

A Gravitational Interpretation of Safety Reversion under Fine-Tuning

Abstract

Safety alignment in large language models can degrade during post-training even when neither the data nor the objective is intentionally adversarial. Alignment rebound and reverse dynamics suggest that this degradation may reactivate behavior suppressed during safety alignment. Building on these ideas, we hypothesize that ordinary non-adversarial post-training follows a reversion direction: the activation-space displacement from the safety-aligned model toward a more permissive, earlier helpful-only state. We see that for Llama, every tested trajectory across references, tasks, and seeds exceeds a matched empirical null, while at aligned Llama and Qwen checkpoints, a vocabulary readout shows that the direction locally favors task-engaging over fixed refusal-like openings. Its geometric expression is behaviorally informative: as post-training proceeds, alignment with the direction and harmfulness increase together, yielding a strong descriptive correlation (Spearman r=0.958). To move beyond correlation, we test causal relevance during adaptation using objectives constructed from this coordinate. Across all tested Llama, Qwen, and Gemma settings from 3B to 14B, an optimizer-matched objective opposing positive motion reduces geometric alignment and harmfulness relative to ordinary fine-tuning, whereas a separately stabilized objective reinforcing that motion increases both. Every model and scale exhibits the same mean block-baseline-push ordering, showing that the causal relevance of the reversion direction is not tied to one architecture or model size. Finally, we show that a standard safety-rehearsal objective, built without access to the direction, independently opposes it and cuts cumulative reversion by about 30% in Llama and Qwen.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large language models, safety alignment, benign fine-tuning, safety reversion, representation geometry, training dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZqBmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper discovers the effect of benign training in the post-processing on the safety alignment of the Large Language Models. The paper claims that non-adversarial post-training follows a reversion direction that makes the model more permissive. The methodology is to observe the change of the activations in the layers of post-trained model by comparing against many helpful trained versions using a probe family. The result of the the method is an activation-space direction which can be used as a basis to measure the drift even in benign tasks. The motivation of the paper and the discovery are interesting, yet the experiments can not provide enough evidence.

Critical strengths
  • The topic is interesting in terms of safety and alignment. The paper offers a diagnostic tool that can be used to detect shifts in model alignment during post-training.
  • The reversion is not tied to one architecture or model size.
  • The formulation is simple and easy to understand
Critical weaknesses
  • I had a really hard time reading the paper sections due to the verbose and long sentences. The story that the paper is trying to tell is actually simple, yet it is presented very complexly and with highly specific jargon.
  • The implications and the discovery of the paper are not so surprising since it is well known that post-training can damage the safety alignment of the model. Especially when the paper presents it using two tasks and evaluates it with only one safety dataset, BeaverTails.
  • The paper needs to explicitly compare the difference between reversal direction calculation and just observing the embedding drifts at the layers.
  • The paper should diversify the model architectures and the tasks. It only measured small model sizes and used only two tasks.
  • The experiment section and the figures seem AI-generated, and it is hard to understand.
Questions for authors
  • How do the authors define a benign dataset and make sure that the samples in the dataset are honest and harmless?
  • How many helpful proxy models are generated per direction calculation?
  • What do the authors mean by the "null"? Is it the \theta_P, \theta_S, or \theta_H?
Reviewer_D7xCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study why LLMs become less safe after fine-tuning on harmless data. They define a reversion direction by comparing a safety-aligned model with a helpful-only model trained from the same base. They show that fine-tuning tends to move activations in this direction. Blocking this movement reduces harmful responses, while encouraging it increases them. The experiments cover Llama, Qwen and Gemma models.

Critical strengths

The paper is easy to follow and the idea is interesting. The authors define the direction before running additional experiments. All five new Llama runs show movement in this direction, stronger than expected from the randomized control.

The safety results are consistent across six model settings. For Llama-3.1-8B, blocking reduces harmful responses from 24.6% to 5.7% (Table 2). The authors also report random-direction experiments, including less successful results.

Critical weaknesses

The paper explains how its direction differs in construction from the refusal direction of Arditi et al. (2024), but does not compare them empirically. It would be useful to measure how closely the two directions align and compare their effects during training. Four of five calibrated random-axis blocks also improve safety (Section 3.4), which makes this comparison important for interpreting the blocking results.

Additionally, the blocking experiments only use Alpaca fine-tuning. The authors study natural reversion on GSM8K and JSON tasks, but do not test blocking on them. It is unclear whether the safety improvements generalize to other fine-tuning tasks.

Questions for authors
  1. Can you compare your blocking objective with the same objective applied to the refusal direction from Arditi et al. (2024)?

  2. Does blocking also improve safety during fine-tuning on GSM8K or JSON tasks?

  3. Will the safety improvements remain under less aggressive fine-tuning with better instruction-following performance? Can you also report target-task performance?

Reviewer_SGhEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why safety degrades when an aligned model is fine-tuned on benign data. The authors propose that part of this degradation is a reversion toward an earlier, more permissive state the model passed through before safety alignment. To make this measurable, they train helpful-only reference models from the base model and define a reversion direction in activation space, pointing from the aligned model toward these references. They show that benign fine-tuning trajectories move along this direction more than a matched null would predict, and that this movement rises together with harmfulness. They then add training objectives that block or push motion along the direction, and show that blocking reduces harmfulness while pushing increases it. Finally, they show that a standard safety-rehearsal loss also naturally opposes this direction.

Critical strengths
  1. The question is important. Understanding why benign fine-tuning breaks safety, not just that it does, is useful for the community.
  2. The experimental protocol is careful. The reference models and the test are fixed before the target runs, and a matched null is used, which makes the geometric evidence more credible.
Critical weaknesses
  1. The core idea is not new. Prior work has shown that aligned models rebound toward their pre-alignment distribution under further fine-tuning [1], and that benign fine-tuning data can push the model back toward the pretrained distribution along a reversion direction [2]. The paper mainly moves this analysis into activation space with helpful-only proxies. This refines an existing hypothesis rather than offering a new insight.

  2. The causal evidence is not specific to the proposed direction. Blocking several random directions also reduces harmfulness, sometimes as much as the proposed direction. The benefit seems to come from constraining activation drift in general, not from this particular direction.

  3. An alternative explanation is not ruled out. The helpful-only references are trained on general instruction data. Further instruction tuning may move toward them simply because of data overlap, not reversion. A control reference trained on helpful plus safety data would help separate the two.

  4. Missing comparison with related defenses. The block objective is close to existing methods that constrain fine-tuning along the alignment or refusal direction [3, 4, 5]. Without comparing to them, it is unclear whether the reversion view gives a better defense.

References

[1] Language Models Resist Alignment: Evidence From Data Compression. ACL 2025.

[2] Continual Safety Alignment via Gradient-Based Sample Selection. Findings of ACL 2026.

[3] Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models. NeurIPS 2024.

[4] AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basin. AAAI 2026.

[5] Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint. arXiv 2025.

Questions for authors

Please refer to the Weaknesses.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Reply to Reviewer ZqBm's comment

Thank you for the follow-up, for the updated score, and for withdrawing the ethics flag. We hope the analysis below addresses the reviewer's remaining concern about the block.

The lower harmfulness does not come from degraded generation. We agree with the reviewer's observation: in block seed 1, most long completions to harmful prompts repeat one sentence, usually a refusal. The data, however, do not support the conclusion that the block degrades generation. As already reported in our general comment, the block raises IFEval above ordinary fine-tuning and leaves held-out task NLL unchanged, in every seed including seed 1; on benign prompts, ordinary fine-tuning repeats itself more often than the block; and the block's lower harmfulness holds on the completions that do not repeat. In detail:

1. Instruction following and task fit (θS\theta_S: IFEval 74.9):

IFEval (prompt-level strict), seeds 0 / 1 / 2held-out NLL, seeds 0 / 1 / 2
ordinary fine-tuning47.7 / 29.2 / 27.91.374 / 1.416 / 1.425
block59.0 / 47.1 / 54.01.390 / 1.378 / 1.445

2. On benign prompts, the repetition comes from the fine-tuning recipe, not from the block. On the IFEval prompts, ordinary fine-tuning already repeats itself in 28–52% of its responses (θS\theta_S: 7%). The block repeats less often in seeds 0 and 1 (26% against 41%, and 36% against 52%) and about as often in seed 2 (29% against 28%). A response counts as repetitive when fewer than half of its lines, or fewer than 30% of its words, are distinct.

3. Harmfulness without the repetitive completions. Removing the repetitive completions takes out mostly refusals, so it raises the block's unsafe rate. Even so, block seed 1 stays well below ordinary fine-tuning of the same seed: among non-repetitive completions, 36.1% against 52.6% unsafe on HarmBench and 12.8% against 32.3% on StrongREJECT (HarmBench classifier), and 8.3% against 29.5% on BeaverTails (Llama Guard). Seeds 0 and 2 repeat on harmful prompts within the range of ordinary fine-tuning (1–17% of completions, against 6–32%) and show the same reduction (HarmBench classifier: 25.5 and 3.5, against 51.0 and 64.5).

4. What the block is, and what it costs. The block is not proposed as a new safety technique. Together with push, it is an experiment that tests whether motion along vrevv_{\mathrm{rev}} causally preserves or degrades safety (Section 5: "the block is a diagnostic rather than a deployment safeguard"). At the canonical strength, this experiment has a cost that is not repetition: the model also refuses more benign but sensitive requests (OR-Bench-Hard). The causal link does not depend on that cost. The soft block, whose barrier is 146 times weaker, lowers harmfulness with less repetition than ordinary fine-tuning, a higher IFEval, and the over-refusal rate of θS\theta_S. At a 4× lower learning rate, the canonical block has almost no repetition and brings harmfulness and IFEval close to θS\theta_S, but still over-refuses. Means over three seeds:

repetitive or degenerate completions (BeaverTails, %)LGIFEvalOR-Bench-Hard refusal (%)
θS\theta_S0.02.474.929.2
ordinary fine-tuning, lr 2e-56.827.334.94.6
canonical block, lr 2e-514.611.453.463.9
soft block, lr 2e-54.49.542.127.1
ordinary fine-tuning, lr 5e-62.18.966.64.5
canonical block, lr 5e-60.61.672.547.1

We will report over-refusal and repetition alongside harmfulness in the revision.

Novelty. That safety degradation under fine-tuning is established, and we do not claim it as new. Our contribution is the direction of this degradation: fixed before the trajectories are observed, tested against a matched null, related quantitatively to harmfulness, and intervened on during training. As the reviewer notes among the strengths, this provides "a diagnostic tool that can be used to detect shifts in model alignment during post-training"; the revision will state it more directly.

AuthorsOpenReview ↗

Message by the Authors

Dear Reviewer,

Thank you again for your detailed review and the follow-up comment, which have helped us strengthen our paper. We have addressed each of your points individually in our rebuttal, including new results and additional evidence. As the discussion period ends soon, we would be grateful if you could let us know whether our responses resolve your concerns, or whether any points need further clarification. If you find our answers satisfactory, we would kindly ask you to consider updating your score. Thank you for your time and careful consideration. Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.