- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
The paper discovers the effect of benign training in the post-processing on the safety alignment of the Large Language Models. The paper claims that non-adversarial post-training follows a reversion direction that makes the model more permissive. The methodology is to observe the change of the activations in the layers of post-trained model by comparing against many helpful trained versions using a probe family. The result of the the method is an activation-space direction which can be used as a basis to measure the drift even in benign tasks. The motivation of the paper and the discovery are interesting, yet the experiments can not provide enough evidence.
Critical strengths
- The topic is interesting in terms of safety and alignment. The paper offers a diagnostic tool that can be used to detect shifts in model alignment during post-training.
- The reversion is not tied to one architecture or model size.
- The formulation is simple and easy to understand
Critical weaknesses
- I had a really hard time reading the paper sections due to the verbose and long sentences. The story that the paper is trying to tell is actually simple, yet it is presented very complexly and with highly specific jargon.
- The implications and the discovery of the paper are not so surprising since it is well known that post-training can damage the safety alignment of the model. Especially when the paper presents it using two tasks and evaluates it with only one safety dataset, BeaverTails.
- The paper needs to explicitly compare the difference between reversal direction calculation and just observing the embedding drifts at the layers.
- The paper should diversify the model architectures and the tasks. It only measured small model sizes and used only two tasks.
- The experiment section and the figures seem AI-generated, and it is hard to understand.
Questions for authors
- How do the authors define a benign dataset and make sure that the samples in the dataset are honest and harmless?
- How many helpful proxy models are generated per direction calculation?
- What do the authors mean by the "null"? Is it the \theta_P, \theta_S, or \theta_H?
Reply to Reviewer ZqBm's comment
Thank you for the follow-up, for the updated score, and for withdrawing the ethics flag. We hope the analysis below addresses the reviewer's remaining concern about the block.
The lower harmfulness does not come from degraded generation. We agree with the reviewer's observation: in block seed 1, most long completions to harmful prompts repeat one sentence, usually a refusal. The data, however, do not support the conclusion that the block degrades generation. As already reported in our general comment, the block raises IFEval above ordinary fine-tuning and leaves held-out task NLL unchanged, in every seed including seed 1; on benign prompts, ordinary fine-tuning repeats itself more often than the block; and the block's lower harmfulness holds on the completions that do not repeat. In detail:
1. Instruction following and task fit (θS: IFEval 74.9):
2. On benign prompts, the repetition comes from the fine-tuning recipe, not from the block. On the IFEval prompts, ordinary fine-tuning already repeats itself in 28–52% of its responses (θS: 7%). The block repeats less often in seeds 0 and 1 (26% against 41%, and 36% against 52%) and about as often in seed 2 (29% against 28%). A response counts as repetitive when fewer than half of its lines, or fewer than 30% of its words, are distinct.
3. Harmfulness without the repetitive completions. Removing the repetitive completions takes out mostly refusals, so it raises the block's unsafe rate. Even so, block seed 1 stays well below ordinary fine-tuning of the same seed: among non-repetitive completions, 36.1% against 52.6% unsafe on HarmBench and 12.8% against 32.3% on StrongREJECT (HarmBench classifier), and 8.3% against 29.5% on BeaverTails (Llama Guard). Seeds 0 and 2 repeat on harmful prompts within the range of ordinary fine-tuning (1–17% of completions, against 6–32%) and show the same reduction (HarmBench classifier: 25.5 and 3.5, against 51.0 and 64.5).
4. What the block is, and what it costs. The block is not proposed as a new safety technique. Together with push, it is an experiment that tests whether motion along vrev causally preserves or degrades safety (Section 5: "the block is a diagnostic rather than a deployment safeguard"). At the canonical strength, this experiment has a cost that is not repetition: the model also refuses more benign but sensitive requests (OR-Bench-Hard). The causal link does not depend on that cost. The soft block, whose barrier is 146 times weaker, lowers harmfulness with less repetition than ordinary fine-tuning, a higher IFEval, and the over-refusal rate of θS. At a 4× lower learning rate, the canonical block has almost no repetition and brings harmfulness and IFEval close to θS, but still over-refuses. Means over three seeds:
We will report over-refusal and repetition alongside harmfulness in the revision.
Novelty. That safety degradation under fine-tuning is established, and we do not claim it as new. Our contribution is the direction of this degradation: fixed before the trajectories are observed, tested against a matched null, related quantitatively to harmfulness, and intervened on during training. As the reviewer notes among the strengths, this provides "a diagnostic tool that can be used to detect shifts in model alignment during post-training"; the revision will state it more directly.