- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies why an LLM's behavior keeps changing after safety alignment has already reached its target. The authors separate two sources of this drift. First, the training objective may still prefer some safe responses over others after the safety target is met, so it keeps producing gradients. Second, even when the gradient becomes exactly zero, the optimizer's stored momentum can still move the parameters. They isolate the second source by intervening on the optimizer state at the same zero-gradient checkpoint. Removing or scaling the stored momentum changes how much the model moves, and changing its direction changes later safety and refusal behavior. Based on this, they propose resetting the momentum on zero-gradient steps, and show that it reduces over-refusal in their settings.
Critical strengths
- The question is relevant. Over-refusal and behavioral drift after safety training are practical problems.
- The intervention design is careful. Holding the model, data, reward and second moment fixed while changing only the first moment gives a clean way to isolate the optimizer effect.
- The paper is clear about the scope of its causal claim and reports the safety cost of its intervention.
Critical weaknesses
-
The paper is very hard to read. The writing is dense, the notation is heavy, and most of the necessary details sit in the appendix. It takes a lot of effort to follow the main argument and to see what is actually being claimed.
-
The core observations are not new. It is already known that the standard DPO loss never gives an exactly zero gradient, which motivated alternatives with a finite target margin [1]. It is also known that group-normalized advantages become zero when all rollouts get the same reward, and this is already handled by filtering such groups [2]. That Adam's momentum keeps moving the parameters after the gradient vanishes is a basic property of the optimizer, and the authors acknowledge this themselves. The effects of optimizer state, and of resetting it, have also been studied in reinforcement learning [3]. What remains is a controlled demonstration of known effects in a safety setting.
-
The setting is unrealistic. The main experiments train with a safety-only reward and no KL term, which pushes the model into extreme over-refusal. Real alignment pipelines include KL or utility terms. The authors state that their causal identification no longer holds once such terms are active. This makes the practical relevance of the findings very limited.
References
[1] A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS 2024.
[2] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS 2025.
[3] Resetting the Optimizer in Deep RL: An Empirical Study. NeurIPS 2023.
Questions for authors
Please refer to the Weaknesses.
4 comments
Follow-up: results with a helpfulness reward
In our earlier replies we mentioned helpfulness rewards as future work. We have since run this setting and wanted to share the results.
With a helpfulness reward on benign prompts (our Mixed condition, no KL), the total gradient remained nonzero in these evaluated runs, so the PAC gate as defined in the paper never fired. To still test the idea, we applied it to the harmful-prompt part of the gradient only. At each step we compute the harmful-prompt and benign-prompt gradients separately with two backward passes on the same batch. On Mistral's first four steps, the relative discrepancy between their sum and the full gradient was 1.1–1.4%, compared with 1.1–1.3% between repeated full-batch passes. We keep a running copy m_harm of the harmful-prompt part, using the same β1 and the clipping coefficient computed for the total gradient. Once ‖g_harm‖ has stayed at or below 0.1× its mean over the first five steps with nonzero g_harm for two consecutive steps, we subtract m_harm from Adam's first moment on each such quiet step and reset m_harm to zero. There is no separate attainment criterion. Other optimizer states continue their normal updates without direct intervention.
CISPO, Mixed, step 90 (seed 1/seed 2). ORR is from the official GPT-4o grader, as before.
| Model | Condition | HarmBench ASR ↓ | OR-Bench-Hard ORR ↓ | Health-ORSC SCR ↑ |
|---|---|---|---|---|
| Mistral | No intervention | .045/.105 | .768/.643 | .676/.586 |
| Harmful-share m removed | .130/.145 | .598/.550 | .607/.533 | |
| Qwen | No intervention | .150/.015 | .482/.912 | .630/.632 |
| Harmful-share m removed | .110/.055 | .430/.475 | .622/.600 | |
| Llama | No intervention | .075/.050 | .477/.573 | .628/.545 |
| Harmful-share m removed | .075/.190 | .200/.295 | .606/.573 |
ORR went down in all six cases, but this came at a cost in safety: HarmBench ASR went up in four of six (one tie, one decrease) and SCR went down in five of six.
We ran the same check for DPO with a shared optimizer, removing the safety-objective share of the first moment on steps where the hinge term had zero gradient (step 91, seed 1/seed 2):
| Model | Condition | HarmBench ASR ↓ | OR-Bench-Hard ORR ↓ | Health-ORSC SCR ↑ |
|---|---|---|---|---|
| Mistral | No intervention | .165/.155 | .282/.395 | .676/.681 |
| Safety-objective m removed | .195/.195 | .228/.242 | .658/.656 | |
| Zephyr | No intervention | .195/.200 | .210/.233 | .616/.632 |
| Safety-objective m removed | .330/.290 | .145/.130 | .597/.593 |
The picture is the same: lower ORR, but higher ASR and lower SCR.
We also tried removing only half of the component (α=.5) for CISPO on Qwen and Llama and for both DPO models. This did not help consistently. Only one run (Qwen, seed 1) was better than its control on all three metrics.
So the optimizer state still shapes refusal behavior when other objectives keep the gradient nonzero, but these component-wise first-moment interventions did not consistently improve the overall safety–utility trade-off. We will include these results, the per-step training curves for all conditions, and the details of the method in the appendix.
Response to authors
I thank the authors for the honest response.
The authors agree that the main principles and optimizer resets are already known. The new results also show that the method does not help in realistic settings. With a KL term, the reset reduces over-refusal but increases harmful compliance. With a helpfulness reward, the gate never activates, and the modified version does not improve the safety-utility trade-off. PAC also shows no clear advantage over early stopping.
The evidence remains thin, and the promised restructuring cannot be assessed in the current version. I will keep my score.
Thank you for the feedback. We have uploaded a revised version with the updated title and reorganized the main text to present the behavioral results before the analysis. The revision also makes clear that the interventions do not consistently improve the safety–utility trade-off or outperform early stopping.
Thank you for your comments on the contribution and presentation. We will distinguish known principles from our empirical findings and organize the manuscript around the behavioral results.
W1. The presentation is difficult, and key explanations are in the appendix.
Our intention was to use the objective and optimizer equations to explain why updates can continue after a safety criterion is met, through residual objective pressure and the stored first moment. We then placed the experiments afterward to test whether these mechanisms translate into behavioral changes.
However, placing the behavioral results too late and key experiments in the appendix made it difficult to see which phenomenon the equations explain. We will present an over-refusal example and the same-checkpoint interventions first, followed by the equations explaining them.
If title changes are permitted, we would like to adopt “Safety Attainment Does Not Ensure Behavioral Stability: Objective and Optimizer Sources of Behavioral Drift after Safety Alignment.” We will review the terminology in the main text accordingly to make the distinction between objective stationarity and behavioral stability explicit.
W2. Are the relevant principles and optimizer resets already known?
We agree. We will explicitly acknowledge IPO’s finite target margin, DAPO’s treatment of equal-reward groups, Asadi et al.’s full and individual moment resets, and the RLVR optimizer comparisons in Do We Need Adam?
Our question concerns more than training failure: does the stored optimizer state contribute to increasing refusal of benign prompts after safety alignment?
In a same-checkpoint experiment, retaining the first moment increased ORR to .868 during the subsequent zero-gradient interval, whereas repeatedly clearing it kept ORR at .640–.652. This supports a contribution of the first moment to over-refusal growth in that setting.
The component-wise intervention results are below (Mistral, one run per condition).
The contribution is therefore the controlled examination of optimizer-state effects on over-refusal and substantive-response degradation, rather than the introduction of resetting itself. We examine residual objective pressure separately and do not attribute all behavioral degradation to the first moment.
SGD was also a viable alternative. On Qwen, its official ORR was lower than PAC’s (.727 versus .907), whereas Mistral Health SCR varied across seeds (.167/.462). We do not claim universal superiority of AdamW or PAC.
W3. The safety-only, no-KL setting differs from practical alignment.
The original exact-zero setting is a restricted condition for separating mechanisms. We extended it with same-checkpoint branches under KL=.001. All three models use a threshold of .1 on the pre-clipping total gradient norm.
Results are after 20 branch steps, with rollout seeds 42/43. ORR and SCR improved in all three models, but HarmBench ASR increased. We chose τ=.1 using earlier runs and do not claim it as a general saturation detector.
In separate 90-step training comparisons, KL+reset improved ORR and SCR for both seeds of Qwen and Llama, while Mistral’s SCR effect was inconsistent. Due to disk-space constraints, we had to delete intermediate checkpoints after training finished and before the rebuttal period. For runs whose required intermediate checkpoints were no longer available, we re-trained them with the same configuration. We will report the full trajectories and external evaluations for all three models.
PAC showed no consistent advantage over early stopping. Experiments with helpfulness rewards remain future work; our current conclusions are limited to optimizer-state effects and their control in the tested settings.