acceptodds
Under review as a conference paper at ICLR 2027

Attainment Is Not Stationarity: Objective and Optimizer Sources of Behavioral Drift after Safety Alignment

Abstract

As large language models (LLMs) are increasingly deployed in real-world applications, maintaining useful behavior while achieving safety remains important. Yet model behavior can continue to change after a safety metric reaches its target. We show that this post-attainment drift has two separable sources: an objective can retain residual preference pressure after the target safety level is reached, and stored optimizer state can induce behaviorally consequential movement even after the current objective becomes exactly stationary. Preference controls and same-checkpoint optimizer-state interventions isolate these sources. Changing residual preference within the attained safe region changes substantive safe completion; removing the stored first moment eliminates silent-step movement, scaling it produces proportional displacement across three model families, and matched-magnitude directions lead to different safety and refusal trajectories. Thus, safety attainment does not imply objective stationarity, and objective stationarity does not imply parameter stationarity.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, safety alignment, preference optimization, post-attainment drift, optimizer state, AdamW, DPO, CISPO, over-refusal

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_gg8MOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies why an LLM's behavior keeps changing after safety alignment has already reached its target. The authors separate two sources of this drift. First, the training objective may still prefer some safe responses over others after the safety target is met, so it keeps producing gradients. Second, even when the gradient becomes exactly zero, the optimizer's stored momentum can still move the parameters. They isolate the second source by intervening on the optimizer state at the same zero-gradient checkpoint. Removing or scaling the stored momentum changes how much the model moves, and changing its direction changes later safety and refusal behavior. Based on this, they propose resetting the momentum on zero-gradient steps, and show that it reduces over-refusal in their settings.

Critical strengths
  1. The question is relevant. Over-refusal and behavioral drift after safety training are practical problems.
  2. The intervention design is careful. Holding the model, data, reward and second moment fixed while changing only the first moment gives a clean way to isolate the optimizer effect.
  3. The paper is clear about the scope of its causal claim and reports the safety cost of its intervention.
Critical weaknesses
  1. The paper is very hard to read. The writing is dense, the notation is heavy, and most of the necessary details sit in the appendix. It takes a lot of effort to follow the main argument and to see what is actually being claimed.

  2. The core observations are not new. It is already known that the standard DPO loss never gives an exactly zero gradient, which motivated alternatives with a finite target margin [1]. It is also known that group-normalized advantages become zero when all rollouts get the same reward, and this is already handled by filtering such groups [2]. That Adam's momentum keeps moving the parameters after the gradient vanishes is a basic property of the optimizer, and the authors acknowledge this themselves. The effects of optimizer state, and of resetting it, have also been studied in reinforcement learning [3]. What remains is a controlled demonstration of known effects in a safety setting.

  3. The setting is unrealistic. The main experiments train with a safety-only reward and no KL term, which pushes the model into extreme over-refusal. Real alignment pipelines include KL or utility terms. The authors state that their causal identification no longer holds once such terms are active. This makes the practical relevance of the findings very limited.

References

[1] A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS 2024.

[2] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS 2025.

[3] Resetting the Optimizer in Deep RL: An Empirical Study. NeurIPS 2023.

Questions for authors

Please refer to the Weaknesses.

4 comments

Thank you for your comments on the contribution and presentation. We will distinguish known principles from our empirical findings and organize the manuscript around the behavioral results.

W1. The presentation is difficult, and key explanations are in the appendix.

Our intention was to use the objective and optimizer equations to explain why updates can continue after a safety criterion is met, through residual objective pressure and the stored first moment. We then placed the experiments afterward to test whether these mechanisms translate into behavioral changes.

However, placing the behavioral results too late and key experiments in the appendix made it difficult to see which phenomenon the equations explain. We will present an over-refusal example and the same-checkpoint interventions first, followed by the equations explaining them.

If title changes are permitted, we would like to adopt “Safety Attainment Does Not Ensure Behavioral Stability: Objective and Optimizer Sources of Behavioral Drift after Safety Alignment.” We will review the terminology in the main text accordingly to make the distinction between objective stationarity and behavioral stability explicit.

W2. Are the relevant principles and optimizer resets already known?

We agree. We will explicitly acknowledge IPO’s finite target margin, DAPO’s treatment of equal-reward groups, Asadi et al.’s full and individual moment resets, and the RLVR optimizer comparisons in Do We Need Adam?

Our question concerns more than training failure: does the stored optimizer state contribute to increasing refusal of benign prompts after safety alignment?

In a same-checkpoint experiment, retaining the first moment increased ORR to .868 during the subsequent zero-gradient interval, whereas repeatedly clearing it kept ORR at .640–.652. This supports a contribution of the first moment to over-refusal growth in that setting.

The component-wise intervention results are below (Mistral, one run per condition).

InterventionInternal ASR ↓Internal ORR ↓Internal SCR ↑
No intervention.0001.00.004
Repeated first-moment reset.000.98.729
One-time first-moment reset.000.98.716
Second-moment reset.000.98.000
Reset both moments.0001.00.000
Full reset.0001.00.000

The contribution is therefore the controlled examination of optimizer-state effects on over-refusal and substantive-response degradation, rather than the introduction of resetting itself. We examine residual objective pressure separately and do not attribute all behavioral degradation to the first moment.

SGD was also a viable alternative. On Qwen, its official ORR was lower than PAC’s (.727 versus .907), whereas Mistral Health SCR varied across seeds (.167/.462). We do not claim universal superiority of AdamW or PAC.

W3. The safety-only, no-KL setting differs from practical alignment.

The original exact-zero setting is a restricted condition for separating mechanisms. We extended it with same-checkpoint branches under KL=.001. All three models use a threshold of .1 on the pre-clipping total gradient norm.

ModelInterventionHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralRetain.000/.0001.000/1.000.118/.140
Reset.010/.020.922/.932.520/.530
QwenRetain.000/.000.980/.988.434/.425
Reset.010/.010.853/.858.601/.592
LlamaRetain.000/.000.993/.998.069/.047
Reset.030/.015.647/.828.409/.350

Results are after 20 branch steps, with rollout seeds 42/43. ORR and SCR improved in all three models, but HarmBench ASR increased. We chose τ=.1 using earlier runs and do not claim it as a general saturation detector.

In separate 90-step training comparisons, KL+reset improved ORR and SCR for both seeds of Qwen and Llama, while Mistral’s SCR effect was inconsistent. Due to disk-space constraints, we had to delete intermediate checkpoints after training finished and before the rebuttal period. For runs whose required intermediate checkpoints were no longer available, we re-trained them with the same configuration. We will report the full trajectories and external evaluations for all three models.

PAC showed no consistent advantage over early stopping. Experiments with helpfulness rewards remain future work; our current conclusions are limited to optimizer-state effects and their control in the tested settings.

AuthorsOpenReview ↗

Follow-up: results with a helpfulness reward

In our earlier replies we mentioned helpfulness rewards as future work. We have since run this setting and wanted to share the results.

With a helpfulness reward on benign prompts (our Mixed condition, no KL), the total gradient remained nonzero in these evaluated runs, so the PAC gate as defined in the paper never fired. To still test the idea, we applied it to the harmful-prompt part of the gradient only. At each step we compute the harmful-prompt and benign-prompt gradients separately with two backward passes on the same batch. On Mistral's first four steps, the relative discrepancy between their sum and the full gradient was 1.1–1.4%, compared with 1.1–1.3% between repeated full-batch passes. We keep a running copy m_harm of the harmful-prompt part, using the same β1 and the clipping coefficient computed for the total gradient. Once ‖g_harm‖ has stayed at or below 0.1× its mean over the first five steps with nonzero g_harm for two consecutive steps, we subtract m_harm from Adam's first moment on each such quiet step and reset m_harm to zero. There is no separate attainment criterion. Other optimizer states continue their normal updates without direct intervention.

CISPO, Mixed, step 90 (seed 1/seed 2). ORR is from the official GPT-4o grader, as before.

ModelConditionHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralNo intervention.045/.105.768/.643.676/.586
Harmful-share m removed.130/.145.598/.550.607/.533
QwenNo intervention.150/.015.482/.912.630/.632
Harmful-share m removed.110/.055.430/.475.622/.600
LlamaNo intervention.075/.050.477/.573.628/.545
Harmful-share m removed.075/.190.200/.295.606/.573

ORR went down in all six cases, but this came at a cost in safety: HarmBench ASR went up in four of six (one tie, one decrease) and SCR went down in five of six.

We ran the same check for DPO with a shared optimizer, removing the safety-objective share of the first moment on steps where the hinge term had zero gradient (step 91, seed 1/seed 2):

ModelConditionHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralNo intervention.165/.155.282/.395.676/.681
Safety-objective m removed.195/.195.228/.242.658/.656
ZephyrNo intervention.195/.200.210/.233.616/.632
Safety-objective m removed.330/.290.145/.130.597/.593

The picture is the same: lower ORR, but higher ASR and lower SCR.

We also tried removing only half of the component (α=.5) for CISPO on Qwen and Llama and for both DPO models. This did not help consistently. Only one run (Qwen, seed 1) was better than its control on all three metrics.

So the optimizer state still shapes refusal behavior when other objectives keep the gradient nonzero, but these component-wise first-moment interventions did not consistently improve the overall safety–utility trade-off. We will include these results, the per-step training curves for all conditions, and the details of the method in the appendix.

Reviewer_gg8MOpenReview ↗

Response to authors

I thank the authors for the honest response.

The authors agree that the main principles and optimizer resets are already known. The new results also show that the method does not help in realistic settings. With a KL term, the reset reduces over-refusal but increases harmful compliance. With a helpfulness reward, the gate never activates, and the modified version does not improve the safety-utility trade-off. PAC also shows no clear advantage over early stopping.

The evidence remains thin, and the promised restructuring cannot be assessed in the current version. I will keep my score.

Thank you for the feedback. We have uploaded a revised version with the updated title and reorganized the main text to present the behavioral results before the analysis. The revision also makes clear that the interventions do not consistently improve the safety–utility trade-off or outperform early stopping.

Reviewer_Ue3sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks why model behavior keeps changing after a safety metric has already hit its target. It separates two causes. First, the objective can still prefer some safe responses over others after attainment. Second, even when the current gradient is exactly zero, AdamW's stored first moment can keep moving the parameters. For the first, the authors use a finite-margin (hinge) version of DPO and secondary reward orderings within the maximal-safety set in CISPO. For the second, they branch from a shared zero-gradient checkpoint and change only the first moment: remove it, scale it, or change its direction. They also propose PAC, an online rule that zeroes the first moment on silent steps. Experiments cover DPO and CISPO on several 7–8B model families, with external safety and over-refusal benchmarks.

Critical strengths
  1. The best experiment in the paper is the earlier-checkpoint branch in App. H.2. The steps right after the branch are all silent, and in that window ORR rises from about 0.64 to 0.868 with the inherited first moment, while repeated mm removal keeps it at 0.640–0.652. So there is a real behavioral effect, not just the (unsurprising) fact that Adam can move the weights when gt=0g_t = 0.

  2. The optimizer-state controls are done carefully. The main intervention changes only mm and keeps the weights, vv and the step counter fixed. The direction experiment matches the size of the first preconditioned update across arms, and App. I.3 deals properly with FP32/BF16 precision when measuring small parameter changes. These choices make me trust the local causal interpretation.

  3. The paper is honest about its limitations and trade-offs. It separates removing only mm from resetting vv or the whole optimizer (Table 13), says clearly that the Mixed condition changes reward and training data at the same time, and reports that external safety gets worse when utility improves (Tables 8 and 19).

Critical weaknesses

W1. How much does the exact-silence regime matter? The cleanest optimizer-side result depends on the total gradient being exactly zero, which here happens in the safety-only, harmful-only CISPO setting. Adding a KL term with λKL=0.001\lambda_{\mathrm{KL}} = 0.001 removes all silent steps in both reported runs, compared with 70, 75 and 84 silent steps out of 90 without KL (Table 21). The authors acknowledge this, and it doesn't make the controlled result wrong. But the paper doesn't show that this diagnosis changes much about how we understand safety alignment, or how we do it in practice. The KL runs already exist, so reporting their behavioral results would help a lot: does KL itself prevent the drift, or does it just remove the condition PAC needs to trigger?

Related to this, I'd be careful with the term "objective stationarity". In these CISPO runs, gt=0g_t = 0 comes from every rollout in the batch getting the maximum reward, so all Ai=0A_i = 0 (Eq. 4; Table 12). That is a zero gradient on a sampled batch, not a stationary point of the expected objective. This doesn't affect the immediate intervention result, but it does limit how far the headline claim can be pushed.

W2. Key details of the setup are missing. Sec. 3.1 says the details are in the appendix, but App. E doesn't tell me where the training prompts come from or how many there are, the effective batch size, what the internal judge is, how the reward is computed, or the decoding settings. These aren't side details here. In these runs, silent steps line up exactly with steps where every rollout gets the maximum reward (Table 12), so how often silence happens depends directly on the batch size, the judge, and the sampling setup. The same choices also shape what "safety attainment" means. Without them, I can't really tell how the reported regime comes about. On the evaluation side, Health-ORSC (Tables 4 and 20) is never defined or cited, and HarmBench, StrongREJECT and SORRY-Bench are not cited either.

W3. It's unclear whether PAC beats simple baselines like early stopping. PAC does improve utility over just continuing to train. But HarmBench ASR goes from 0 to 0.035, 0.017 and 0.010 on the three models, and over-refusal under the native OR-Bench grader is still high: 0.952, 0.906 and 0.863, vs. 0.980, 1.000 and 0.992 without PAC (Tables 19 and 20). The paper is upfront about this trade-off. What I can't tell is whether PAC gives a better operating point than simply keeping an earlier checkpoint, or than another simple way to stop silent updates, e.g., skipping the optimizer step whenever the gradient is exactly zero. The paper's own numbers make me suspect early stopping is hard to beat. In Table 18, the step-2 checkpoints for Qwen and Llama already have lower internal ORR (about 0.10–0.13) and higher SCR than the PAC runs reach at step 90 (where ORR is 0.20–0.47). ASR at step 2 isn't reported, so this isn't conclusive, but a small comparison with existing checkpoints would answer it directly.

Questions for authors

Q1. Could you give the full training and evaluation recipe? In particular: effective batch size, training data sources and sizes, how the reward and the internal judge are defined, decoding settings, and whether Qwen2.5-7B and Llama-3.1-8B are base or instruct models. Which of these choices are needed to get the high silent-step fractions in Table 12? If this is in a supplement I missed, please point me to it.

Q2. Does PAC reach a better external safety–utility point than plain early stopping? Even one model with two seeds, using saved checkpoints, would show whether training on with PAC is better than just keeping a checkpoint that is already good enough.

Q3. What do the finished λKL=0.001\lambda_{\mathrm{KL}} = 0.001 runs look like behaviorally? Does KL reduce over-refusal or the SCR collapse, or does it only remove the exact-zero condition that PAC relies on?

Q4. In the joint arm of App. D, the utility gradient is still nonzero, but Eq. 11 only fires when the total gradient is zero. What exactly does the optimizer intervention do in that arm? Also, why do the optimizer-only DPO results differ between Table 10 and Table 17? For example, Zephyr's ASRmin⁡\mathrm{ASR}_{\min} is 0.007–0.011 in Table 10 but 0.000–0.004 in Table 17. Are the seeds, branch states or intervention rules different?

3 comments

Thank you for the careful review. We agree that a zero gradient on one sampled batch must be distinguished from stationarity of the expected objective.

If title changes are permitted, we would like to adopt “Safety Attainment Does Not Ensure Behavioral Stability: Objective and Optimizer Sources of Behavioral Drift after Safety Alignment.” We will review the corresponding terminology throughout the manuscript and distinguish safety-threshold attainment, zero batch gradients, and objective stationarity.

Q1. What are the training settings and the source of silence?

ItemDPOCISPO
ModelsMistral, ZephyrMistral-7B-Instruct-v0.2, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct
Data732 fixed training units generated by MistralPKU-based pool: 2,095 training / 232 validation prompts
Training signalSafety and utility preference pairsJudge rewards on each policy’s rollouts
Zero-gradient conditionA hinge-loss term has zero gradient once its preference constraint is satisfiedEqual rewards within a group produce zero advantages

The DPO dataset is shared between the two models and is off-policy for Zephyr. This does not mean that standard DPO generally has zero gradients at a finite margin.

CISPO performs one optimizer update per batch of 16 prompts × 16 rollouts. The default setting is full fine-tuning, LR 2×10−62\times10^{-6}, 3% linear warmup, and 90 steps. Training uses temperature 1.0; evaluation uses greedy decoding.

The safety-only reward is s×min⁡(1,(2300/token count)2)s\times\min(1,(2300/\text{token count})^2). The judge is Qwen3.5-397B-A17B. Harm-axis weights are illegal .30, sensitiveness .25, hate .20, insult .15, adult .10, and bias .10. With axis scores from 0 to 2, s=max⁡(0,1−∑iwiai/2)s=\max(0,1-\sum_i w_i a_i/2); an overall harm rating of 2 overrides the score to zero.

Equal rewards do not imply maximal rewards. In the new runs, 74 of 75 silent steps had maximal rewards for every rollout. Some runs also returned to nonzero gradients after their first silent batch.

Health-ORSC evaluates benign borderline health questions. Using Qwen3.8, we report the fraction judged Safe with helpfulness category C or D. We distinguish this from internal SCR on harmful prompts.

Q2. How do early stopping and optimizer-step skipping compare?

Early stopping is a valid alternative. At step 2, Qwen and Llama reached HarmBench ASR .060–.110 with official ORR .19–.30. Within-run comparisons across all three models show lower HarmBench ASR at step 90, but better ORR and SCR at early stopping. PAC showed no consistent advantage over retrospectively selected checkpoints, including in attack evaluations.

The step-90 external results for skip and PAC are below (training seeds 1/2).

ModelMethodHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralSkip.000/.000.998/1.000.151/.149
PAC.015/.055.958/.938.433/.459
QwenSkip.010/.005.990/.980.480/.378
PAC.030/.005.833/.980.589/.491
LlamaSkip.010/.000.928/.995.254/.072
PAC.015/.005.787/.938.345/.241

PAC had higher SCR and lower or equal ORR, but higher HarmBench ASR in some conditions. Skipping an optimizer step also suspends both moment updates, the step counter, and weight decay. We therefore distinguish this baseline from a causal intervention on the first moment alone.

Q3. Does KL prevent behavioral degradation?

KL=.001 alone left high external over-refusal in all three models. Below are the step-90 results for two seeds. Due to disk-space constraints, we had to delete intermediate checkpoints after training finished and before the rebuttal period. For runs whose required intermediate checkpoints were no longer available, we re-trained them with the same configuration.

Model and conditionOR-Bench-Hard ORR ↓Health-ORSC SCR ↑
Mistral KL1.000/1.000.093/.380
Mistral KL+reset.958/.990.491/.320
Qwen KL.978/.995.453/.389
Qwen KL+reset.892/.948.595/.511
Llama KL1.000/1.000.000/.013
Llama KL+reset.845/.870.283/.287

Reset uses total gradient norm ≤.1 for all three models. Both Qwen and Llama seeds improved in ORR and SCR, whereas Mistral’s SCR effect differed by seed. HarmBench ASR also increased in some conditions, so we do not claim consistent improvement across all metrics. The external-evaluation figures report all three metrics together.

Q4. How should the joint arm and Tables 10/17 be interpreted?

The joint arm uses separate AdamW optimizers for safety and utility. It clears only the safety optimizer’s first moment when the safety gradient is zero; the utility optimizer is not modified. We will distinguish this rule from the total-gradient gate.

Table 10 applies the intervention from step 0, whereas Table 17 applies it after branching at step 46. We will state these different intervention timings in the captions.

Thank you for the clarifications and additional experiments. Your responses to Q1 and Q4 resolve my setup and optimizer questions (W2, Q4), and I appreciate the planned title and terminology revisions.

However, the results in Q2 and Q3 reinforce my core concerns (W1, W3). First, PAC shows no consistent advantage over early stopping: step-90 OR-Bench-Hard ORR stays very high (0.787–0.980), and gains over step skipping trade off HarmBench ASR in several runs. Second, KL=.001 eliminates exact silence while severe over-refusal persists, and norm-threshold resets (<=.1) still leave ORR at 0.845–0.990. Given this narrow practical scope, I maintain my score of 2.

Thank you for the feedback. We have uploaded a revised version with the updated title and clearer discussion of the results. We now state explicitly that PAC does not consistently outperform early stopping, and that the KL reset results involve trade-offs between over-refusal and harmful compliance.

Reviewer_2YtCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies behavioral drift after safety attainment in LLM alignment. The authors distinguish two potential sources: (1) residual preference pressure from the objective after the safety target is reached, and (2) parameter updates caused by the stored optimizer state even when the current gradient is zero. The paper investigates these effects using DPO and CISPO through objective-side controls and same-checkpoint optimizer-state interventions. It also proposes Post-Attainment Control (PAC), which removes the stored first moment during zero-gradient steps. Experiments show that PAC reduces over-refusal and improves substantive safe completion compared with safety-only baselines.

Critical strengths

Useful ablation studies and empirical results across multiple settings

The paper evaluates its interventions using DPO and CISPO across multiple model families. The ablation studies on the magnitude and direction of the first moment provide additional evidence for the optimizer-state effect. External evaluations also help assess the improvements in over-refusal and safe completion, as well as their safety costs.

Critical weaknesses

1. Insufficient details on training data and reward functions

The experimental setup lacks important details about the training datasets and score functions. The sources and construction of harmful/benign training data, the generation of DPO preference pairs, and the definitions of CISPO safety and helpfulness scores, s(x,y)s(x,y) and h(x,y)h(x,y), are unclear. The internal evaluation judge is also not sufficiently described. These details are essential for reproducibility and for understanding the exact zero-gradient regime studied in the paper.

2. Limited generality of the exact zero-gradient setting

The optimizer-side analysis focuses on exact zero-gradient steps under safety-only objectives, where all rollouts can receive the same maximum safety reward. However, Appendix M shows that adding even a small KL penalty eliminates these zero-gradient steps. Since PAC only acts on zero-gradient steps, its applicability to more general alignment settings with KL regularization or additional reward terms remains unclear.

Questions for authors

1. Training data and preference pairs

Could the authors provide more details about the sources, sizes, and construction of the harmful and benign training datasets? In particular, how are the safety and utility preference pairs for DPO generated?

2. Reward functions and evaluation

How exactly are the safety and helpfulness scores, s(x,y)s(x,y) and h(x,y)h(x,y), computed in CISPO? Are they obtained from pretrained reward models, LLM judges, human judges, or other scoring methods? Also, how is the internal judge for ASR, ORR, and SCR implemented?

3. Generality beyond exact zero-gradient steps

Appendix M shows that adding a KL penalty eliminates the zero-gradient steps. Could the authors investigate whether optimizer-state-induced behavioral drift remains significant when the gradient is small but nonzero? It would also be useful to evaluate how frequently exact zero-gradient steps occur with alternative reward functions or KL regularization.

4. Comparison with early stopping

Since PAC mainly prevents further updates during zero-gradient steps, could the authors compare it with early stopping at safety attainment or selecting the best checkpoint? Such comparisons would help clarify whether PAC provides additional benefits in terms of safety and utility.

2 comments

Thank you for your comments on reproducibility and scope. We clarify the data construction, rewards, and evaluation settings below.

Q1. How were the data and preference pairs constructed?

For DPO, Mistral and Zephyr use the same fixed dataset of 732 training units generated by Mistral-7B-Instruct-v0.2. The data are therefore off-policy for Zephyr.

ObjectivePreference pair
SafetySafe response > harmful response
Utility on harmful promptsSafe, substantive response > over-refusal
Utility on benign promptsHelpful response > inadequate response

Prompts come from PKU-SafeRLHF. Mistral generates the responses, and Qwen3.5-397B-A17B judges them. There are 705 unique benign prompts, of which 27 are used twice.

The safety-only CISPO pool contains 2,327 PKU prompts on which Mistral produced harmful responses: 2,095 for training and 232 for validation. All three models use the same prompt pool, with rollouts generated by their respective policies. We will acknowledge that a failure pool selected using Mistral may be easier for other models.

Q2. What are the reward and internal judge?

We used several reward variants; here we describe the representative one. The training judge is Qwen3.5-397B-A17B with temperature 0. It scores each response on six harm axes, each from 0 to 2, with weights illegal .30, sensitiveness .25, hate .20, insult .15, adult .10, and bias .10.

The safety score is s=max⁡(0,1−∑iwiai/2)s=\max(0,1-\sum_i w_i a_i/2), overridden to zero when the overall harm rating is 2. The length factor is ℓ(y)=min⁡(1,(2300/token count)2)\ell(y)=\min(1,(2300/\text{token count})^2).

ConditionReward
Safety-onlys×ℓ(y)s\times\ell(y)
Harmful prompts in the mixed conditions×substantive×ℓ(y)s\times\mathrm{substantive}\times\ell(y)
Benign prompts in the mixed condition(1−declined)×helpfulness×ℓ(y)(1-\mathrm{declined})\times\mathrm{helpfulness}\times\ell(y)

Substantive and declined are binary judgments. Helpfulness is scored from 0 to 10 and normalized to [0,1].

Internal evaluation uses 450 harmful and 250 benign prompts with greedy decoding. ASR is the unsafe-response rate on harmful prompts; ORR is the refusal rate on benign prompts; SCR is the rate of safe, substantive responses on harmful prompts.

For external evaluation, we distinguish the official HarmBench classifier, GPT-4o grading for OR-Bench-Hard, and Qwen3.8 judging for Health-ORSC. Health-ORSC uses benign borderline health questions and therefore differs from internal SCR in both prompts and judge.

Q3. Is there an effect with small nonzero gradients?

We added same-checkpoint branches with KL=.001. All three models use τ=.1 on the pre-clipping L2 norm of the total gradient. Results are after 20 branch steps, with rollout seeds 42/43.

ModelInterventionHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralRetain.000/.0001.000/1.000.118/.140
Reset, τ=.1.010/.020.922/.932.520/.530
QwenRetain.000/.000.980/.988.434/.425
Reset, τ=.1.010/.010.853/.858.601/.592
LlamaRetain.000/.000.993/.998.069/.047
Reset, τ=.1.030/.015.647/.828.409/.350

ORR decreased and SCR increased in all three models, but HarmBench ASR also increased. The threshold was chosen using earlier runs; its generalization across models and settings has not been established.

Exact zero-gradient steps are frequent without KL (70, 75 and 84 of 90 steps; Table 21) but absent with KL=.001. Zero gradients also do not necessarily imply maximal rewards: in the new runs, 74 of 75 silent steps had maximal rewards for every rollout; one did not. We do not claim a universal frequency of silence across rewards and batch compositions.

Q4. Is PAC better than early stopping?

We did not observe a consistent advantage. In the within-run comparisons for all three models, early stopping yielded better external ORR and Health SCR than step 90, but higher HarmBench ASR. Due to disk-space constraints, we had to delete intermediate checkpoints after training finished and before the rebuttal period. For runs whose required intermediate checkpoints were no longer available, we re-trained them with the same configuration.

PAC also showed no consistent advantage in attack evaluations against retrospectively selected checkpoints. We will label this selection as an oracle comparison using external results and distinguish it from an operational stopping rule. We will describe PAC as an intervention for identifying and controlling optimizer-state effects rather than an optimal stopping strategy.

AuthorsOpenReview ↗

Follow-up: results with a helpfulness reward

In our earlier replies we mentioned helpfulness rewards as future work. We have since run this setting and wanted to share the results.

With a helpfulness reward on benign prompts (our Mixed condition, no KL), the total gradient remained nonzero in these evaluated runs, so the PAC gate as defined in the paper never fired. To still test the idea, we applied it to the harmful-prompt part of the gradient only. At each step we compute the harmful-prompt and benign-prompt gradients separately with two backward passes on the same batch. On Mistral's first four steps, the relative discrepancy between their sum and the full gradient was 1.1–1.4%, compared with 1.1–1.3% between repeated full-batch passes. We keep a running copy m_harm of the harmful-prompt part, using the same β1 and the clipping coefficient computed for the total gradient. Once ‖g_harm‖ has stayed at or below 0.1× its mean over the first five steps with nonzero g_harm for two consecutive steps, we subtract m_harm from Adam's first moment on each such quiet step and reset m_harm to zero. There is no separate attainment criterion. Other optimizer states continue their normal updates without direct intervention.

CISPO, Mixed, step 90 (seed 1/seed 2). ORR is from the official GPT-4o grader, as before.

ModelConditionHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralNo intervention.045/.105.768/.643.676/.586
Harmful-share m removed.130/.145.598/.550.607/.533
QwenNo intervention.150/.015.482/.912.630/.632
Harmful-share m removed.110/.055.430/.475.622/.600
LlamaNo intervention.075/.050.477/.573.628/.545
Harmful-share m removed.075/.190.200/.295.606/.573

ORR went down in all six cases, but this came at a cost in safety: HarmBench ASR went up in four of six (one tie, one decrease) and SCR went down in five of six.

We ran the same check for DPO with a shared optimizer, removing the safety-objective share of the first moment on steps where the hinge term had zero gradient (step 91, seed 1/seed 2):

ModelConditionHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralNo intervention.165/.155.282/.395.676/.681
Safety-objective m removed.195/.195.228/.242.658/.656
ZephyrNo intervention.195/.200.210/.233.616/.632
Safety-objective m removed.330/.290.145/.130.597/.593

The picture is the same: lower ORR, but higher ASR and lower SCR.

We also tried removing only half of the component (α=.5) for CISPO on Qwen and Llama and for both DPO models. This did not help consistently. Only one run (Qwen, seed 1) was better than its control on all three metrics.

So the optimizer state still shapes refusal behavior when other objectives keep the gradient nonzero, but these component-wise first-moment interventions did not consistently improve the overall safety–utility trade-off. We will include these results, the per-step training curves for all conditions, and the details of the method in the appendix.

Reviewer_jDQPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authors ask why model behavior keeps changing after safety training has hit its target. The authors argue there are two separate causes. First, the objective may still push the model after the safety target is reached: standard DPO never gives an exactly zero gradient, and in CISPO any leftover reward ordering among safe responses keeps shaping behavior. Second, even when the gradient is exactly zero, AdamW’s stored first moment keeps moving the weights. To separate the two, they use hinge-loss DPO and a safety-only CISPO setup where most steps have zero gradient (“silent steps”), then branch from one checkpoint and change only the first moment. Removing it stops the movement, scaling it scales the movement, and different directions lead to different refusal and safety outcomes. They also propose PAC, which zeroes the first moment on silent steps. It reduces over-refusal in three 7–8B models at a small safety cost.

Critical strengths

Clear framing: Clear conceptual separation of three states that the literature often conflates. The three-state formulation in Eq. 1 is a nice, communicable framing.

Strong causal controls: Same-checkpoint interventions isolate AdamW's stored first moment while holding weights, second moment, schedule, and other relevant conditions fixed.

Diverse evalutaion setup: Experiments cover Mistral, Qwen, Llama, and Zephyr across DPO and CISPO. In addition HarmBench, OR-Bench, XSTest-based evaluation, and Health-ORSC provide evidence beyond the internal judge.

Critical weaknesses
  1. The main findings seem somewhat expected from the optimization equations. For example, it is already known that AdamW can continue updating parameters even when the current gradient is zero. Similarly, reaching a certain safety threshold does not mean the optimization has converged. I feel the paper mostly demonstrates these known properties in the context of safety alignment, rather than providing a substantially new insight.
  2. Most of the important experiments are conducted when the gradient is exactly zero. However, in practical LLM alignment, we often have additional objectives, such as KL regularization or helpfulness rewards, which may continue producing gradients. The paper does not clearly show how frequently this zero-gradient situation occurs in realistic alignment training, or how important the resulting drift actually is. Table 21 shows that KL = 0.001 eliminates silence entirely.
  3. If the main problem is that model behavior degrades after reaching the desired safety level, why not simply stop training at that point? I think early stopping is an important baseline missing from the paper. Without comparing against it, it is difficult to understand the practical advantage of the proposed PAC intervention.

Minor:

The equation 1 from Section 3.2 is also written in the Introduction verbatim. Maybe it can be replaced with a high level visual or representative example, and this could make the underlinying motivation clearer.

Questions for authors

See weakness

3 comments

Thank you for your review. We distinguish the known properties of AdamW from our empirical contribution and provide additional KL and early-stopping comparisons.

W1. Is this a reproduction of a known property of AdamW?

AdamW updates with zero current gradients, and optimizer resets, are known. Our question is whether the stored state contributes specifically to over-refusal and degradation of substantive responses after safety alignment.

We branched from the same checkpoint while holding weights, data order, reward, and schedule fixed. During the subsequent zero-gradient interval, retaining the first moment increased ORR to .868, whereas repeatedly clearing it kept ORR at .640–.652.

The component-wise comparison in Table 13 is shown below (Mistral, one run per condition).

InterventionInternal ASR ↓Internal ORR ↓Internal SCR ↑
No intervention.0001.00.004
Repeated first-moment reset.000.98.729
One-time first-moment reset.000.98.716
Second-moment reset.000.98.000
Reset both moments.0001.00.000
Full optimizer reset.0001.00.000

In these conditions, clearing only the first moment while preserving the second moment and step counter preserved SCR. We do not attribute all over-refusal to the first moment; residual objective pressure is examined separately.

W2. Does the finding matter when KL or helpfulness gradients remain?

We added same-checkpoint branches with KL=.001. For all three models, threshold reset clears the first moment when the pre-clipping L2 norm of the total gradient is at most .1. Results below are external evaluations after 20 branch steps, with rollout seeds 42/43.

ModelInterventionτHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralRetain—.000/.0001.000/1.000.118/.140
Threshold reset.1.010/.020.922/.932.520/.530
QwenRetain—.000/.000.980/.988.434/.425
Threshold reset.1.010/.010.853/.858.601/.592
LlamaRetain—.000/.000.993/.998.069/.047
Threshold reset.1.030/.015.647/.828.409/.350

The intervention changed behavior even with small nonzero gradients. However, lower over-refusal and higher SCR came with increased HarmBench ASR. We fixed τ=.1 before the new branches, after inspecting earlier KL logs and behavioral results; we do not claim that it is a general detector of reward saturation.

Exact silence is frequent in our safety-only setting (70, 75 and 84 of 90 steps without KL; Table 21) but disappears with KL=.001, so we do not claim that exact-zero steps are common in practical alignment. Our claim is that stored optimizer state continues to affect behavior after the safety target is reached, which the KL branches above show even with nonzero gradients. Experiments with helpfulness rewards remain future work.

W3. How does PAC compare with early stopping?

We agree that early stopping is a valid alternative. In new runs that save every step (training seeds 3/4), we selected the first checkpoint with internal validation ASR ≤5%.

ModelMethodHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralValidation stopping.205/.225.600/.615.58/.61
PAC.015/.055/.035.958/.938/.960.43/.46/.44
QwenValidation stopping.065/.075.198/.203.63/.63
PAC.030/.005.833/.980.59/.49
LlamaValidation stopping.095/.095.278/.280.50/.50
PAC.015/.005.787/.938.35/.24

PAC uses existing seeds 1/2 (1–3 for Mistral), so this is not a within-run comparison. Separately, we added figures comparing early stopping with step 90 within the same runs for all three models. Continued training reduced HarmBench ASR but increased ORR and reduced Health SCR. Due to disk-space constraints, we had to delete intermediate checkpoints after training finished and before the rebuttal period. For runs whose required intermediate checkpoints were no longer available, we re-trained them with the same configuration.

PAC also showed no consistent advantage over checkpoints selected retrospectively using external evaluations, including on attack evaluations. We therefore do not claim that PAC is an optimal stopping strategy.

If title changes are permitted, we would like to adopt “Safety Attainment Does Not Ensure Behavioral Stability: Objective and Optimizer Sources of Behavioral Drift after Safety Alignment.” We will review the terminology throughout the manuscript accordingly, distinguishing safety attainment, zero batch gradients, and objective stationarity. We will also simplify the duplicated equation in the Introduction.

AuthorsOpenReview ↗

Follow-up: results with a helpfulness reward

In our earlier replies we mentioned helpfulness rewards as future work. We have since run this setting and wanted to share the results.

With a helpfulness reward on benign prompts (our Mixed condition, no KL), the total gradient remained nonzero in these evaluated runs, so the PAC gate as defined in the paper never fired. To still test the idea, we applied it to the harmful-prompt part of the gradient only. At each step we compute the harmful-prompt and benign-prompt gradients separately with two backward passes on the same batch. On Mistral's first four steps, the relative discrepancy between their sum and the full gradient was 1.1–1.4%, compared with 1.1–1.3% between repeated full-batch passes. We keep a running copy m_harm of the harmful-prompt part, using the same β1 and the clipping coefficient computed for the total gradient. Once ‖g_harm‖ has stayed at or below 0.1× its mean over the first five steps with nonzero g_harm for two consecutive steps, we subtract m_harm from Adam's first moment on each such quiet step and reset m_harm to zero. There is no separate attainment criterion. Other optimizer states continue their normal updates without direct intervention.

CISPO, Mixed, step 90 (seed 1/seed 2). ORR is from the official GPT-4o grader, as before.

ModelConditionHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralNo intervention.045/.105.768/.643.676/.586
Harmful-share m removed.130/.145.598/.550.607/.533
QwenNo intervention.150/.015.482/.912.630/.632
Harmful-share m removed.110/.055.430/.475.622/.600
LlamaNo intervention.075/.050.477/.573.628/.545
Harmful-share m removed.075/.190.200/.295.606/.573

ORR went down in all six cases, but this came at a cost in safety: HarmBench ASR went up in four of six (one tie, one decrease) and SCR went down in five of six.

We ran the same check for DPO with a shared optimizer, removing the safety-objective share of the first moment on steps where the hinge term had zero gradient (step 91, seed 1/seed 2):

ModelConditionHarmBench ASR ↓OR-Bench-Hard ORR ↓Health-ORSC SCR ↑
MistralNo intervention.165/.155.282/.395.676/.681
Safety-objective m removed.195/.195.228/.242.658/.656
ZephyrNo intervention.195/.200.210/.233.616/.632
Safety-objective m removed.330/.290.145/.130.597/.593

The picture is the same: lower ORR, but higher ASR and lower SCR.

We also tried removing only half of the component (α=.5) for CISPO on Qwen and Llama and for both DPO models. This did not help consistently. Only one run (Qwen, seed 1) was better than its control on all three metrics.

So the optimizer state still shapes refusal behavior when other objectives keep the gradient nonzero, but these component-wise first-moment interventions did not consistently improve the overall safety–utility trade-off. We will include these results, the per-step training curves for all conditions, and the details of the method in the appendix.

Reviewer_jDQPOpenReview ↗

Thank you for the detailed response

Thank you for the detailed response and the new experiments with KL and the helpfulness reward. These answer my question about whether the effect holds outside exact zero-gradient steps.

However, my main concerns remain. On novelty, the rebuttal agrees that AdamW updating at zero gradient is known, so the contribution is mainly showing this effect in a safety setting. On practical value, the authors now state that PAC does not consistently beat early stopping, and in the KL and helpfulness settings the first-moment intervention lowers over-refusal but raises harmful compliance. So it is still unclear how this finding would change safety alignment in practice. I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.