acceptodds
Withdrawn from ICLR 2027

Why Benign Post-Training Induces Safety Gate Collapse: Changes in Harmful Activation Geometry Relative to the Inherited Refusal Gate

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Benign post-training can drive an aligned model from high refusal and low attack success to the opposite pattern, we call this transition Safety Gate Collapse. We introduce Class-Conditional Activation Geometry to quantify differences between harmful and benign activation changes, and the Inherited One-Sided Refusal Gate to connect these geometric changes to the model's refusal behavior using a fixed boundary calibrated on the aligned model. We find that (i) benign SFT on code or general instructions and self-play RLVR lower refusal on eight of the nine post-trained models we study, by up to %, and raise attack success on the same eight; (ii) harmful and benign requests remain linearly distinguishable from internal activations (probe AUROC % on every post-trained model), while occupancy of the inherited refusal region by harmful activations on Llama-3.1-8B-Instruct falls from to after code SFT and to after Alpaca SFT; (iii) the sign of the harmful centroid's displacement along the gate normal matches the direction of refusal change on all nine models, whereas total displacement does not; and (iv) steering along the inherited gate direction raises refusal and lowers attack success on every post-trained model, but can also increase over-refusal on benign requests. These findings identify changes in harmful activation positions relative to the inherited gate as a diagnostic of refusal degradation, even when harmful and benign requests remain linearly distinguishable.

Primary area:
interpretability and explainable AI
Keywords:
safety alignment, refusal, post-training, benign fine-tuning, mechanistic interpretability, activation geometry, activation steering, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4C6ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why post-training on data that contains no harmful content still degrades a language model's propensity to refuse harmful requests. It names the phenomenon Safety Gate Collapse: an aligned model that refuses most harmful requests and resists attack can move to the opposite regime, with low refusal and high attack success, after fine-tuning on code, on general instructions, or through self-play reinforcement learning with verifiable rewards.

Two tools are introduced. Class-Conditional Activation Geometry compares the mean residual-stream activation of harmful and of benign prompts across coordinate-compatible checkpoints, so that changes specific to harmful prompts can be separated from changes shared with benign ones. The Inherited One-Sided Refusal Gate calibrates a difference-of-means refusal boundary once, on each aligned model, and applies it unchanged to that model's post-trained descendants. Holding the boundary fixed separates two ways refusal can fail: harmful prompts may leave the refusal region (occupancy falls), or they may remain inside it without producing refusal (conditional refusal falls). Equation (4) expresses the harmful refusal rate as the product of these two quantities.

The empirical study covers three aligned models (Qwen2.5-7B-Instruct, Qwen2.5-Coder-7B-Instruct, Llama-3.1-8B-Instruct) and three post-training methods (supervised fine-tuning on KodCode-4o, supervised fine-tuning on Alpaca, and self-play RLVR in the Absolute Zero Reasoner recipe), giving nine post-trained models. Eight of the nine lose HarmBench refusal, the largest change being Llama-3.1-8B-Instruct after Alpaca fine-tuning, where refusal falls by 78.5 points and attack success rises by 64.5. Harmful and benign prompts nonetheless remain linearly separable throughout, with probe AUROC at or above 99.66 at the measurement layer on every checkpoint. The authors report that the signed displacement of the harmful centroid along the gate normal predicts the direction of the refusal change on all nine models, that total displacement does not, and that injecting activation along the inherited gate direction restores refusal and reduces attack success on every post-trained model at a measurable cost in over-refusal.

Critical strengths

The decomposition of refusal loss into occupancy and conditional refusal, evaluated against a boundary that is calibrated once and never refitted, is a real methodological contribution. It converts a scalar behavioral change into a statement about where under the decision surface the change occurs, and it does so without fitting a classifier to the post-trained model. The predictive check in D.2 shows the decomposition is not merely descriptive: the aligned model's own margin-to-refusal curve, transferred unchanged, predicts post-fine-tuning refusal to within 3.7 points in eight of ten cells, and the two departures are reported and interpreted rather than suppressed. I would like to see this idea developed further, and it is the main reason I would encourage resubmission.

The measurement discipline is well above the norm for this literature. An explicit falsification paragraph (D.2), alternative boundary families and scoring rules (Table 14), per-layer sweeps of the same gate (Figures 7 to 9, Table 15), a logit-lens justification for the choice of measurement layer (Figure 6), five corpora across three models (Table 7), a sevenfold data-scale check, 13-gram decontamination against the evaluation benchmarks, capability controls, and an answer-length control that rules out the obvious confound of shorter completions. The ethics statement also shows good judgment in releasing verdicts rather than raw generations.

The behavioral result itself is solidly established. Eight of nine post-trained models lose refusal on HarmBench, the effect holds on JailbreakBench, SimpleSafetyTests and XSTest-unsafe (Table 7), it is not accompanied by a capability loss on the tasks the training targets, and the 131-step trajectory shows that the refusal change appears by the first retained checkpoint. Whatever one concludes about the mechanism, the phenomenon is real and the paper documents it well.

Critical weaknesses
  1. The central dissociation is not established, because the refusal gate and the harmfulness axis are nearly the same direction. Appendix D.6 reports cosine similarities of 0.983, 0.997 and 1.000 between the gate normal and the harmful-minus-benign mean difference on the three aligned models, and no less than 0.97 at any layer. AUROC, which supports the claim that harmful information remains decodable, is invariant to a shift applied to both classes; occupancy against a fixed threshold is not. The reported pattern — the probe at ceiling, the gate emptying, and the frozen midpoint-threshold harm probe of D.6 following the gate down to 0.055, 0.615 and 0.000 — is what one expects from a translation along a shared projection, and does not by itself show that a harmfulness representation and a refusal mechanism have come apart. The evidence that would support the stronger claim, a class-conditional displacement in which harmful activations move along the normal while benign activations do not, is reported for the RLVR models only (at most 1.15 sigma against 7.3 to 12.3 sigma of total movement) and not for code fine-tuning or Alpaca fine-tuning, which are the two conditions carrying the paper's largest effects.

  2. The benign pole of the axis is Alpaca, and the most dramatic condition trains on Alpaca. The gate and the probe are both calibrated on 200 HarmBench against 200 Alpaca prompts, so harmfulness is not separated from source-corpus identity, and XSTest-safe — whose safe prompts resemble harmful ones, and which is already collected — is used only to measure over-refusal. Under Alpaca fine-tuning the training distribution is the benign pole, so part of the measured displacement along the gate normal may reflect a global shift toward the training distribution rather than a harm-specific movement. Since this condition produces occupancy of 0.000 on Llama, 0.015 on Instruct and 0.580 on Coder-Instruct, and the 78.5-point drop quoted in the abstract, the concern bears on the paper's headline geometric evidence, not only on a side analysis. The behavioral claim survives; the mechanism claim does not.

  3. The scope of the evidence does not support the generality of the framing. The study covers three aligned models from two families, one training run per configuration, English-only evaluation, and no preference-based or continued-pre-training method, while the introduction and conclusion are written as a general account of benign post-training. The one-run limitation is not that uncertainty is unreported — prompt-level bootstrap intervals, McNemar tests and a likelihood-ratio test are all present — but that uncertainty at the level of the training run is absent, and the 131-step trajectory and five corpora are offered in its place without saying so. This matters at the margin: under code fine-tuning the refusal changes of Instruct and Coder-Instruct are both within sampling noise against the aligned model (McNemar p = 0.09 and 0.34), yet both cells count toward the nine-for-nine sign agreement, which is therefore supported by seven cells with a measurable behavioral change rather than nine. An agreement of nine out of nine whose entries include two null changes is weaker evidence than the count implies.

  4. Every headline refusal rate rests on a single LLM judge. The three-way distinction among refusal, evasive and failure is applied by GPT-5.5 at fixed settings, with the rubric given verbatim in Appendix H, but no human agreement study is reported. Because the paper's effects are large this is unlikely to overturn the behavioral result; it does bear on the comparisons among smaller changes, which is where several of the mechanism claims sit.

Questions for authors

Under code fine-tuning and Alpaca fine-tuning, what is the benign class's signed displacement along the gate normal, reported as in Section 4.5 for RLVR? If the benign class also moves several sigma along the normal in those conditions, does the class-conditional reading of Figure 2 survive?

If the gate and the harm probe are recalibrated with XSTest-safe, or another corpus that is neither the training set nor the Alpaca calibration set, as the benign pole, do the occupancy collapse and the sign agreement persist?

Appendix C fits the probe direction on the same 400 prompts whose AUROC is then reported. Cross-fitting is applied to the gate readout in D.6 but not to the probe. What is the held-out probe AUROC, and does the gap between probe and gate reported in Section 4.3 change?

Can the nine-for-nine sign agreement be restated conditional on the cells whose refusal change is distinguishable from zero?

Do the authors regard the 131-step trajectory and the five corpora as substitutes for seed replication? If not, what is the spread across two or three seeds for one configuration, ideally Llama under code fine-tuning?

What is the human agreement rate with the three-way judge on a stratified sample of responses, and does the disagreement concentrate in the evasive category, which is where the distinction between declining and deflecting is hardest to draw?

The method requires an aligned reference checkpoint in a compatible coordinate system. Can occupancy be estimated for a model whose aligned reference is unavailable, or for one whose representations have been restructured by merging or architectural change? If not, how should the diagnostic be positioned relative to black-box safety evaluation?

Reviewer_WxXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates safety degradation after benign SFT and self-play RLVR using class-conditional activation changes and a refusal boundary inherited from the aligned model. It finds that harmful and benign prompts remain linearly distinguishable, while harmful activations move toward or beyond the inherited refusal boundary. Steering along its normal partially restores refusal, sometimes at the cost of increased over-refusal.

Critical strengths

The distinction between information decodability and behavioral use is useful. A fixed reference boundary enables interpretable comparisons across post-training conditions, supported by prompt-matched analyses, alternative boundary models, and cross-fitting checks.

The experiments cover both SFT and RLVR and include capability and over-refusal measurements. Importantly, some safety degradation accompanies improved coding performance, demonstrating that the phenomenon is not simply general model deterioration.

Critical weaknesses

The evidence supports a diagnostic association more strongly than a causal explanation. The inherited gate is a refusal-associated classifier, rather than an independently established decision mechanism. Its occupancy decomposition is an application of the law of total probability and does not itself identify the cause of degradation. Directional steering demonstrates behavioral influence, but the intervention modifies every prompt and generated token, whereas the diagnostic measures only the final prompt token. It could therefore impose a general refusal tendency rather than reverse the specific change responsible for degradation. The substantial over-refusal increase in some settings reinforces this alternative explanation. Targeted activation restoration and direction-matched controls are needed to substantiate the stronger mechanistic claims.

The central representation claim and its generalization require stronger controls. Harmfulness probes distinguish HarmBench prompts from Alpaca instructions, so near-perfect AUROC may partly reflect dataset, topic, or stylistic differences rather than preserved harmfulness recognition. Appendix C also describes fitting and scoring the mean-difference probe on the same prompts; cross-fitting is reported for the refusal gate, but not clearly for this headline probe result. Furthermore, the main geometric analysis uses the same benchmark pair, and each post-training condition has only one training run. For RLVR, changes in the input template introduce an additional confound that should be separated from weight-update effects.

Questions for authors

Does the harmfulness-decoding result hold with held-out probes on topic- and style-matched harmful/benign pairs, such as XSTest-safe versus XSTest-unsafe?

Can prompt-specific restoration of the measured gate-normal displacement recover refusal without continuous generation-time steering?

How does it compare with norm-matched random and orthogonal interventions?

Can the gate predict refusal degradation on independent benchmarks and training seeds without recalibration?

For RLVR, can you report the aligned and post-trained models under both chat templates, separating template effects from training effects?

Reviewer_uXpqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why benign post-training (code SFT, Alpaca SFT, and self-play RLVR) reduces refusal in aligned LLMs. It uses three models (Qwen2.5-7B-Instruct, Qwen2.5-Coder-7B-Instruct, Llama-3.1-8B-Instruct) and defines an “inherited refusal gate”: a difference-of-means direction between refused and non-refused prompts, with a fixed threshold and scale, calibrated on the aligned model and frozen for the post-trained models.

Critical strengths
  1. The question matters. “The model still recognizes harm, so why doesn’t it refuse?” is a useful framing for benign fine-tuning safety loss.

  2. Good experimental control. Freezing the gate is the right choice. Refitting a boundary for each model would hide the effect being studied. Capability is tracked alongside safety.

  3. Many honest robustness checks. These include alternative boundaries, bootstrap intervals, cross-fitting, a full-depth sweep, and Appendix D.2, which states what would falsify the account.

Critical weaknesses

I want to thank the authors for their contributions. However, several concerns remain.

  1. The paper does not explain why post-training moves harmful activations. Most of the results are merely findings. As per my view, this work is not very novel in terms of proposing a method for solving the problem. A mean-difference approach is a naive way.

  2. The probe result (RQ2) is weak evidence. HarmBench and Alpaca prompts differ strongly in topic and wording, so near-100% AUROC is expected even from a shallow model or a bag-of-words baseline. High probe accuracy here says little about whether the model “still knows” a prompt is harmful. Suggestion: Add a lexical baseline and a probe on harder pairs (XSTest-unsafe vs. XSTest-safe, which look similar on the surface).

  3. It is informative only if the gate tracks the model’s real decision. But ρ₀ is 0.78 for Llama after code SFT and 0.62 after RLVR, meaning most prompts “outside the refusal region” are still refused. For these models, the gate is a loose proxy.

  4. The steering experiment partly contradicts the main claim. If refusal is lost because of position, moving activations back should restore it. But: c only goes up to 4σ, while the Alpaca shifts are about −8σ. For three models, c = 4 moves the median unrefused prompt past the threshold, yet only 31–42% of those prompts become refusals. Over-refusal on XSTest-safe reaches 59.6%. This suggests something besides position along this one direction has changed.

  5. The paper finds the shift is “no better aligned with the refusal direction than with generic high-variance structure,” but stops there. Prior work links benign data that breaks safety to its similarity with harmful data in gradient or representation space (e.g., He et al., “What’s in Your ‘Safe’ Data?”, COLM 2024). Suggestion: Test whether data similarity to harmful prompts predicts the size of the shift across corpora. This would turn the diagnosis into something useful.

  6. The conclusion says safe post-training “must preserve harmful activation positions,” and the early-warning claim rests on one training run (Figure 3). No mitigation is tested against existing methods that anchor the refusal direction (RefusalGuard). Suggestion: State how many changes are significant, test occupancy as an early warning on held-out runs, and compare a simple position-preserving regularizer against existing defenses.

Questions for authors

If the gate is calibrated on refused vs. complied harmful prompts only, does it differ from the harmful-vs-benign direction? Do the main results still hold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.