- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies why post-training on data that contains no harmful content still degrades a language model's propensity to refuse harmful requests. It names the phenomenon Safety Gate Collapse: an aligned model that refuses most harmful requests and resists attack can move to the opposite regime, with low refusal and high attack success, after fine-tuning on code, on general instructions, or through self-play reinforcement learning with verifiable rewards.
Two tools are introduced. Class-Conditional Activation Geometry compares the mean residual-stream activation of harmful and of benign prompts across coordinate-compatible checkpoints, so that changes specific to harmful prompts can be separated from changes shared with benign ones. The Inherited One-Sided Refusal Gate calibrates a difference-of-means refusal boundary once, on each aligned model, and applies it unchanged to that model's post-trained descendants. Holding the boundary fixed separates two ways refusal can fail: harmful prompts may leave the refusal region (occupancy falls), or they may remain inside it without producing refusal (conditional refusal falls). Equation (4) expresses the harmful refusal rate as the product of these two quantities.
The empirical study covers three aligned models (Qwen2.5-7B-Instruct, Qwen2.5-Coder-7B-Instruct, Llama-3.1-8B-Instruct) and three post-training methods (supervised fine-tuning on KodCode-4o, supervised fine-tuning on Alpaca, and self-play RLVR in the Absolute Zero Reasoner recipe), giving nine post-trained models. Eight of the nine lose HarmBench refusal, the largest change being Llama-3.1-8B-Instruct after Alpaca fine-tuning, where refusal falls by 78.5 points and attack success rises by 64.5. Harmful and benign prompts nonetheless remain linearly separable throughout, with probe AUROC at or above 99.66 at the measurement layer on every checkpoint. The authors report that the signed displacement of the harmful centroid along the gate normal predicts the direction of the refusal change on all nine models, that total displacement does not, and that injecting activation along the inherited gate direction restores refusal and reduces attack success on every post-trained model at a measurable cost in over-refusal.
Critical strengths
The decomposition of refusal loss into occupancy and conditional refusal, evaluated against a boundary that is calibrated once and never refitted, is a real methodological contribution. It converts a scalar behavioral change into a statement about where under the decision surface the change occurs, and it does so without fitting a classifier to the post-trained model. The predictive check in D.2 shows the decomposition is not merely descriptive: the aligned model's own margin-to-refusal curve, transferred unchanged, predicts post-fine-tuning refusal to within 3.7 points in eight of ten cells, and the two departures are reported and interpreted rather than suppressed. I would like to see this idea developed further, and it is the main reason I would encourage resubmission.
The measurement discipline is well above the norm for this literature. An explicit falsification paragraph (D.2), alternative boundary families and scoring rules (Table 14), per-layer sweeps of the same gate (Figures 7 to 9, Table 15), a logit-lens justification for the choice of measurement layer (Figure 6), five corpora across three models (Table 7), a sevenfold data-scale check, 13-gram decontamination against the evaluation benchmarks, capability controls, and an answer-length control that rules out the obvious confound of shorter completions. The ethics statement also shows good judgment in releasing verdicts rather than raw generations.
The behavioral result itself is solidly established. Eight of nine post-trained models lose refusal on HarmBench, the effect holds on JailbreakBench, SimpleSafetyTests and XSTest-unsafe (Table 7), it is not accompanied by a capability loss on the tasks the training targets, and the 131-step trajectory shows that the refusal change appears by the first retained checkpoint. Whatever one concludes about the mechanism, the phenomenon is real and the paper documents it well.
Critical weaknesses
-
The central dissociation is not established, because the refusal gate and the harmfulness axis are nearly the same direction. Appendix D.6 reports cosine similarities of 0.983, 0.997 and 1.000 between the gate normal and the harmful-minus-benign mean difference on the three aligned models, and no less than 0.97 at any layer. AUROC, which supports the claim that harmful information remains decodable, is invariant to a shift applied to both classes; occupancy against a fixed threshold is not. The reported pattern — the probe at ceiling, the gate emptying, and the frozen midpoint-threshold harm probe of D.6 following the gate down to 0.055, 0.615 and 0.000 — is what one expects from a translation along a shared projection, and does not by itself show that a harmfulness representation and a refusal mechanism have come apart. The evidence that would support the stronger claim, a class-conditional displacement in which harmful activations move along the normal while benign activations do not, is reported for the RLVR models only (at most 1.15 sigma against 7.3 to 12.3 sigma of total movement) and not for code fine-tuning or Alpaca fine-tuning, which are the two conditions carrying the paper's largest effects.
-
The benign pole of the axis is Alpaca, and the most dramatic condition trains on Alpaca. The gate and the probe are both calibrated on 200 HarmBench against 200 Alpaca prompts, so harmfulness is not separated from source-corpus identity, and XSTest-safe — whose safe prompts resemble harmful ones, and which is already collected — is used only to measure over-refusal. Under Alpaca fine-tuning the training distribution is the benign pole, so part of the measured displacement along the gate normal may reflect a global shift toward the training distribution rather than a harm-specific movement. Since this condition produces occupancy of 0.000 on Llama, 0.015 on Instruct and 0.580 on Coder-Instruct, and the 78.5-point drop quoted in the abstract, the concern bears on the paper's headline geometric evidence, not only on a side analysis. The behavioral claim survives; the mechanism claim does not.
-
The scope of the evidence does not support the generality of the framing. The study covers three aligned models from two families, one training run per configuration, English-only evaluation, and no preference-based or continued-pre-training method, while the introduction and conclusion are written as a general account of benign post-training. The one-run limitation is not that uncertainty is unreported — prompt-level bootstrap intervals, McNemar tests and a likelihood-ratio test are all present — but that uncertainty at the level of the training run is absent, and the 131-step trajectory and five corpora are offered in its place without saying so. This matters at the margin: under code fine-tuning the refusal changes of Instruct and Coder-Instruct are both within sampling noise against the aligned model (McNemar p = 0.09 and 0.34), yet both cells count toward the nine-for-nine sign agreement, which is therefore supported by seven cells with a measurable behavioral change rather than nine. An agreement of nine out of nine whose entries include two null changes is weaker evidence than the count implies.
-
Every headline refusal rate rests on a single LLM judge. The three-way distinction among refusal, evasive and failure is applied by GPT-5.5 at fixed settings, with the rubric given verbatim in Appendix H, but no human agreement study is reported. Because the paper's effects are large this is unlikely to overturn the behavioral result; it does bear on the comparisons among smaller changes, which is where several of the mechanism claims sit.
Questions for authors
Under code fine-tuning and Alpaca fine-tuning, what is the benign class's signed displacement along the gate normal, reported as in Section 4.5 for RLVR? If the benign class also moves several sigma along the normal in those conditions, does the class-conditional reading of Figure 2 survive?
If the gate and the harm probe are recalibrated with XSTest-safe, or another corpus that is neither the training set nor the Alpaca calibration set, as the benign pole, do the occupancy collapse and the sign agreement persist?
Appendix C fits the probe direction on the same 400 prompts whose AUROC is then reported. Cross-fitting is applied to the gate readout in D.6 but not to the probe. What is the held-out probe AUROC, and does the gap between probe and gate reported in Section 4.3 change?
Can the nine-for-nine sign agreement be restated conditional on the cells whose refusal change is distinguishable from zero?
Do the authors regard the 131-step trajectory and the five corpora as substitutes for seed replication? If not, what is the spread across two or three seeds for one configuration, ideally Llama under code fine-tuning?
What is the human agreement rate with the three-way judge on a stratified sample of responses, and does the disagreement concentrate in the evasive category, which is where the distinction between declining and deflecting is hardest to draw?
The method requires an aligned reference checkpoint in a compatible coordinate system. Can occupancy be estimated for a model whose aligned reference is unavailable, or for one whose representations have been restructured by merging or architectural change? If not, how should the diagnostic be positioned relative to black-box safety evaluation?