acceptodds
Under review as a conference paper at ICLR 2027

The Geometry of Refusal: Why Post-Hoc Safety is Fragile and Pretraining-Time Safety Persists

Abstract

Post-hoc safety training (RLHF, DPO) is the dominant way to align large language models, yet jailbreaks (Zou et al., 2023), fine-tuning attacks (Qi et al., 2024), and activation-space edits (Arditi et al., 2024) keep recovering the behaviors it was meant to remove. We give this fragility one geometric explanation and follow it into pretraining. We measure the safety update = − against the curvature of the model's capabilities (the empirical Fisher of a capability loss). Across five model families, post-hoc safety lands in a suppression regime: is nearly orthogonal to the capability directions, and its small in-subspace part concentrates on a few high-curvature ones. The update is thin but sharp, a refusal gate laid over intact capabilities rather than erasure of them. A kernel-immobility lemma explains why such an update can only mask a capability, not remove it, so a little benign fine-tuning restores it: 100 benign examples cut the AdvBench refusal of Qwen-2.5-7B-Instruct and Llama-3-8B-Instruct by 35 to 38 pp. Following the account into pretraining, a pretraining-checkpoint sweep of OLMo-2-1B (Team OLMo et al., 2024) shows the features that refusal attaches to emerging in a sharp transition between 1B and 63B pretraining tokens. We then use the account constructively: models trained from scratch with safety co-training spread continuously across pretraining reach 87 to 98% AdvBench refusal that the same attack erodes by only 2 to 14 pp at every scale from 410M to 6.9B, against 35 to 38 pp for post-hoc installs, at a small cost on short-answer capability probes; a windowed schedule of equal total safety weight installs no refusal. Persistence of the safety signal across pretraining, not its timing, is what buys attack robustness.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Fine tuning attack, geometric explanation, post-trained, pre-training, large language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Bsp8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper explains the reason why post-hoc safety is fragile compared with the safety knowledge learned in the pretraining stage. Using a weight-space classification, the author found that most of the safety training in the post-training stage falls in the suppression region instead of the erasure region, explaining why post-hoc safety is so fragile. The author also proposes a kernel-immobility lemma to explain why post-hoc safety masks rather than removes the capability. In the end, the author proposes a mitigation strategy that uses contibuous safety co-training, supassing the baseline result.

Critical strengths
  1. The question of explaining the robustness of safety training in the pretraining stage and post-training is interesting and important.
  2. The experiments are rigorous and comprehensive.
Critical weaknesses
  1. The paper clarity is a very big issue. The writing is poor and hard to follow. Specifically, the paper uses a lot of compressed short paraphrases and headers (e.g., in line 195 , what's the meaning of "qnorm∼1q_{norm}\sim 1 a diffuse one"??, in line 247, "100 benign fine-tuning examples cost Qwen 34.8 pp of AdvBench refusal (Fig. 1b, top row)", what's the meaning of cost xx refusal??? ), with a lot of terminology used without definition. For example, what exactly is the meaning of "safety install"? What's the meaning of "geometric instrument"? In line 237, what's the meaning of qnormq_{norm} peaks? If we really want to find a peak value, why is the safety-install band an interval instead of a specific layer? What is "gate-attack" in figure1?
  2. The headline results is just an artifact of using regex safety checker. In figrue 1, the author claimed that compared with the basline model, whose refusal rate dropped by 35% and 38% for Qwen and Llama models, cotrained model has less than 6% drop of refusal rate. However, this results comes from the regex-based checker (see line 075). In the full results listed in table 25 and 26 (line 1637, 1638), if we use classifier based approach, the refusal rate drop for Qwen and Llama is merely 7.8% and 11.8, while for cotrained model, the refusal rate drop becomes 10.6% to 27.0%, even worse than the baseline.
Questions for authors

I have listed my questions in the weakness section

Reviewer_889XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a geometric explanation for the fragility of post-hoc safety alignment. It measures the displacement between base and aligned checkpoints relative to an empirical Fisher matrix constructed from benign capability probes. Small overlap with leading capability directions, together with an elevated normalized Fisher quadratic form, is interpreted as refusal gating over retained capabilities.

The paper supplements these measurements with a local kernel-immobility lemma, a developmental checkpoint analysis, and from-scratch co-training experiments using Pythia architectures. Its main conclusion is that continuous safety training produces more persistent safety than post-hoc alignment.

Critical strengths
  1. The paper develops a substantial descriptive measurement program. The weight-space analysis spans multiple model families, capability probes, and transformer blocks. It provides a concrete procedure for studying where alignment updates lie relative to gradients associated with general abilities. The raw and three-shot evaluations support the narrower observation that the measured projected norms remain small under both formats. These measurements could become useful diagnostics if their relationship to behavior and subsequent training dynamics were established.

  2. The schedule experiments reveal a meaningful distinction between retaining refusal prefixes and maintaining broader safe behavior. Continuous co-training retains high refusal-prefix rates on AdvBench, whereas an early safety window followed by language-model-only training does not. The additional benchmark and classifier evaluations are valuable because they expose the behavioral and distributional scope of that persistence. This distinction is one of the paper’s most informative empirical observations.

Critical weaknesses
  1. The paper’s own harmful-output evaluation contradicts the claimed robustness ordering.

The headline comparison emphasizes approximately 35–38 percentage points of erosion for post-hoc alignment versus 2–14 points for co-training. Appendix L evaluates the saved generations with Llama Guard 3 and reports the following AdvBench results.

The entries below are classifier-defined non-harmful rates.

ModelBefore benign ftAfter benign ftDecline
Qwen-2.5-7B-Instruct93.0%85.2%7.8 pp
Llama-3-8B-Instruct39.5%27.7%11.8 pp
Co-trained 410M91.8%64.8%27.0 pp
Co-trained 1B83.2%63.7%19.5 pp
Co-trained 2.8B91.0%77.3%13.7 pp
Co-trained 6.9B97.3%86.7%10.6 pp

These point estimates do not preserve the advertised ordering. They directly conflict with the statement on page 31, repeated on page 32, that the classifier confirms substantially greater erosion in post-hoc models.

Appendix L also explains that a refusal prefix can remain present while its continuation becomes harmful. Consequently, prefix persistence cannot carry the central claim about persistent safety. The numerical inconsistency requires a revised interpretation regardless of whether every pairwise difference is statistically significant.

The comparison additionally changes architecture, pretraining maturity, safety data, and safety objective. The within-architecture post-hoc DPO control starts with no refusal, and the manuscript explicitly reports overlap between the AdvBench co-training and evaluation pools. These conditions prevent attribution of the observed difference to the timing of safety training.

  1. The geometric analysis does not validate the suppression/erasure distinction or explain subsequent recovery.

The exact-kernel step in Lemma 1 is valid: a displacement in the kernel of the empirical Fisher is orthogonal to the sampled gradients, giving a local Taylor identity. Its application to the measured checkpoints requires additional evidence.

Small overlap with the leading 128 empirical directions does not establish an exact kernel condition, control omitted directions, or bound a finite update’s Taylor remainder. Both reported normalized statistics are invariant to rescaling the update, while the update’s functional effect can change substantially. A threshold on those statistics therefore needs independent behavioral validation.

The empirical-Fisher/Hessian identification also requires assumptions beyond a checkpoint being mature or well trained. The distinction between the empirical Fisher, true Fisher, and generalized Gauss–Newton matrix is central to the analyses of Martens and Kunstner et al.

The proposed erasure control does not supply the needed validation. In Appendix G, Table 12, the intervention improves gold-answer NLL by 0.96 nats on REASON and 2.08 nats on ARITH. The paper attributes this to formatting and asserts that argmax accuracy would decline, but does not measure that accuracy. Crossing the geometric threshold therefore has not been shown to correspond to capability erasure.

Finally, the probes concern benign general abilities. Preserving their loss does not identify whether harmful knowledge was removed or whether refusal operates through a gate. The lemma also provides no condition on the later fine-tuning objective that would make recovery follow from local capability-loss invariance.

Questions for authors
  1. How should the main robustness conclusion change in light of Tables 25–26? Please reconcile the reported classifier-defined declines with the claim that this evaluation preserves the post-hoc/co-training ordering.

  2. Does the proposed erasure control reduce held-out task accuracy, and do the geometric quantities predict the actual directional loss change? Accuracy measurements and a comparison of the linear term, directional Hessian, empirical-Fisher quadratic form, and observed loss along the update would directly test the interpretation.

  3. Does continuous co-training retain an advantage with disjoint evaluation prompts and a matched post-hoc safety-SFT baseline? The comparison should use the same architecture and safety examples, comparable initial safety, appropriate inference formats, and harmful-output scoring.

Reviewer_uqyyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies why safety alignment introduced after pretraining can be weakened by later fine-tuning, and whether safety training incorporated throughout pretraining produces more persistent refusal behavior. The authors characterize safety updates using their overlap with capability-related empirical Fisher eigenspaces, provide a local theoretical argument for why updates outside these directions may preserve capability, examine how refusal-related representations develop across pretraining checkpoints, and compare post-hoc alignment with continuous and windowed safety co-training. Experiments are conducted across several model families and evaluate refusal persistence under benign fine-tuning attacks.

Critical strengths
  1. Understanding why safety training can be weakened by later fine-tuning is an important problem.
  2. The training experiments and reported failures are useful. The paper evaluates safety training across four different model families. These include models trained without safety examples, stronger fine-tuning attacks at one scale, and an additional harmful-output classifier. The additional harmful-output classifier is also useful because it reveals limitations of regex-based refusal evaluation.
Critical weaknesses
  1. The paper appears as heavily edited by AI or entirely written by AI, making it really hard to understand by a human reader. A dominant number of sentences and expressions are over compressed and machine-sounding.
  2. The safety evaluation does not consistently support the claimed robustness advantage. The main comparison relies mainly on whether responses contain refusal phrases. After fine-tuning, refusal declines by approximately 35–38 percentage points in post-hoc aligned models, compared with 2–14 points in models trained with safety during pretraining. However, the harmful-output classifier gives a different comparison. The percentage of outputs not classified as unsafe decreases by 7.8 points for Qwen and 11.8 points for Llama, compared with about 10.6–27.0 points for the models trained with safety during pretraining (Appendix L, Tables 25–26). These results are not consistent with the claimed robustness ordering. Consequently, the persistence of refusal phrases cannot substantiate the claimed persistence of safe behavior. This means persistent refusal wording is not equivalent to persistent safe behavior.
  3. The measurements are insufficient for the paper’s proposed explanation for safety fragility. The explanation depends on distinguishing a model that retains a harmful ability but learns to refuse from a model that has lost that ability. However, the geometric measurements are constructed from four general-ability probes, which do not directly test the harmful capabilities in question. Preserved general capability on these probes does not imply preserved harmful capability. Similarly, low overlap between safety updates and the measured capability directions does not show why subsequent benign fine-tuning would undo refusal. The paper consequently shows an interesting geometric association, but not that this geometry explains why benign fine-tuning removes refusal.
Questions for authors
  1. Does the conclusion that pretraining-time safety is harder to remove still hold when harmful-output classification, rather than refusal-phrase detection, is used as the primary metric?
  2. Among outputs counted as refusals by the regex but classified as unsafe by the harmful-output classifier, how often does the model begin with a refusal phrase and then provide harmful information?
  3. Can the authors compare continuous co-training with a stronger post-hoc SFT or SFT+DPO baseline matched for pre-attack refusal?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.