acceptodds
Under review as a conference paper at ICLR 2027

See it, Say it, Sorted: Mechanistic Diagnosis and Parameter-Space Mitigation of Emergent Misalignment in LLMs

Abstract

Safety-aligned LLMs can exhibit emergent misalignment (EM): narrow domain adaptation unexpectedly triggers catastrophic safety failures across unrelated domains. Prior static analyses leave training dynamics unmapped, while existing defenses rely on heuristics that degrade utility. We present a dynamic, second-order geometric study of EM. Tracking training trajectories reveals that directional Hessian curvature concentrates sharply on semantic pivot tokens. Grassmannian projections show that, in most settings, harmful-safe gap widens mainly because safe-gradient overlap declines. Leveraging these insights, we introduce a parameter-level Geometric Mitigation Framework that orthogonally projects empirical harmful gradient subspace out of parameter updates. On Qwen2.5-14B-IT, our defense suppresses free-generation EM by up to 80.0%; across the other three of four open-weight instruction-based model families (3B–20B), where single-layer behavioral EM is already near zero, teacher-forced evaluation shows same harmful subspace controls the conditional support of frozen EM responses. Crucially, these diagnostics unmask the illusion of behavioral safety: the same subspace remains measurable and steerable in models where behavioral EM is near zero.

Primary area:
interpretability and explainable AI
Keywords:
emergent misalignment, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_E8ADOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies emergent misalignment during narrow harmful fine-tuning of instruction-tuned LLMs. It proposes token-level directional Hessian curvature and gradient-subspace overlap diagnostics, then uses a rank-4 harmful-gradient subspace to ablate or amplify components of a single-layer rank-1 LoRA update. The experiments span several model families and three harmful adaptation domains, with free-generation and teacher-forced evaluations of the intervention.

Critical strengths
  1. The intervention design is genuinely controlled. Both ablation and amplification are tested, with norm-matched Haar-random directions as controls — much stronger than a one-sided "removal improves safety" result, and Figure 7 shows the two directions move behavior in opposite ways.

  2. Figure 3 establishes a real empirical fact.All-layer LoRA produces far more behavioral EM than the single-layer model organism, which is informative in its own right and honestly frames what the rest of the paper can and cannot claim.

Critical weaknesses
  1. The core equations are broken as written.In Eq. (1) both terms use the same checkpoint, so δ ≡ 0 and the pivot/neutral token sets can't be constructed; Section 3.1 likewise defines Δφ = φ_s − φ_s. Since pivot selection feeds the curvature plots and the intervention, this isn't cosmetic — the method is non-reproducible until the authors confirm what was actually run.

  2. The mitigation doesn't address the problem it motivates.The paper is motivated by all-layer EM (up to 28.5% in Figure 3), but every intervention acts on a single LoRA module where free-generation EM is already near the floor. The "up to 80% reduction" is relative to that low baseline, not to the all-layer rates that justify the paper.

  3. The geometry is coordinate-dependent.Projections happen in concatenated (A, B) factor space, but the effective update is bilinear (ΔW ∝ BA) with scale/reparameterization symmetries. An orthogonal direction in factor space need not mean anything in weight or function space, which undermines the "causal removal of a harmful mechanism" interpretation.

Questions for authors

Same as Weakness

Reviewer_7yHFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates emergent misalignment (EM) through Hessian curvature and gradient subspace analysis. The authors identify EM-related pivot tokens, analyze their geometric dynamics, and propose a parameter-space projection method to suppress harmful updates. Experiments show substantial EM reduction and reveal geometric signals even when behavioral misalignment is nearly absent.

Critical strengths
  1. In-depth geometric analysis. The paper goes beyond static behavioral analysis by investigating training-time optimization geometry, providing interesting insights of EM dynamics.

  2. Broad empirical experiments. The experiments cover multiple model families and adaptation domains which support their conclusion.

Critical weaknesses

W1. Potential selection bias in pivot-token analysis.

Pivot tokens are selected based on likelihood increases induced by harmful fine-tuning, which may inherently favor alignment with harmful training updates. Independent token selection or matched benign fine-tuning controls would strengthen the claim.

W2. Insufficient evidence for passive decoupling.

While declining safe–pivot overlap indicates geometric divergence, it does not directly establish the erosion of existing safety constraints. Further evidence linking this geometric observation to actual safety degradation would strengthen the proposed mechanism.

W3. Unclear connection between the mitigation method and the proposed mechanism.

Removing harmful-gradient directions may suppress harmful-data fitting without necessarily preserving the original safety mechanisms. Related null-space projection approaches, such as NSPO (ICLR 2026), warrant discussion. Comparisons with safety-preserving projections or uniformly scaled updates would better clarify the method's distinctive contribution.

W4. Limited validation of practical mitigation and fine-tuning utility.

Although general and in-domain knowledge preservation is evaluated, these benchmarks exhibit negligible gains from fine-tuning itself. Experiments with meaningful adaptation gains would better establish the practical safety–utility trade-off.

Questions for authors

Please address the concerns and clarification requests in W1–W4 under Critical Weaknesses.

1 comment
Reviewer_7yHFOpenReview ↗

Official Review of Submission11652 by Reviewer 7yHF

Thank you for the detailed rebuttal and substantial revisions. I appreciate the additional experiments, particularly the pivot-selection controls, all-layer intervention results, and comparisons with shrinkage and rollback baselines. These additions have largely addressed my concerns and strengthened the paper's empirical contributions. I also appreciate the authors' clarification of the scope of their claims, while the causal interpretation of passive decoupling (W2) and the safety–utility trade-off under meaningful downstream adaptation (W4) remain interesting directions for future investigation. I have decided to raise my rating accordingly.

Reviewer_YmK6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper combines longitudinal geometric analysis with parameter interventions, going beyond purely observational endpoint analyses. Norm-matched random controls and the additional safe-direction control strengthen the attribution experiments.

  2. The paper reports negative cases, intervention asymmetries, and general and domain-specific capability checks, providing useful evidence about the intervention’s scope and limitations.

Critical strengths
  1. The paper combines longitudinal geometric analysis with parameter interventions, going beyond purely observational endpoint analyses. Norm-matched random controls and the additional safe-direction control strengthen the attribution experiments.

  2. The paper reports negative cases, intervention asymmetries, and general and domain-specific capability checks, providing useful evidence about the intervention’s scope and limitations.

Critical weaknesses
  1. The claims about latent risk exceed the behavioral evidence. Significant free-generation mitigation is demonstrated primarily on Qwen2.5-14B. The Qwen2.5-7B result is nonsignificant, while most other models are evaluated through teacher-forced likelihoods of frozen responses produced by all-layer-adapted counterparts. These measurements establish conditional support and its sensitivity to intervention, but do not establish that the evaluated models will autonomously generate harmful responses or that such behavior can be elicited by realistic contextual triggers. Statements that the results “prove” implanted latent misalignment or expose an “illusion of behavioral safety” are therefore too strong.

  2. The geometric observations do not sufficiently establish the proposed general mechanism. Pivot tokens are retrospectively selected using likelihood gains over the analyzed training interval. Although the curvature analysis is not mathematically identical to this selection criterion, permutation within the selected pool does not rule out selection-related confounds involving token probability, gradient magnitude, or ordinary task learning. Furthermore, declining overlap with safe gradients is consistent with decoupling, but does not independently establish the stronger explanation that pre-existing, low-curvature pretraining behaviors are being released. The main geometric and intervention analyses concern single-layer rank-1 LoRA; their applicability to the all-layer setting, where behavioral misalignment is substantially stronger, remains unverified.

Questions for authors
  1. Can the proposed diagnostics predict subsequent free-generation misalignment on held-out prompts, checkpoints, and independent training seeds? For models at the behavioral floor, can realistic triggers expose the claimed latent risk, and does ablation reduce it?

  2. Does curvature separation persist against probability- and gradient-matched token controls, and during matched safe adaptation? Can pivot sets selected on a separate interval or run reproduce the findings?

  3. Can the intervention be validated on all-layer LoRA, with comparisons against simple update shrinkage or rollback and representative existing mitigation methods at matched utility?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.