acceptodds
Withdrawn from ICLR 2027

Shared Parametric Geometry and Data Provenance in Model Organisms for Emergent Misalignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent work has discovered that large language models (LLMs) can develop broadly misaligned behaviors after being fine-tuned on narrowly harmful datasets, a phenomenon known as emergent misalignment (EM). However, the fundamental mechanisms enabling such harmful generalization across disparate domains remain poorly understood. We therefore adopt a geometric perspective to study EM in the weight space, starting with established LoRA-based model organisms that exhibit a cross-task linear structure: different, narrowly harmful supervised fine-tuning tasks converge to shared low-dimensional parameter subspaces and are behaviorally equivalent via linear mode connectivity, with interpolations between the resulting models maintaining coherent, broadly misaligned behavior. We further test the robustness of this geometric structure across LoRA ranks and model sizes, while also examining benign controls, data-generation templates, and data-generating LLMs, revealing a key role for data provenance: harmful and benign fine-tunes sharing data-generation templates exhibit comparable subspace overlap, whereas the tested cross-template comparisons yield little overlap. We subsequently examine the implications for safety through two post-hoc vaccination strategies, negation and orthonormal projection, finding that both improve alignment across the tested model families and that EM-derived projections yield greater alignment gains on average than benign-derived projections, while orthonormal projection also better preserves reasoning performance than negation. Collectively, our analyses connect data provenance to shared parametric geometry, showing that geometric convergence need not be specific to misalignment but can instead reflect common structure inherited from the data-generation process, while remaining behaviorally consequential for targeted interventions in LLM safety.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Alignment, Emergent Misalignment, AI Safety, Linear Mode Connectivity, Parametric Geometry, Weight-Space Analyses

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_STPdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Emergent Misalignment (EM) from a weight-space perspective, examining shared geometric structures and behavioral similarities across LoRA updates obtained from different harmful fine-tuning tasks. The authors further study the influence of data provenance, including data-generating models, generation templates, and benign fine-tuning datasets, on parameter geometry. Finally, they explore post-hoc vaccination through weight subtraction and orthonormal projection to improve model alignment while preserving reasoning capabilities.

Critical strengths
  1. The paper investigates a meaningful research question. In particular, the observation that shared parameter subspaces are not unique to EM, and may also emerge from benign fine-tuning with shared data-generation templates, provides an interesting empirical perspective on the relationship between parameter geometry and misalignment.

  2. The paper provides relatively comprehensive experimental analyses across different model sizes, LoRA ranks, training checkpoints, linear mode connectivity, data-provenance settings, and vaccination strategies. The authors also examine the behavioral implications of parameter geometry and consider the trade-off between alignment and reasoning performance.

Critical weaknesses

1. Distinctions from prior work and further analysis of data provenance.

The analyses of shared LoRA parameter subspaces and linear mode connectivity are closely related to prior work [1]. It would be helpful for the authors to clarify the additional contributions of this paper. Regarding data provenance, the paper considers different data-generating models and templates, but the main geometric results are reported in aggregate. Could the authors separately report the subspace similarities between GPT-4.1 and Claude Sonnet 4.5 under the same task and generation template, and compare them with results obtained using different templates but the same data-generating model? In addition, the number of different-template, same-domain comparisons is relatively limited. Further discussion of how this affects the conclusions would be useful.

2. Mechanistic interpretation and experimental comparisons for vaccination.

The paper observes similar subspace overlaps between EM and benign fine-tuning under shared templates, while projections derived from these adapters exhibit different effects on model alignment. It would be helpful to further explain this difference and clarify how the geometric analysis informs the selection of effective intervention directions. Additionally, comparisons with Safe LoRA [2] and simple LoRA scaling under matched safety–utility evaluation settings would help clarify the benefits of the proposed vaccination strategies.

References

[1] Shared Parameter Subspaces and Cross-Task Linearity in Emergently Misaligned Behavior. 2025.
https://arxiv.org/abs/2511.02022

[2] Hsu et al. (2024). Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models. NeurIPS 2024.
https://arxiv.org/abs/2405.16833

Questions for authors

see Weaknesses

Reviewer_cXrbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the weight-space geometry of emergent misalignment, finding substantial overlap between LoRA updates from different harmful tasks, but also between benign fine-tunes sharing data-generation templates. It examines behavioral connectivity between models and tests two interventions that remove adapter components to improve alignment.

Critical strengths
  1. The benign controls provide an interesting observation. Comparable overlap among harmful and benign fine-tunes shows that shared parameter geometry is not necessarily specific to misalignment (Section 5.1).
  2. The distinction between geometry and behavior is also useful. The template and interpolation experiments suggest that behavioral connectivity can persist despite substantial differences in update subspaces (Sections 5.2 and 5.3).
Critical weaknesses
  1. The theoretical justification for vaccination seems to contain mathematical errors. Appendix L.7 turns an average projection-energy identity over random directions into a deterministic bound for an individual update (Eq. 6 to Eq. 7). Average overlap cannot establish this guarantee. a worst-case bound requires additional assumptions and control of the largest principal angle instead. ALso, Appendix L.8 explicitly states that negation "projects onto a rank-1 subspace" (lines 2712 to 2713), although the intervention is fixed subtraction, Δ−m\Delta-m, rather than orthogonal projection, Δ−⟨Δ,m⟩∥m∥2m\Delta-\frac{\langle\Delta,m\rangle}{\|m\|^2}m. Fixed subtraction lacks the target-dependent coefficient and can overshoot, so the squared-cosine argument does not establish the residual-energy bound in Eq. (11).

  2. The experiments do not convincingly establish preservation of useful fine-tuning gains. In Appendix O, the original Llama base model achieves 89.44 alignment and 74.91% GSM8K accuracy, exceeding all three projected mixed-data models on both metrics. Simply discarding their adapters apparently performs better. For Qwen, the mixed-data models are already highly aligned, and projection lowers their alignment. The results therefore raise some concerns whether vaccination can repair newly introduced misalignment while retaining useful capabilities that reverting to the base model would lose.

Questions for authors

See weaknesses.

Reviewer_dwDnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies emergent misalignment in parameter space: a model fine-tuned on unsafe data in one domain may also give unsafe responses in other domains.

The authors compare LoRA updates from different unsafe fine-tuning tasks and find similar update directions and strongly overlapping low-dimensional subspaces. When they linearly interpolate between the weights of different unsafe models, the models along the path also produce coherent but unsafe responses.

The authors then add safe-data controls and compare different data-generation templates. They find that safe and unsafe models can have similar parameter structures when their training data use the same template. Subspace overlap drops sharply when different templates are used. This suggests that the geometric similarity may partly come from how the data were generated and cannot directly be taken as evidence of a shared mechanism for unsafe behavior.

Finally, the authors try to reduce unsafe responses while keeping math ability. They either subtract the average unsafe update or remove components of the target update within a selected subspace. In the tested settings, projection preserves math ability better than direct subtraction. Projections built from unsafe models also improve safety scores more on average than those built from safe models. However, safety and math performance do not always improve together.

Critical strengths

The paper includes useful control experiments. It compares not only different unsafe models, but also models trained on safe data from the same domains and generation templates. This shows that similar parameter structures are not necessarily unique to unsafe behavior. This is the most valuable part of the paper.

The paper combines geometric analysis, behavior tests, and weight changes. Besides cosine similarity and subspace overlap, it tests model behavior after weight interpolation and removal of projected components. This provides a more complete picture than reporting parameter similarity alone.

Critical weaknesses

The major issue is that the link between geometric similarity and unsafe behavior remains unclear. The same template may produce similar words, sentence patterns, and formats, leading to similar subspaces. Safe and unsafe behavior may still depend on different update directions or strengths within these subspaces. The paper does not separate these components, so it does not yet explain how unsafe behavior spreads across domains.

In addition, the evidence for template effects is limited. Geometric comparisons across different templates mainly use Llama and do not cover all combinations of domain, safety, data-generating model, and template. There are only two pairs comparing the same domain with different templates. More independent training runs are needed to support the conclusions.

If the authors can show that the template effects hold across repeated experiments and that the projection method outperforms simple control methods, this paper can be more solid.

Questions for authors

How do the authors separate parameter structure related to shared wording and format from structure related to unsafe behavior? Within the shared subspace, do safe and unsafe updates differ in direction or strength?

Do the template effects hold across more domains, both model families, and independent training runs? The same-domain, different-template comparison currently contains only two pairs.

How does the proposed projection compare with random-subspace removal or simply scaling down the LoRA update, when the amount of change is matched? Can these controls show that the method removes components specifically related to unsafe behavior?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.