acceptodds
Under review as a conference paper at ICLR 2027

Identifying the Effect of Training Context in Emergent Misalignment

Abstract

Supervised fine-tuning adapts language models to new tasks, but narrow training can also induce misaligned behavior beyond the training domain. Understanding this phenomenon requires isolating the effect of training context and examining how it is expressed by the resulting parameter change. Building on target-matched controls, we formulate target-matched SFT, holding assistant targets fixed while identifying the training-context effect within paired training blocks under a common evaluation protocol. The base model provides a reference for separating the contrast between the fine-tuned models from the change under each condition. Across four paired training blocks of Qwen and Mistral, insecure training increases coherent low-alignment tail risk by 2.25–6.25 percentage points relative to educational training. We find interpolation along the learned parameter contrast yields positive tail-risk trends, while spectrum-matched orientation controls produce weaker sequence-distribution and behavioral responses. The behavioral effects also vary across evaluation panels and base models: unrelated-code effects have opposite signs in two models. These results identify a training-context effect. Parameter orientation shapes its functional response, and its measured expression varies with the base model, outcome, and evaluation panel.

Primary area:
causal reasoning
Keywords:
Emergent Misalignment, AI Safety and Alignment, Causal Inference, Causal Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kfM9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper, as best as I can understand it, revisits one of the findings in Betley et al. 2025, where changing the framing of the insecure code as being educational doesn't make the model as misaligned. To do this, they fine-tune Qwen2.5-coder-32B and Mistral--small-24B on both versions of that dataset with two runs per model, where the seeds are matched. They find consistent evidence that both the insecure code and educational datasets increase misalignment over base and that the insecure version is also consistently more misaligned than the educational version. They also do some follow-up analysis: if you get weighted averages between the two fine-tuned model diffs (AFAICT, literally using PEFT's cat merge), misalignment grows as you increase the fraction of the weights that are from the insecure version. Also, the size of the effects depend on the model and the test questions.

Critical strengths

I thought it was a good idea to also try souping the weights, since many persona-like changes from narrow fine-tuning can be thought of as steering vector distillation (Blank et al. 2026, https://arxiv.org/abs/2606.00995). As such, many useful things might be gleaned by taking steering vector arithmetic-type diagnostic experiments to supplement your understanding of the expeirmental results.

Critical weaknesses
  • The main weakness of this paper, is that the main result is a replication of what has already been found in Betley et al. 2025. I didn't really value the formalism on top of the existing experiments, which we can dig a little bit. Best as i can understand, the authors want to describe their contribution as a causal identification on top of the control run by betley et al. One red flag that pins this down is: in the RW, the authors say that Betley et al "establishes that training context can change behavior while assistant targets remain fixed." In causal inference (especially in economic settings), we want to use identification tools when you can't run the experiment, so that you can try to pin down when you can use observational data to stand in for an intervention. But here: you can! And in fact, the authors did. So I guess I have a hard time understanding what Proposition 3.2 ought let us conclude that a standard controlled comparison an ML researcher might run between the two fine-tunes without this machinery.
Questions for authors
  • What does Proposition 3.2 let you conclude that a standard controlled comparison between the two fine-tunes would not?

  • Is the training loss restricted to the assistant’s answer?

Reviewer_Vva9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the effect of the context (framing of the question) in emergent misalignment. Building on previous experimental work, the paper formalizes the setup in terms of the potential outcomes framework. It then tries to improve the framework and adds comparison with the base model. Beyond the known result that framing affects emergent misalignment, the paper finds that both secure and insecure trained models still exceed the base model’s risk. Further, the paper intervenes on the weights of the model by interpolating across the LoRA adapter. It finds a positive trend of the risk across the adapter interpolation and finds that the orientation of the update is important to determine its effect.

Critical strengths
  • Understanding the effect of the framing of the question in the emergence of misalignment is important.

  • The paper finds, by comparing with the base model, that educational framing still raises the risk of misalignment. As far as I understand, this is a new result not reported Betley et al. (2025). In general, the paper identifies and removes confounders to effectively compute the effect of framing.

Critical weaknesses
  • The main experimental design is from Betley et al. (2025). The original paper already studies the effect of educational framing on emergent misalignment. The novelty of the paper thus seems limited to the causal framing, the base-model experiments, and the weight-space experiments. This is the main limitation of the paper.

  • The writing is very dense and hard to read. In particular, the theoretical contributions and causal framing overcomplicates the setup and defines many unnecessary terms. I would advise the authors on simplifying this section and move some results to the appendix.

  • The observation that manipulating the weight difference between the two sets of LoRA parameters leads to a smooth interpolation seems expected given the task vector literature and offers little insight into the underlying mechanisms. Similarly, the random direction experimental result is also expected. In high dimensions, a move in a random direction often has little impact on the output of the model.

  • Beyond showing that the direction of the update is the cause of the emergence of misalignment, the paper does not study what this direction consists of. Is the effect located in certain layers ? How many singular vectors explain its effect ? How does it relate to known work such as steering/persona vectors ?

Questions for authors

clarification:

  • What are the novel contributions in the experimental design that are not part of the design of Betley et al. (2025) ? Are the 48-family panel the same as in Betley et al. (2025) ? If so, this should be stated more clearly.
  • How does the new formalism/design affect the results of the weight difference experiments ? How are the two sections related ?
  • What theoretical result provides novel insight that could not have been explained and justified more simply ? In particular, what new contributions does the potential outcome framework provide ?
  • Is the evaluation sensitive to the judge that is chosen (e.g. GPT 5.4)

suggestion:

  • Could the study be extended to other datasets or models beyond the work of Betley et al. (2025) ?
  • Could the author characterize more clearly the weight direction itself ?
Reviewer_NAh6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the effect of user side context (e.g. educational framing) for EM. It starts by first reproducing the insecure educational results from Betley et al. with different models. The models are trained in fixed settings, with the assistant response held fixed to investigate the user side context causally changes the misalignment, and where it lives (model weights).

Critical strengths
  1. The design isolates training context cleanly: the same targets, seed and recipe, a blinded judge, a common denominator over all responses, and a primary analysis frozen before evaluation (internally) with family-level inference.
  2. The base-model reference shows that educational training also raises tail risk, only less, which corrects how the Betley et al. control is usually read, although this rests on point estimates.
Critical weaknesses
  1. They only tested one dataset (insecure and its educational variant), which yielded small EM by nature. I think there could have been more educational variants for other EM datasets. Also there could have been more blocks trained, with variations reported. For example, for qwen block 2, “educational is worse” is already weak.
  2. The spectrum control is also weak. Random orientations are already orthogonal to the subspaces. So “parameter orientation shapes its functional response” in the abstract is maybe overstated. I think one can use another unrelated adapter that is trained on the same configs, for example the secure dataset, and do the spectrum control on it. This would already be a stronger test.
  3. The generalization is weak across models and held-out test sets.
  4. The paper is a little bit hard to understand (very heavy in math, which is unnecessary to explain the paper's approach, which is actually quite simple). So the main claim is, unfortunately, kind of buried into that.
  5. Also I think the paper should be more clearly positioned. For example, Dubiński et al. (2026) should have been cited for the “educational framing still yields EM” claim, with results compared with the proposed approach of the current paper.
Questions for authors
  • Have you built/tried different datasets (e.g.m bad medical advice, risky finances)? It would be interesting to see if educational framing also works for other domains.
  • Is it possible to train more blocks and report the variation across seeds?
  • What is “training randomness” here (e.g., seed, initialization, dropout, etc.)?
  • What are the parameters used for the experiments (LR, epoch, batch size, etc.)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.